local-ai-platform/README.md

# Local AI Platform

A self-hosted AI application built to demonstrate production-minded backend engineering, local LLM integration, retrieval-augmented generation (RAG), and reliable API design.

The project emphasizes measurable behavior, reproducible development, automated testing, and documented architectural trade-offs.

## Project Goals

- Build a well-defined API for local LLM inference.
- Integrate retrieval-augmented generation with source-grounded responses.
- Measure latency, throughput, errors, and retrieval quality.
- Implement structured logging, metrics, and distributed tracing.
- Apply authentication, authorization, input validation, and secure configuration.
- Test failure handling, retries, idempotency, and recovery.
- Automate testing and deployment.
- Document architectural decisions and operational procedures.

These goals will be implemented incrementally.

## Technology Stack

**Currently implemented**

- **Runtime:** Node.js
- **Language:** TypeScript
- **API framework:** Fastify
- **Request validation:** Zod
- **Local inference:** Ollama
- **Testing:** Node.js test runner with `tsx`

**Planned**

- **Database and retrieval:** PostgreSQL with pgvector
- **Containers:** Docker and Docker Compose
- **Observability:** OpenTelemetry and compatible monitoring tools
- **CI/CD:** Automated build, test, and deployment workflows
- **Evaluation:** Performance benchmarks and retrieval-quality tests

Technologies will be introduced as project requirements justify them.

## Current Status

**Status:** Initial API foundation and local inference integration implemented.

The API currently provides:

- `GET /health` — reports API health.
- `GET /ready` — checks whether Ollama is available.
- `POST /echo` — demonstrates request validation and API behavior.
- `POST /generate` — sends a prompt to a local Ollama model and returns generated text.
- Centralized error handling and consistent responses for unknown routes.
- Automated tests for API behavior, validation, inference responses, and failure handling.

The inference client currently uses the `qwen3:8b` model and connects to Ollama at `http://127.0.0.1:11434`.

**Not yet implemented:** RAG, PostgreSQL persistence, authentication, production deployment, metrics, distributed tracing, and CI/CD.

## Getting Started

### Prerequisites

- Node.js and npm
- Git
- [Ollama](https://ollama.com/) for local inference

### Installation

Clone the repository:

```bash
git clone https://git.brandonwolfgang.com/brandon/local-ai-platform.git
cd local-ai-platform
```

Install dependencies:

```bash
npm ci
```

Ensure Ollama is running and the configured model is available:

```bash
ollama pull qwen3:8b
```

### Development

Start the development server:

```bash
npm run dev
```

The API listens on `http://127.0.0.1:3000`.

Check API health:

```bash
curl http://127.0.0.1:3000/health
```

Check inference-service readiness:

```bash
curl http://127.0.0.1:3000/ready
```

Generate a response:

```bash
curl -X POST http://127.0.0.1:3000/generate \
  -H "Content-Type: application/json" \
  -d '{"message":"Explain what a REST API does in one sentence."}'
```

### Available Scripts

| Command | Purpose |
|---|---|
| `npm run dev` | Start the development server with automatic restarts |
| `npm run build` | Compile TypeScript into `dist/` |
| `npm start` | Run the compiled application |
| `npm test` | Run the automated test suite |

## Architecture

The current architecture consists of a TypeScript/Fastify HTTP API and a local Ollama inference service.

The API validates incoming requests, delegates generation to Ollama, and returns generated text. Readiness checks verify the inference service is reachable, while error handling prevents internal exception details from being returned in API responses.

Future iterations will introduce document ingestion, retrieval, persistence, security controls, observability, and deployment automation.

Architecture diagrams and significant design decisions will be maintained alongside the code.

## Testing and Quality

The automated test suite currently covers:

- API health and readiness behavior
- Request validation and input limits
- Successful inference responses
- Inference failure handling
- Unknown routes and generic server errors
- Ollama client response handling

Run the test suite:

```bash
npm test
```

Verify the production build:

```bash
npm run build
```

Performance benchmarks, retrieval evaluations, security regression tests, and recovery tests remain planned work.

Performance and quality claims will be backed by reproducible measurements.

## Security

- Never commit credentials, access tokens, private keys, or production data.
- Keep local configuration and secrets outside version control.
- Validate untrusted input and enforce authorization at the API boundary.
- Treat retrieved documents and model-generated output as untrusted.
- Do not expose the API publicly without appropriate authentication and security controls.

Report suspected vulnerabilities privately to the repository owner.

## Documentation

As the project develops, documentation will cover:

- Local setup and deployment
- Architecture decision records
- Performance and evaluation results
- Threat models and security controls
- Operational runbooks and recovery procedures
- Known limitations and future improvements

## License

[MIT License](LICENSE)