# Local AI Platform

A self-hosted AI application built to demonstrate production-minded backend engineering, local LLM integration, retrieval-augmented generation (RAG), and reliable API design.

The project emphasizes measurable behavior, reproducible development, automated testing, and documented architectural trade-offs.

## Project Goals

- Build a well-defined API for local LLM inference.
- Integrate retrieval-augmented generation with source-grounded responses.
- Measure latency, throughput, errors, and retrieval quality.
- Implement structured logging, metrics, and distributed tracing.
- Apply authentication, authorization, input validation, and secure configuration.
- Test failure handling, retries, idempotency, and recovery.
- Automate testing and deployment.
- Document architectural decisions and operational procedures.

These goals are implemented incrementally.

## Technology Stack

**Currently implemented**

- **Runtime:** Node.js
- **Language:** TypeScript
- **API framework:** Fastify
- **Request validation:** Zod
- **Local inference and embeddings:** Ollama
- **Generation model:** `qwen3:8b`
- **Embedding model:** `nomic-embed-text`
- **Database and vector search:** PostgreSQL with pgvector
- **Testing:** Node.js test runner with `tsx`

**Planned**

- **Containers:** Docker and Docker Compose production configuration
- **Observability:** OpenTelemetry and compatible monitoring tools
- **CI/CD:** Automated build, test, and deployment workflows
- **Evaluation:** Performance benchmarks and retrieval-quality tests

Technologies will be introduced as project requirements justify them.

## Current Status

**Status:** API foundation, local inference, document ingestion, semantic retrieval, and RAG question answering implemented.

### API Endpoints

| Endpoint | Purpose |
|---|---|
| `GET /health` | Reports API health. |
| `GET /ready` | Checks whether Ollama is available. |
| `POST /echo` | Demonstrates request validation and API behavior. |
| `POST /generate` | Sends a prompt to a local Ollama model and returns generated text. |
| `POST /documents` | Ingests a document, splits it into chunks, generates embeddings, and persists the document and chunks. |
| `POST /search` | Retrieves relevant document chunks using semantic similarity search. |
| `POST /ask` | Retrieves relevant context, generates an answer using Ollama, and returns source attribution. |

The API includes centralized error handling, input validation, and consistent responses for unknown routes.

### Retrieval-Augmented Generation

The implemented RAG pipeline:

1. Accepts a document for ingestion.
2. Splits document content into overlapping chunks.
3. Generates embeddings using Ollama.
4. Stores document metadata, chunks, and embeddings in PostgreSQL with pgvector.
5. Embeds incoming search queries and retrieves similar chunks.
6. Filters retrieved results using a provisional cosine-similarity threshold of `0.5`.
7. Passes relevant context to the generation model.
8. Returns an answer and its supporting source information.

The answer-generation service returns a fallback response when no retrieved results meet the similarity threshold. Responses include source document IDs, titles, chunk content, and similarity scores.

The similarity threshold is an initial heuristic, not a validated universal cutoff. Retrieval quality and answer grounding require further evaluation.

### Current Limitations

The following capabilities remain future work:

- Authentication and authorization
- Production deployment
- Metrics and distributed tracing
- CI/CD
- Systematic retrieval and answer-quality evaluation
- Performance benchmarking
- Production-grade timeout, retry, and recovery policies

The current implementation is a local development foundation, not a production-ready public service.

## Getting Started

### Prerequisites

- Node.js and npm
- Git
- [Ollama](https://ollama.com/)
- Docker and Docker Compose for the PostgreSQL development environment

### Installation

Clone the repository:

```bash
git clone https://git.brandonwolfgang.com/brandon/local-ai-platform.git
cd local-ai-platform
```

Install dependencies:

```bash
npm ci
```

Ensure Ollama is running and the required models are available:

```bash
ollama pull qwen3:8b
ollama pull nomic-embed-text
```

Start the PostgreSQL development database using the repository's Docker Compose configuration. Configure the required database connection and password in your local `.env` file, following the configuration expected by the application.

Do not commit `.env` or other files containing secrets.

### Development

Start the development server:

```bash
npm run dev
```

The API listens on `http://127.0.0.1:3000`.

Check API health:

```bash
curl http://127.0.0.1:3000/health
```

Check inference-service readiness:

```bash
curl http://127.0.0.1:3000/ready
```

Generate a response:

```bash
curl -X POST http://127.0.0.1:3000/generate \
  -H "Content-Type: application/json" \
  -d '{"message":"Explain what a REST API does in one sentence."}'
```

Ingest a document:

```bash
curl -X POST http://127.0.0.1:3000/documents \
  -H "Content-Type: application/json" \
  -d '{
    "title": "RAG Basics",
    "source": "manual-test",
    "content": "Retrieval augmented generation retrieves relevant information from a knowledge base and uses it to help generate a response."
  }'
```

Search the document collection:

```bash
curl -X POST http://127.0.0.1:3000/search \
  -H "Content-Type: application/json" \
  -d '{"query":"How does retrieval augmented generation use a knowledge base?"}'
```

Ask a question using retrieved context:

```bash
curl -X POST http://127.0.0.1:3000/ask \
  -H "Content-Type: application/json" \
  -d '{"question":"How does retrieval augmented generation use a knowledge base?"}'
```

The `/ask` response contains an `answer` and a `sources` array. If no retrieved chunk meets the current relevance threshold, the service returns its fallback answer with an empty sources array.

### Available Scripts

| Command | Purpose |
|---|---|
| `npm run dev` | Start the development server with automatic restarts |
| `npm run build` | Compile TypeScript into `dist/` |
| `npm start` | Run the compiled application |
| `npm test` | Run the automated test suite |

## Architecture

The current architecture consists of a TypeScript/Fastify API, a local Ollama inference service, and a PostgreSQL database with pgvector.

The API validates incoming requests and delegates inference, ingestion, retrieval, and answer generation to separate application modules. Document ingestion persists embeddings for later retrieval. The answering service uses retrieved context to construct a prompt and returns source attribution with the generated answer.

The application uses dependency injection to test API routes and answer-generation behavior without requiring live inference or database dependencies in every unit test.

Future iterations will focus on deployment, security controls, operational visibility, evaluation, and reliability.

Architecture diagrams and significant design decisions will be maintained alongside the code.

## Testing and Quality

The automated test suite covers:

- API health and readiness behavior
- Request validation and input limits
- Successful inference responses
- Inference failure handling
- Document ingestion and search routes
- Answer generation and source attribution
- Empty retrieval and low-similarity filtering
- Dependency failure handling
- Unknown routes and generic server errors

Run the test suite:

```bash
npm test
```

Verify the production build:

```bash
npm run build
```

The current implementation has been manually exercised end to end with relevant and unrelated questions. Further work will add a reproducible evaluation dataset, retrieval-quality measurements, performance benchmarks, security regression tests, and recovery tests.

Performance and quality claims will be backed by reproducible measurements.

## Security

- Never commit credentials, access tokens, private keys, or production data.
- Keep local configuration and secrets outside version control.
- Validate untrusted input and enforce authorization at the API boundary.
- Treat retrieved documents and model-generated output as untrusted.
- Do not expose the API publicly without appropriate authentication and security controls.

Report suspected vulnerabilities privately to the repository owner.

## Documentation

As the project develops, documentation will cover:

- Local setup and deployment
- Architecture decision records
- Performance and evaluation results
- Threat models and security controls
- Operational runbooks and recovery procedures
- Known limitations and future improvements

## License

[MIT License](LICENSE)