c2b47e7645
- Add document ingestion with text chunking, embedding generation, and PostgreSQL persistence - Configure PostgreSQL with pgvector, document schemas, and database migrations - Implement semantic search using Ollama embeddings and pgvector cosine similarity - Add POST /documents, POST /search, and POST /ask endpoints with validation and sanitized error handling - Generate context-grounded answers with source attribution - Filter retrieved chunks using a minimum similarity threshold of 0.5 and return fallback responses when no relevant context is available - Add unit and integration tests for ingestion, search, question answering, validation, and failure handling - Update README with Phase 3 capabilities, setup instructions, and API documentation |
||
|---|---|---|
| db/migrations | ||
| src | ||
| .gitignore | ||
| compose.yaml | ||
| LICENSE | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| tsconfig.json | ||
Local AI Platform
A self-hosted AI application built to demonstrate production-minded backend engineering, local LLM integration, retrieval-augmented generation (RAG), and reliable API design.
The project emphasizes measurable behavior, reproducible development, automated testing, and documented architectural trade-offs.
Project Goals
- Build a well-defined API for local LLM inference.
- Integrate retrieval-augmented generation with source-grounded responses.
- Measure latency, throughput, errors, and retrieval quality.
- Implement structured logging, metrics, and distributed tracing.
- Apply authentication, authorization, input validation, and secure configuration.
- Test failure handling, retries, idempotency, and recovery.
- Automate testing and deployment.
- Document architectural decisions and operational procedures.
These goals are implemented incrementally.
Technology Stack
Currently implemented
- Runtime: Node.js
- Language: TypeScript
- API framework: Fastify
- Request validation: Zod
- Local inference and embeddings: Ollama
- Generation model:
qwen3:8b - Embedding model:
nomic-embed-text - Database and vector search: PostgreSQL with pgvector
- Testing: Node.js test runner with
tsx
Planned
- Containers: Docker and Docker Compose production configuration
- Observability: OpenTelemetry and compatible monitoring tools
- CI/CD: Automated build, test, and deployment workflows
- Evaluation: Performance benchmarks and retrieval-quality tests
Technologies will be introduced as project requirements justify them.
Current Status
Status: API foundation, local inference, document ingestion, semantic retrieval, and RAG question answering implemented.
API Endpoints
| Endpoint | Purpose |
|---|---|
GET /health |
Reports API health. |
GET /ready |
Checks whether Ollama is available. |
POST /echo |
Demonstrates request validation and API behavior. |
POST /generate |
Sends a prompt to a local Ollama model and returns generated text. |
POST /documents |
Ingests a document, splits it into chunks, generates embeddings, and persists the document and chunks. |
POST /search |
Retrieves relevant document chunks using semantic similarity search. |
POST /ask |
Retrieves relevant context, generates an answer using Ollama, and returns source attribution. |
The API includes centralized error handling, input validation, and consistent responses for unknown routes.
Retrieval-Augmented Generation
The implemented RAG pipeline:
- Accepts a document for ingestion.
- Splits document content into overlapping chunks.
- Generates embeddings using Ollama.
- Stores document metadata, chunks, and embeddings in PostgreSQL with pgvector.
- Embeds incoming search queries and retrieves similar chunks.
- Filters retrieved results using a provisional cosine-similarity threshold of
0.5. - Passes relevant context to the generation model.
- Returns an answer and its supporting source information.
The answer-generation service returns a fallback response when no retrieved results meet the similarity threshold. Responses include source document IDs, titles, chunk content, and similarity scores.
The similarity threshold is an initial heuristic, not a validated universal cutoff. Retrieval quality and answer grounding require further evaluation.
Current Limitations
The following capabilities remain future work:
- Authentication and authorization
- Production deployment
- Metrics and distributed tracing
- CI/CD
- Systematic retrieval and answer-quality evaluation
- Performance benchmarking
- Production-grade timeout, retry, and recovery policies
The current implementation is a local development foundation, not a production-ready public service.
Getting Started
Prerequisites
- Node.js and npm
- Git
- Ollama
- Docker and Docker Compose for the PostgreSQL development environment
Installation
Clone the repository:
git clone https://git.brandonwolfgang.com/brandon/local-ai-platform.git
cd local-ai-platform
Install dependencies:
npm ci
Ensure Ollama is running and the required models are available:
ollama pull qwen3:8b
ollama pull nomic-embed-text
Start the PostgreSQL development database using the repository's Docker Compose configuration. Configure the required database connection and password in your local .env file, following the configuration expected by the application.
Do not commit .env or other files containing secrets.
Development
Start the development server:
npm run dev
The API listens on http://127.0.0.1:3000.
Check API health:
curl http://127.0.0.1:3000/health
Check inference-service readiness:
curl http://127.0.0.1:3000/ready
Generate a response:
curl -X POST http://127.0.0.1:3000/generate \
-H "Content-Type: application/json" \
-d '{"message":"Explain what a REST API does in one sentence."}'
Ingest a document:
curl -X POST http://127.0.0.1:3000/documents \
-H "Content-Type: application/json" \
-d '{
"title": "RAG Basics",
"source": "manual-test",
"content": "Retrieval augmented generation retrieves relevant information from a knowledge base and uses it to help generate a response."
}'
Search the document collection:
curl -X POST http://127.0.0.1:3000/search \
-H "Content-Type: application/json" \
-d '{"query":"How does retrieval augmented generation use a knowledge base?"}'
Ask a question using retrieved context:
curl -X POST http://127.0.0.1:3000/ask \
-H "Content-Type: application/json" \
-d '{"question":"How does retrieval augmented generation use a knowledge base?"}'
The /ask response contains an answer and a sources array. If no retrieved chunk meets the current relevance threshold, the service returns its fallback answer with an empty sources array.
Available Scripts
| Command | Purpose |
|---|---|
npm run dev |
Start the development server with automatic restarts |
npm run build |
Compile TypeScript into dist/ |
npm start |
Run the compiled application |
npm test |
Run the automated test suite |
Architecture
The current architecture consists of a TypeScript/Fastify API, a local Ollama inference service, and a PostgreSQL database with pgvector.
The API validates incoming requests and delegates inference, ingestion, retrieval, and answer generation to separate application modules. Document ingestion persists embeddings for later retrieval. The answering service uses retrieved context to construct a prompt and returns source attribution with the generated answer.
The application uses dependency injection to test API routes and answer-generation behavior without requiring live inference or database dependencies in every unit test.
Future iterations will focus on deployment, security controls, operational visibility, evaluation, and reliability.
Architecture diagrams and significant design decisions will be maintained alongside the code.
Testing and Quality
The automated test suite covers:
- API health and readiness behavior
- Request validation and input limits
- Successful inference responses
- Inference failure handling
- Document ingestion and search routes
- Answer generation and source attribution
- Empty retrieval and low-similarity filtering
- Dependency failure handling
- Unknown routes and generic server errors
Run the test suite:
npm test
Verify the production build:
npm run build
The current implementation has been manually exercised end to end with relevant and unrelated questions. Further work will add a reproducible evaluation dataset, retrieval-quality measurements, performance benchmarks, security regression tests, and recovery tests.
Performance and quality claims will be backed by reproducible measurements.
Security
- Never commit credentials, access tokens, private keys, or production data.
- Keep local configuration and secrets outside version control.
- Validate untrusted input and enforce authorization at the API boundary.
- Treat retrieved documents and model-generated output as untrusted.
- Do not expose the API publicly without appropriate authentication and security controls.
Report suspected vulnerabilities privately to the repository owner.
Documentation
As the project develops, documentation will cover:
- Local setup and deployment
- Architecture decision records
- Performance and evaluation results
- Threat models and security controls
- Operational runbooks and recovery procedures
- Known limitations and future improvements