05f6583870
- Add tests for successful document ingestion and response payloads - Verify invalid document input returns HTTP 400 without invoking ingestion - Verify ingestion failures return sanitized HTTP 503 responses - Inject the ingestion function through buildApp for isolated testing - Increase the maximum document source length to 2,000 characters |
||
|---|---|---|
| db/migrations | ||
| src | ||
| .gitignore | ||
| compose.yaml | ||
| LICENSE | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| tsconfig.json | ||
Local AI Platform
A self-hosted AI application built to demonstrate production-minded backend engineering, local LLM integration, retrieval-augmented generation (RAG), and reliable API design.
The project emphasizes measurable behavior, reproducible development, automated testing, and documented architectural trade-offs.
Project Goals
- Build a well-defined API for local LLM inference.
- Integrate retrieval-augmented generation with source-grounded responses.
- Measure latency, throughput, errors, and retrieval quality.
- Implement structured logging, metrics, and distributed tracing.
- Apply authentication, authorization, input validation, and secure configuration.
- Test failure handling, retries, idempotency, and recovery.
- Automate testing and deployment.
- Document architectural decisions and operational procedures.
These goals will be implemented incrementally.
Technology Stack
Currently implemented
- Runtime: Node.js
- Language: TypeScript
- API framework: Fastify
- Request validation: Zod
- Local inference: Ollama
- Testing: Node.js test runner with
tsx
Planned
- Database and retrieval: PostgreSQL with pgvector
- Containers: Docker and Docker Compose
- Observability: OpenTelemetry and compatible monitoring tools
- CI/CD: Automated build, test, and deployment workflows
- Evaluation: Performance benchmarks and retrieval-quality tests
Technologies will be introduced as project requirements justify them.
Current Status
Status: Initial API foundation and local inference integration implemented.
The API currently provides:
GET /health— reports API health.GET /ready— checks whether Ollama is available.POST /echo— demonstrates request validation and API behavior.POST /generate— sends a prompt to a local Ollama model and returns generated text.- Centralized error handling and consistent responses for unknown routes.
- Automated tests for API behavior, validation, inference responses, and failure handling.
The inference client currently uses the qwen3:8b model and connects to Ollama at http://127.0.0.1:11434.
Not yet implemented: RAG, PostgreSQL persistence, authentication, production deployment, metrics, distributed tracing, and CI/CD.
Getting Started
Prerequisites
- Node.js and npm
- Git
- Ollama for local inference
Installation
Clone the repository:
git clone https://git.brandonwolfgang.com/brandon/local-ai-platform.git
cd local-ai-platform
Install dependencies:
npm ci
Ensure Ollama is running and the configured model is available:
ollama pull qwen3:8b
Development
Start the development server:
npm run dev
The API listens on http://127.0.0.1:3000.
Check API health:
curl http://127.0.0.1:3000/health
Check inference-service readiness:
curl http://127.0.0.1:3000/ready
Generate a response:
curl -X POST http://127.0.0.1:3000/generate \
-H "Content-Type: application/json" \
-d '{"message":"Explain what a REST API does in one sentence."}'
Available Scripts
| Command | Purpose |
|---|---|
npm run dev |
Start the development server with automatic restarts |
npm run build |
Compile TypeScript into dist/ |
npm start |
Run the compiled application |
npm test |
Run the automated test suite |
Architecture
The current architecture consists of a TypeScript/Fastify HTTP API and a local Ollama inference service.
The API validates incoming requests, delegates generation to Ollama, and returns generated text. Readiness checks verify the inference service is reachable, while error handling prevents internal exception details from being returned in API responses.
Future iterations will introduce document ingestion, retrieval, persistence, security controls, observability, and deployment automation.
Architecture diagrams and significant design decisions will be maintained alongside the code.
Testing and Quality
The automated test suite currently covers:
- API health and readiness behavior
- Request validation and input limits
- Successful inference responses
- Inference failure handling
- Unknown routes and generic server errors
- Ollama client response handling
Run the test suite:
npm test
Verify the production build:
npm run build
Performance benchmarks, retrieval evaluations, security regression tests, and recovery tests remain planned work.
Performance and quality claims will be backed by reproducible measurements.
Security
- Never commit credentials, access tokens, private keys, or production data.
- Keep local configuration and secrets outside version control.
- Validate untrusted input and enforce authorization at the API boundary.
- Treat retrieved documents and model-generated output as untrusted.
- Do not expose the API publicly without appropriate authentication and security controls.
Report suspected vulnerabilities privately to the repository owner.
Documentation
As the project develops, documentation will cover:
- Local setup and deployment
- Architecture decision records
- Performance and evaluation results
- Threat models and security controls
- Operational runbooks and recovery procedures
- Known limitations and future improvements