Stand up an Ollama LLM Engine + FastAPI + Chroma Vector DB + LangGraph Agent on a laptop with a single Docker Compose file. The cloud AI Agent calls only this server, and no paid external APIs are used.
This project clearly separates the cloud (public touchpoint) from the laptop (private AI engine).
Browser / mobile
β HTTPS
βΌ
βββββββββββββββββββββββββββββββ
β Cloud AI Agent β Cloudflare Pages / Next.js
β - User auth, sessions β
β - Rate limit, CORS β
β - Audit logs, policy checksβ
ββββββββββββ¬βββββββββββββββββββ
β x-api-key (server-side only)
β Cloudflare Tunnel or VPN
βΌ
βββββββββββββββββββββββββββββββ
β Laptop β Docker Compose β i7-12700H Β· 16GB Β· CPU only
β β
β βββββββββββ βββββββββββββ β
β β Ollama β β Chroma β β
β β :11434 β β :8001 β β
β ββββββ¬βββββ βββββββ¬ββββββ β
β ββββββββ¬ββββββ β
β βββββΌβββββββββ β
β β FastAPI β β port 8000 (local binding)
β β /v1/chat β β
β β /v1/rag β β
β β /v1/agent β β
β ββββββββββββββ β
βββββββββββββββββββββββββββββββ| Layer | Responsibility | Publicly exposed |
|---|---|---|
| Cloud AI Agent | Auth, sessions, routing, policy, audit logs | Yes (HTTPS) |
| FastAPI (laptop) | LLM calls, RAG, Agent workflow | No (Tunnel only) |
| Ollama (laptop) | LLM inference engine | No (internal network) |
| Chroma (laptop) | Vector DB | No (internal network) |
| Item | Version | How to check |
|---|---|---|
| Docker Desktop | 4.30+ | docker --version |
| Python | 3.11+ | python --version |
| Node.js (for cloud integration) | 20 LTS | node --version |
| Free RAM | 8GB or more | Task Manager |
| Free disk | 30GB or more | File Explorer |
This guide assumes an i7-12700H (14 cores), 16GB RAM, CPU-only machine with no GPU. Even without a GPU, llama3.1:8b runs at about 5 tok/s.
# Check that Docker Desktop is installed
docker --version # Docker version 27.x
docker compose version # Docker Compose version v2.x
# Check that Docker Desktop is running
docker pslaptop-llm-api-server/
βββ docker-compose.yml # service definitions
βββ .env # environment variables (git-ignored)
βββ .env.example # example file (committed)
β
βββ api/ # FastAPI server
β βββ Dockerfile
β βββ requirements.txt
β βββ app/
β βββ main.py # FastAPI app entry point
β βββ auth.py # API key authentication
β βββ chat.py # LLM Chat endpoint
β βββ rag.py # RAG pipeline
β βββ agent.py # LangGraph Agent
β βββ schemas.py # Pydantic models
β
βββ documents/ # RAG source documents (git-ignored)
βββ sample.md # sample documentservices:
ollama:
image: ollama/ollama:latest
container_name: local-ollama
ports:
- "127.0.0.1:11434:11434" # not exposed externally
volumes:
- ollama_data:/root/.ollama
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
interval: 30s
timeout: 10s
retries: 3
chroma:
image: chromadb/chroma:latest
container_name: local-chroma
ports:
- "127.0.0.1:8001:8000" # not exposed externally
volumes:
- chroma_data:/chroma/chroma
environment:
IS_PERSISTENT: "TRUE"
ANONYMIZED_TELEMETRY: "FALSE"
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/api/v1/heartbeat"]
interval: 30s
timeout: 5s
retries: 3
api:
build:
context: ./api
dockerfile: Dockerfile
container_name: local-agent-api
ports:
- "127.0.0.1:8000:8000" # not exposed externally
env_file:
- .env
volumes:
- ./documents:/app/documents:ro
- ./logs:/app/logs
depends_on:
ollama:
condition: service_healthy
chroma:
condition: service_healthy
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 5s
retries: 3
volumes:
ollama_data:
chroma_data:# API authentication
LOCAL_API_KEY=change-this-to-a-random-secret-key-32chars
# Ollama (address on the container network)
OLLAMA_BASE_URL=http://ollama:11434
LLM_MODEL=llama3.1:8b
LLM_MODEL_FAST=llama3.2:3b
MODEL_TIMEOUT_SECONDS=120
# Chroma (address on the container network)
CHROMA_HOST=chroma
CHROMA_PORT=8000
RAG_COLLECTION=private_docs
# Embeddings
EMBEDDING_MODEL=nomic-embed-text
# Logging
LOG_LEVEL=INFOFROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ ./app/
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]fastapi==0.115.12
uvicorn[standard]==0.34.3
httpx==0.28.1
pydantic==2.11.5
python-dotenv==1.1.0
chromadb==1.0.12
langchain==0.3.25
langchain-community==0.3.25
langchain-ollama==0.3.3
langgraph==0.4.8
sentence-transformers==3.4.1from pydantic import BaseModel
class ChatRequest(BaseModel):
message: str
system: str | None = None
model: str | None = None
class RagRequest(BaseModel):
query: str
collection: str | None = None
top_k: int = 4
class IngestRequest(BaseModel):
text: str
metadata: dict = {}
collection: str | None = None
class AgentRequest(BaseModel):
task: str
context: str | None = None
class StandardResponse(BaseModel):
answer: str
model: str = ""
latency_ms: int = 0
citations: list[dict] = []
trace_id: str = ""import os
import secrets
from fastapi import Header, HTTPException
_API_KEY = os.getenv("LOCAL_API_KEY", "")
def require_api_key(x_api_key: str | None = Header(default=None, alias="x-api-key")) -> None:
"""Validates the API key on every private endpoint."""
if not _API_KEY:
raise RuntimeError("The LOCAL_API_KEY environment variable is not set.")
if not x_api_key or not secrets.compare_digest(x_api_key, _API_KEY):
raise HTTPException(status_code=401, detail="Invalid API key")import time
import uuid
import logging
from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from .chat import router as chat_router
from .rag import router as rag_router
from .agent import router as agent_router
logging.basicConfig(level=__import__('os').getenv('LOG_LEVEL', 'INFO'))
logger = logging.getLogger(__name__)
@asynccontextmanager
async def lifespan(app: FastAPI):
logger.info("Local AI Agent API server starting")
yield
logger.info("Server shutting down")
app = FastAPI(
title="Private Local AI Agent API",
version="1.0.0",
docs_url="/docs", # Swagger UI during development
lifespan=lifespan,
)
app.add_middleware(
CORSMiddleware,
allow_origins=["http://localhost:3000"], # local development only
allow_methods=["POST", "GET"],
allow_headers=["x-api-key", "content-type"],
)
app.include_router(chat_router, prefix="/v1")
app.include_router(rag_router, prefix="/v1")
app.include_router(agent_router, prefix="/v1")
@app.get("/health")
def health():
return {"status": "ok", "version": "1.0.0"}import os
import time
import httpx
from fastapi import APIRouter, Depends
from .auth import require_api_key
from .schemas import ChatRequest, StandardResponse
router = APIRouter(tags=["chat"])
OLLAMA_URL = os.getenv("OLLAMA_BASE_URL", "http://ollama:11434")
DEFAULT_MODEL = os.getenv("LLM_MODEL", "llama3.1:8b")
TIMEOUT = int(os.getenv("MODEL_TIMEOUT_SECONDS", "120"))
@router.post("/chat", response_model=StandardResponse, dependencies=[Depends(require_api_key)])
async def chat(req: ChatRequest):
"""Simple LLM chat endpoint."""
model = req.model or DEFAULT_MODEL
messages = []
if req.system:
messages.append({"role": "system", "content": req.system})
messages.append({"role": "user", "content": req.message})
start = time.monotonic()
async with httpx.AsyncClient(timeout=TIMEOUT) as client:
res = await client.post(
f"{OLLAMA_URL}/api/chat",
json={"model": model, "messages": messages, "stream": False},
)
res.raise_for_status()
data = res.json()
answer = data["message"]["content"]
elapsed = int((time.monotonic() - start) * 1000)
return StandardResponse(answer=answer, model=model, latency_ms=elapsed)
@router.get("/models", dependencies=[Depends(require_api_key)])
async def list_models():
"""Returns the list of installed Ollama models."""
async with httpx.AsyncClient(timeout=10) as client:
res = await client.get(f"{OLLAMA_URL}/api/tags")
res.raise_for_status()
return {"models": [m["name"] for m in res.json().get("models", [])]}import os
import time
import httpx
import chromadb
from fastapi import APIRouter, Depends, HTTPException
from langchain.text_splitter import RecursiveCharacterTextSplitter
from .auth import require_api_key
from .schemas import RagRequest, IngestRequest, StandardResponse
router = APIRouter(tags=["rag"])
OLLAMA_URL = os.getenv("OLLAMA_BASE_URL", "http://ollama:11434")
LLM_MODEL = os.getenv("LLM_MODEL", "llama3.1:8b")
EMBED_MODEL = os.getenv("EMBEDDING_MODEL", "nomic-embed-text")
COLLECTION = os.getenv("RAG_COLLECTION", "private_docs")
TIMEOUT = int(os.getenv("MODEL_TIMEOUT_SECONDS", "120"))
def _chroma():
return chromadb.HttpClient(
host=os.getenv("CHROMA_HOST", "chroma"),
port=int(os.getenv("CHROMA_PORT", "8000")),
)
async def _embed(text: str) -> list[float]:
"""Call the Ollama embedding API."""
async with httpx.AsyncClient(timeout=30) as client:
res = await client.post(
f"{OLLAMA_URL}/api/embed",
json={"model": EMBED_MODEL, "input": text},
)
res.raise_for_status()
return res.json()["embeddings"][0]
async def _llm(prompt: str) -> str:
"""Call the Ollama LLM."""
async with httpx.AsyncClient(timeout=TIMEOUT) as client:
res = await client.post(
f"{OLLAMA_URL}/api/chat",
json={
"model": LLM_MODEL,
"messages": [{"role": "user", "content": prompt}],
"stream": False,
},
)
res.raise_for_status()
return res.json()["message"]["content"]
@router.post("/documents/ingest", dependencies=[Depends(require_api_key)])
async def ingest(req: IngestRequest):
"""Chunk a document and store it in Chroma."""
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)
chunks = splitter.split_text(req.text)
if not chunks:
raise HTTPException(400, "The document is empty.")
collection_name = req.collection or COLLECTION
db = _chroma()
col = db.get_or_create_collection(collection_name)
embeddings = [await _embed(chunk) for chunk in chunks]
ids = [f"doc-{i}-{hash(chunk) & 0xFFFFFF:06x}" for i, chunk in enumerate(chunks)]
metadatas = [{**req.metadata, "chunk_index": i} for i in range(len(chunks))]
col.add(documents=chunks, embeddings=embeddings, ids=ids, metadatas=metadatas)
return {"ingested_chunks": len(chunks), "collection": collection_name}
@router.post("/rag/query", response_model=StandardResponse, dependencies=[Depends(require_api_key)])
async def rag_query(req: RagRequest):
"""Chroma retrieval + LLM answer generation."""
collection_name = req.collection or COLLECTION
db = _chroma()
try:
col = db.get_collection(collection_name)
except Exception:
raise HTTPException(404, f"Collection '{collection_name}' does not exist. Ingest documents first.")
query_vec = await _embed(req.query)
results = col.query(query_embeddings=[query_vec], n_results=req.top_k)
docs = results["documents"][0]
metas = results["metadatas"][0]
if not docs:
return StandardResponse(answer="No relevant documents found.")
context = "\n\n---\n\n".join(docs)
prompt = f"""Answer the question using the documents below.
If the documents do not contain the answer, reply "I could not find relevant content in the documents".
[Reference documents]
{context}
[Question]
{req.query}
[Answer]"""
start = time.monotonic()
answer = await _llm(prompt)
elapsed = int((time.monotonic() - start) * 1000)
citations = [{"text": d[:120] + "...", "metadata": m} for d, m in zip(docs, metas)]
return StandardResponse(answer=answer, model=LLM_MODEL, latency_ms=elapsed, citations=citations)import os
import time
from typing import TypedDict, Annotated
from fastapi import APIRouter, Depends
from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage, AIMessage, BaseMessage
from langgraph.graph import StateGraph, END
from langgraph.graph.message import add_messages
from .auth import require_api_key
from .schemas import AgentRequest, StandardResponse
router = APIRouter(tags=["agent"])
OLLAMA_URL = os.getenv("OLLAMA_BASE_URL", "http://ollama:11434")
LLM_MODEL = os.getenv("LLM_MODEL", "llama3.1:8b")
# ββ State definition ββ
class AgentState(TypedDict):
messages: Annotated[list[BaseMessage], add_messages]
task: str
result: str
# ββ Node functions ββ
def _plan_node(state: AgentState) -> dict:
"""Analyze the task and build an execution plan."""
llm = ChatOllama(model=LLM_MODEL, base_url=OLLAMA_URL, temperature=0)
plan_prompt = f"""Analyze the following task and plan the execution in at most 3 steps.
Task: {state['task']}
Plan (as a numbered list):"""
response = llm.invoke([HumanMessage(content=plan_prompt)])
return {"messages": [response]}
def _execute_node(state: AgentState) -> dict:
"""Execute the plan and produce the result."""
llm = ChatOllama(model=LLM_MODEL, base_url=OLLAMA_URL, temperature=0)
plan = state["messages"][-1].content if state["messages"] else ""
exec_prompt = f"""Produce the final result based on the task and the plan.
Task: {state['task']}
Plan: {plan}
Context: {state.get('context', '')}
Result:"""
response = llm.invoke([HumanMessage(content=exec_prompt)])
return {"messages": [response], "result": response.content}
def _build_graph():
builder = StateGraph(AgentState)
builder.add_node("plan", _plan_node)
builder.add_node("execute", _execute_node)
builder.set_entry_point("plan")
builder.add_edge("plan", "execute")
builder.add_edge("execute", END)
return builder.compile()
_graph = _build_graph()
@router.post("/agent/run", response_model=StandardResponse, dependencies=[Depends(require_api_key)])
async def run_agent(req: AgentRequest):
"""Run the LangGraph Agent."""
start = time.monotonic()
result = _graph.invoke({
"messages": [],
"task": req.task,
"context": req.context or "",
"result": "",
})
elapsed = int((time.monotonic() - start) * 1000)
return StandardResponse(answer=result["result"], model=LLM_MODEL, latency_ms=elapsed)Start Chroma and Ollama first, then download the models.
# 1. Start Chroma + Ollama first
docker compose up -d ollama chroma
# 2. Check container status (wait until healthy)
docker compose ps
# 3. Download the main model (~5GB, takes 5-10 minutes)
docker exec -it local-ollama ollama pull llama3.1:8b
# 4. Download the embedding model (~300MB)
docker exec -it local-ollama ollama pull nomic-embed-text
# 5. (Optional) lightweight model for fast responses
docker exec -it local-ollama ollama pull llama3.2:3b
# 6. Verify the install
docker exec -it local-ollama ollama list# Test the Ollama API directly
curl http://localhost:11434/api/tags
# Sample response:
# {"models":[
# {"name":"llama3.1:8b","size":4661224960},
# {"name":"nomic-embed-text","size":274302480}
# ]}Memory management: on 16GB of RAM, Docker Desktop needs 8GB and Ollama (llama3.1:8b) needs 5-7GB. Keep the IDE and browser tabs to a minimum. If that is too tight, switch to llama3.2:3b (~2GB).
# Start the API server
docker compose up -d api
# Check the status of all services
docker compose ps
# EXPECTED:
# local-ollama running (healthy)
# local-chroma running (healthy)
# local-agent-api running (healthy)
# Health check
curl http://localhost:8000/health
# {"status":"ok","version":"1.0.0"}
# Request without an API key β expect 401
curl -X POST http://localhost:8000/v1/chat \
-H "content-type: application/json" \
-d '{"message":"test"}'
# {"detail":"Invalid API key"}
# Request with an API key β OK
curl -X POST http://localhost:8000/v1/chat \
-H "content-type: application/json" \
-H "x-api-key: change-this-to-a-random-secret-key-32chars" \
-d '{"message":"Hello. Please introduce yourself briefly."}'# About the company
## Service overview
AIDevOps is a technology platform for building and operating AI Agents.
Main services: Finance AI Agent, system performance optimization, LLMOps guides
## Key features
- Real-time market data analysis
- RAG search over filings
- Automated portfolio rebalancing
## Tech stack
Python, FastAPI, LangChain, LangGraph, ChromaDB, Ollama# Test document ingestion
curl -X POST http://localhost:8000/v1/documents/ingest \
-H "content-type: application/json" \
-H "x-api-key: change-this-to-a-random-secret-key-32chars" \
-d '{
"text": "AIDevOps is an AI Agent development platform. Its key features are financial analysis, RAG search, and LangGraph orchestration.",
"metadata": {"source": "sample", "date": "2026-06-06"}
}'
# {"ingested_chunks":1,"collection":"private_docs"}
# Test a RAG query
curl -X POST http://localhost:8000/v1/rag/query \
-H "content-type: application/json" \
-H "x-api-key: change-this-to-a-random-secret-key-32chars" \
-d '{"query": "What are the key features of AIDevOps?"}'# Bulk-ingest every document in a folder (Python script)
python - <<'EOF'
import httpx, pathlib, os
API = "http://localhost:8000"
KEY = os.getenv("LOCAL_API_KEY", "change-this-to-a-random-secret-key-32chars")
for path in pathlib.Path("documents").rglob("*.md"):
text = path.read_text(encoding="utf-8")
r = httpx.post(
f"{API}/v1/documents/ingest",
headers={"x-api-key": KEY},
json={"text": text, "metadata": {"filename": path.name}},
)
print(f"{path.name}: {r.json()}")
EOFThe cloud AI Agent needs a secure tunnel to call FastAPI on the laptop.
Never do this: do not expose the FastAPI port (8000) directly to the internet. Pair Cloudflare Tunnel with a Cloudflare Access policy so that only server-side clients, not browsers, can reach it.
# 1. Install cloudflared (Windows)
winget install --id Cloudflare.cloudflared
# 2. Log in to Cloudflare
cloudflared tunnel login
# 3. Create the tunnel
cloudflared tunnel create laptop-ai-agent
# 4. Create the config file
# C:Users[user].cloudflaredconfig.yml
# 5. Temporary test tunnel (works right away, no domain needed)
cloudflared tunnel --url http://localhost:8000tunnel: laptop-ai-agent
credentials-file: C:\Users\[user]\.cloudflared\[tunnel-id].json
ingress:
- hostname: laptop-api.your-domain.com
service: http://localhost:8000
originRequest:
noTLSVerify: false
- service: http_status:404# Run the tunnel
cloudflared tunnel run laptop-ai-agent
# An Access Policy must be set in the Cloudflare Dashboard:
# - Service Auth: create a Service Token
# - Allow only the cloud server IP (Bypass policy)
# Test the connection
curl https://laptop-api.your-domain.com/health \
-H "CF-Access-Client-Id: your-service-token-id" \
-H "CF-Access-Client-Secret: your-service-token-secret"Server-side code for calling the laptop FastAPI from the cloud AI Agent (Next.js). It is never called directly from the browser.
const LAPTOP_API_URL = process.env.LAPTOP_AGENT_API_URL!;
const LAPTOP_API_KEY = process.env.LAPTOP_AGENT_API_KEY!;
interface AgentResponse {
answer: string;
model: string;
latency_ms: number;
citations: { text: string; metadata: Record<string, unknown> }[];
}
async function callLaptopApi(path: string, body: unknown): Promise<AgentResponse> {
const url = LAPTOP_API_URL + path;
const res = await fetch(url, {
method: "POST",
headers: {
"content-type": "application/json",
"x-api-key": LAPTOP_API_KEY,
},
body: JSON.stringify(body),
signal: AbortSignal.timeout(130_000),
});
if (!res.ok) {
const err = await res.text();
throw new Error("Laptop API " + res.status + ": " + err);
}
return res.json();
}
export async function chat(message: string, system?: string): Promise<AgentResponse> {
return callLaptopApi("/v1/chat", { message, system });
}
export async function ragQuery(query: string, collection?: string): Promise<AgentResponse> {
return callLaptopApi("/v1/rag/query", { query, collection });
}
export async function runAgent(task: string, context?: string): Promise<AgentResponse> {
return callLaptopApi("/v1/agent/run", { task, context });
}import { NextRequest, NextResponse } from "next/server";
import { ragQuery } from "@/lib/laptop-agent";
import { getServerSession } from "next-auth"; // auth middleware
export async function POST(req: NextRequest) {
const session = await getServerSession();
if (!session) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
const { question } = await req.json();
if (!question?.trim()) {
return NextResponse.json({ error: "question is required" }, { status: 400 });
}
try {
const result = await ragQuery(question);
return NextResponse.json(result);
} catch (err) {
const msg = err instanceof Error ? err.message : "Unknown error";
return NextResponse.json({ error: msg }, { status: 502 });
}
}# Cloud server environment variables (never expose to the browser)
LAPTOP_AGENT_API_URL=https://laptop-api.your-domain.com
LAPTOP_AGENT_API_KEY=change-this-to-a-random-secret-key-32charsimport http from "k6/http";
import { check, sleep } from "k6";
import { Trend } from "k6/metrics";
const llmLatency = new Trend("llm_latency_ms", true);
export const options = {
scenarios: {
// Scenario 1: health check (quick verification)
health: {
executor: "constant-vus",
vus: 1,
duration: "10s",
exec: "healthCheck",
},
// Scenario 2: LLM Chat (sequential β the model handles one request at a time)
chat: {
executor: "constant-vus",
vus: 1,
duration: "2m",
startTime: "15s",
exec: "chatTest",
},
// Scenario 3: RAG query
rag: {
executor: "constant-vus",
vus: 1,
duration: "2m",
startTime: "15s",
exec: "ragTest",
},
},
thresholds: {
"http_req_failed": ["rate<0.05"],
"llm_latency_ms": ["p(95)<60000"], // within 60 seconds for AI
},
};
const BASE = "http://localhost:8000";
const HEADERS = {
"content-type": "application/json",
"x-api-key": __ENV.API_KEY || "change-this-to-a-random-secret-key-32chars",
};
export function healthCheck() {
const res = http.get(`${BASE}/health`, { headers: HEADERS });
check(res, { "health ok": (r) => r.status === 200 });
sleep(1);
}
export function chatTest() {
const start = Date.now();
const res = http.post(
`${BASE}/v1/chat`,
JSON.stringify({ message: "Give me three characteristics of the stock market" }),
{ headers: HEADERS },
);
check(res, {
"chat 200": (r) => r.status === 200,
"has answer": (r) => JSON.parse(r.body)?.answer?.length > 0,
});
llmLatency.add(Date.now() - start);
sleep(2);
}
export function ragTest() {
const start = Date.now();
const res = http.post(
`${BASE}/v1/rag/query`,
JSON.stringify({ query: "What are the key features of AIDevOps?" }),
{ headers: HEADERS },
);
check(res, {
"rag 200": (r) => r.status === 200,
"has answer": (r) => JSON.parse(r.body)?.answer?.length > 0,
});
llmLatency.add(Date.now() - start);
sleep(2);
}# Install k6 (Windows)
choco install k6
# Run the test
k6 run k6/laptop-api-test.js
# Inject the API key through an environment variable
k6 run -e API_KEY=your-key k6/laptop-api-test.js
# Expected result (CPU only):
# llm_latency_ms p(95)=18,420ms β (18s, normal)
# http_req_failed rate=0.00% βOn CPU only (llama3.1:8b), 5-30 seconds is the normal range for an LLM response, depending on request length. Set the k6 threshold generously at 60 seconds. Concurrent requests (vus=2+) can run out of memory, so run sequentially with vus=1.
# β
1. Check that every container is healthy
docker compose ps
# All 3 services must be "running (healthy)"
# β
2. Check the Ollama model list
curl http://localhost:11434/api/tags
# Confirm it includes llama3.1:8b and nomic-embed-text
# β
3. Check Chroma status
curl http://localhost:8001/api/v1/heartbeat
# {"nanosecond heartbeat": ...}
# β
4. Confirm requests without an API key are blocked (401 response)
curl -X POST http://localhost:8000/v1/chat \
-H "content-type: application/json" \
-d '{"message":"test"}'
# β
5. Confirm LLM Chat works
curl -X POST http://localhost:8000/v1/chat \
-H "content-type: application/json" \
-H "x-api-key: change-this-to-a-random-secret-key-32chars" \
-d '{"message":"1+1=?"}'
# β
6. Confirm a RAG query works (after ingesting documents)
curl -X POST http://localhost:8000/v1/rag/query \
-H "content-type: application/json" \
-H "x-api-key: change-this-to-a-random-secret-key-32chars" \
-d '{"query":"What are the key features of AIDevOps?"}'
# β
7. Confirm the Agent runs
curl -X POST http://localhost:8000/v1/agent/run \
-H "content-type: application/json" \
-H "x-api-key: change-this-to-a-random-secret-key-32chars" \
-d '{"task":"Analyze what AAPL and Samsung Electronics have in common"}'| Symptom | Cause | Fix |
|---|---|---|
| Ollama not responding / timing out | Out of memory, model still loading | Check docker logs local-ollama, then switch to llama3.2:3b |
OOM killed | 16GB of RAM is not enough | Lower the Docker Desktop memory allocation, use llama3.2:3b |
| Chroma connection failure | Container not started | docker compose up -d chroma |
| 401 Unauthorized | API key mismatch in .env | Recheck the .env file, docker compose restart api |
| Empty RAG answer | Documents not ingested | Re-run the Step 4 document ingestion |
| Port conflict | Another process is using 8000 | Check with netstat -ano | findstr 8000 |
# Check the logs
docker logs local-agent-api --tail 50
docker logs local-ollama --tail 20
# Restart everything
docker compose down && docker compose up -d
# Open a shell inside the container (debugging)
docker exec -it local-agent-api bash
docker exec -it local-ollama bash