Skip to main content
AIDevOps
  • Learn
  • Learning Paths
  • Practice
  • Open Source
  • Books
  • Engineering

    AI DevOpsThe full map of building and operating AI servicesLLMOpsLLM deployment Β· evaluation Β· observabilityHands-on ProjectsBuild AI Agent projects

    Knowledge

    DocsTechnical documentationBlogEngineering articlesPloggerDevelopment log feed

    Validate

    Certification3-level skills certification Β· coming soon
AI Models
LlamaTutorial|MistralTutorial|GemmaTutorial|DeepSeekTutorial|QwenTutorial
🧠 AI Agent Development
Finance AI AgentLLM API ServerStock Investing AgentAIOps AI AgentEducation AI AgentCoding AI Agent
πŸ€– AI Applied Development
Applied AI Intro & RoadmapHugging FaceLangChainLlamaIndexLLMOps|LangGraphMCPMulti-AgentAgent Evaluation
🧠 AI Core
AI Intro & RoadmapML FundamentalsLLM Fundamentals|Python AIC++|PyTorchTensorFlowJAX
🌱 Spring Cloud
Spring Intro & RoadmapSpring Cloud GatewaySpring BootJava|Spring AISpring SecuritySpring BatchSpring JPA
🐳 DevOps
DevOps Intro & RoadmapLinuxDockerCI/CD|Kubernetes BasicsK8s AdvancedPrometheusGrafana
🧱 Infrastructure
Infrastructure Intro & RoadmapNginxRedis
☁️ Cloud
Cloud Intro & RoadmapAWSGCPAzureNCPCloudflare
🎨 Frontend
Frontend Intro & RoadmapJavaScriptTypeScript|ReactNext.js|VueNuxt|Electron
πŸ“± Mobile
Mobile Intro & RoadmapKotlinAndroidFlutter
βš™οΈ Backend
Backend Intro & RoadmapPython BasicsFastAPIDjangoFlask|CGoGinNode.js
πŸ’Ύ Database
DB Intro & RoadmapCore SQLOracleMySQLPostgreSQL|MongoDBVector DB
πŸ§ͺ Testing
k6JMeternGrinder
AIDevOps

Engineering AI. From Code to Production.
An engineering learning platform for building and operating AI and AI Agents

Learn

  • All Guides
  • Learning Paths
  • AI Tutorial
  • Practice
  • Books

Resources

  • AI DevOps
  • LLMOps
  • Hands-on Projects
  • Docs
  • Blog
  • Plogger
  • Open Source
  • Certification (coming soon)

Start Here

  • AI Core Roadmap
  • AI Applied Development Roadmap
  • Spring Cloud Roadmap
  • DevOps Roadmap
  • Infrastructure Roadmap

Β 

  • Cloud Roadmap
  • Frontend Roadmap
  • Mobile Roadmap
  • Backend Roadmap
  • Database Roadmap
Β© 2026 AIDevOps. All rights reserved.
Terms of ServicePrivacy PolicySitemaptestforge.kr
LLM Infra Β· Laptop API Server

πŸ’» Laptop LLM API Server Private AI Agent Server

Visitors

Stand up an Ollama LLM Engine + FastAPI + Chroma Vector DB + LangGraph Agent on a laptop with a single Docker Compose file. The cloud AI Agent calls only this server, and no paid external APIs are used.

← Projects
πŸ’»

Contents

0 / 16
  1. Architecture & separation of roles
  2. Prerequisites
  3. Project file structure
  4. Step 1 β€” Docker Compose
  5. Step 2 β€” FastAPI server
  6. main.py & auth
  7. LLM Chat endpoint
  8. RAG pipeline
  9. LangGraph Agent
  10. Step 3 β€” Install Ollama models
  11. Step 4 β€” Ingest documents & verify
  12. Step 5 β€” Cloudflare Tunnel
  13. Step 6 β€” Next.js client
  14. Step 7 β€” k6 performance check
  15. Final verification checklist
  16. Troubleshooting
Contents 16 sections
  1. Architecture & separation of roles
  2. Prerequisites
  3. Project file structure
  4. Step 1 β€” Docker Compose
  5. Step 2 β€” FastAPI server
  6. main.py & auth
  7. LLM Chat endpoint
  8. RAG pipeline
  9. LangGraph Agent
  10. Step 3 β€” Install Ollama models
  11. Step 4 β€” Ingest documents & verify
  12. Step 5 β€” Cloudflare Tunnel
  13. Step 6 β€” Next.js client
  14. Step 7 β€” k6 performance check
  15. Final verification checklist
  16. Troubleshooting

Architecture & separation of roles

This project clearly separates the cloud (public touchpoint) from the laptop (private AI engine).

TEXT
Browser / mobile
        β”‚  HTTPS
        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Cloud AI Agent             β”‚  Cloudflare Pages / Next.js
β”‚  - User auth, sessions      β”‚
β”‚  - Rate limit, CORS         β”‚
β”‚  - Audit logs, policy checksβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
           β”‚  x-api-key  (server-side only)
           β”‚  Cloudflare Tunnel or VPN
           β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Laptop β€” Docker Compose    β”‚  i7-12700H Β· 16GB Β· CPU only
β”‚                             β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚ Ollama  β”‚ β”‚  Chroma   β”‚  β”‚
β”‚  β”‚ :11434  β”‚ β”‚  :8001    β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜  β”‚
β”‚       β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜        β”‚
β”‚          β”Œβ”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”     β”‚
β”‚          β”‚  FastAPI   β”‚     β”‚  port 8000 (local binding)
β”‚          β”‚  /v1/chat  β”‚     β”‚
β”‚          β”‚  /v1/rag   β”‚     β”‚
β”‚          β”‚  /v1/agent β”‚     β”‚
β”‚          β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
LayerResponsibilityPublicly exposed
Cloud AI AgentAuth, sessions, routing, policy, audit logsYes (HTTPS)
FastAPI (laptop)LLM calls, RAG, Agent workflowNo (Tunnel only)
Ollama (laptop)LLM inference engineNo (internal network)
Chroma (laptop)Vector DBNo (internal network)

Prerequisites

ItemVersionHow to check
Docker Desktop4.30+docker --version
Python3.11+python --version
Node.js (for cloud integration)20 LTSnode --version
Free RAM8GB or moreTask Manager
Free disk30GB or moreFile Explorer
ℹ️

This guide assumes an i7-12700H (14 cores), 16GB RAM, CPU-only machine with no GPU. Even without a GPU, llama3.1:8b runs at about 5 tok/s.

BASH
# Check that Docker Desktop is installed
docker --version          # Docker version 27.x
docker compose version    # Docker Compose version v2.x

# Check that Docker Desktop is running
docker ps

Project file structure

TEXT
laptop-llm-api-server/
β”œβ”€β”€ docker-compose.yml          # service definitions
β”œβ”€β”€ .env                        # environment variables (git-ignored)
β”œβ”€β”€ .env.example                # example file (committed)
β”‚
β”œβ”€β”€ api/                        # FastAPI server
β”‚   β”œβ”€β”€ Dockerfile
β”‚   β”œβ”€β”€ requirements.txt
β”‚   └── app/
β”‚       β”œβ”€β”€ main.py             # FastAPI app entry point
β”‚       β”œβ”€β”€ auth.py             # API key authentication
β”‚       β”œβ”€β”€ chat.py             # LLM Chat endpoint
β”‚       β”œβ”€β”€ rag.py              # RAG pipeline
β”‚       β”œβ”€β”€ agent.py            # LangGraph Agent
β”‚       └── schemas.py          # Pydantic models
β”‚
└── documents/                  # RAG source documents (git-ignored)
    └── sample.md               # sample document

Step 1 β€” Docker Compose

docker-compose.ymlYAML
services:
  ollama:
    image: ollama/ollama:latest
    container_name: local-ollama
    ports:
      - "127.0.0.1:11434:11434"   # not exposed externally
    volumes:
      - ollama_data:/root/.ollama
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 3

  chroma:
    image: chromadb/chroma:latest
    container_name: local-chroma
    ports:
      - "127.0.0.1:8001:8000"     # not exposed externally
    volumes:
      - chroma_data:/chroma/chroma
    environment:
      IS_PERSISTENT: "TRUE"
      ANONYMIZED_TELEMETRY: "FALSE"
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/api/v1/heartbeat"]
      interval: 30s
      timeout: 5s
      retries: 3

  api:
    build:
      context: ./api
      dockerfile: Dockerfile
    container_name: local-agent-api
    ports:
      - "127.0.0.1:8000:8000"     # not exposed externally
    env_file:
      - .env
    volumes:
      - ./documents:/app/documents:ro
      - ./logs:/app/logs
    depends_on:
      ollama:
        condition: service_healthy
      chroma:
        condition: service_healthy
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      retries: 3

volumes:
  ollama_data:
  chroma_data:
.envBASH
# API authentication
LOCAL_API_KEY=change-this-to-a-random-secret-key-32chars

# Ollama (address on the container network)
OLLAMA_BASE_URL=http://ollama:11434
LLM_MODEL=llama3.1:8b
LLM_MODEL_FAST=llama3.2:3b
MODEL_TIMEOUT_SECONDS=120

# Chroma (address on the container network)
CHROMA_HOST=chroma
CHROMA_PORT=8000
RAG_COLLECTION=private_docs

# Embeddings
EMBEDDING_MODEL=nomic-embed-text

# Logging
LOG_LEVEL=INFO
api/DockerfileDOCKERFILE
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/

EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]
api/requirements.txtTEXT
fastapi==0.115.12
uvicorn[standard]==0.34.3
httpx==0.28.1
pydantic==2.11.5
python-dotenv==1.1.0
chromadb==1.0.12
langchain==0.3.25
langchain-community==0.3.25
langchain-ollama==0.3.3
langgraph==0.4.8
sentence-transformers==3.4.1

Step 2 β€” FastAPI server

main.py & auth

api/app/schemas.pyPYTHON
from pydantic import BaseModel

class ChatRequest(BaseModel):
    message: str
    system: str | None = None
    model: str | None = None

class RagRequest(BaseModel):
    query: str
    collection: str | None = None
    top_k: int = 4

class IngestRequest(BaseModel):
    text: str
    metadata: dict = {}
    collection: str | None = None

class AgentRequest(BaseModel):
    task: str
    context: str | None = None

class StandardResponse(BaseModel):
    answer: str
    model: str = ""
    latency_ms: int = 0
    citations: list[dict] = []
    trace_id: str = ""
api/app/auth.pyPYTHON
import os
import secrets
from fastapi import Header, HTTPException

_API_KEY = os.getenv("LOCAL_API_KEY", "")

def require_api_key(x_api_key: str | None = Header(default=None, alias="x-api-key")) -> None:
    """Validates the API key on every private endpoint."""
    if not _API_KEY:
        raise RuntimeError("The LOCAL_API_KEY environment variable is not set.")
    if not x_api_key or not secrets.compare_digest(x_api_key, _API_KEY):
        raise HTTPException(status_code=401, detail="Invalid API key")
api/app/main.pyPYTHON
import time
import uuid
import logging
from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from .chat import router as chat_router
from .rag import router as rag_router
from .agent import router as agent_router

logging.basicConfig(level=__import__('os').getenv('LOG_LEVEL', 'INFO'))
logger = logging.getLogger(__name__)

@asynccontextmanager
async def lifespan(app: FastAPI):
    logger.info("Local AI Agent API server starting")
    yield
    logger.info("Server shutting down")

app = FastAPI(
    title="Private Local AI Agent API",
    version="1.0.0",
    docs_url="/docs",       # Swagger UI during development
    lifespan=lifespan,
)

app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://localhost:3000"],  # local development only
    allow_methods=["POST", "GET"],
    allow_headers=["x-api-key", "content-type"],
)

app.include_router(chat_router,  prefix="/v1")
app.include_router(rag_router,   prefix="/v1")
app.include_router(agent_router, prefix="/v1")

@app.get("/health")
def health():
    return {"status": "ok", "version": "1.0.0"}

LLM Chat endpoint

api/app/chat.pyPYTHON
import os
import time
import httpx
from fastapi import APIRouter, Depends
from .auth import require_api_key
from .schemas import ChatRequest, StandardResponse

router = APIRouter(tags=["chat"])

OLLAMA_URL   = os.getenv("OLLAMA_BASE_URL", "http://ollama:11434")
DEFAULT_MODEL = os.getenv("LLM_MODEL", "llama3.1:8b")
TIMEOUT       = int(os.getenv("MODEL_TIMEOUT_SECONDS", "120"))

@router.post("/chat", response_model=StandardResponse, dependencies=[Depends(require_api_key)])
async def chat(req: ChatRequest):
    """Simple LLM chat endpoint."""
    model = req.model or DEFAULT_MODEL
    messages = []
    if req.system:
        messages.append({"role": "system", "content": req.system})
    messages.append({"role": "user", "content": req.message})

    start = time.monotonic()
    async with httpx.AsyncClient(timeout=TIMEOUT) as client:
        res = await client.post(
            f"{OLLAMA_URL}/api/chat",
            json={"model": model, "messages": messages, "stream": False},
        )
        res.raise_for_status()

    data    = res.json()
    answer  = data["message"]["content"]
    elapsed = int((time.monotonic() - start) * 1000)

    return StandardResponse(answer=answer, model=model, latency_ms=elapsed)


@router.get("/models", dependencies=[Depends(require_api_key)])
async def list_models():
    """Returns the list of installed Ollama models."""
    async with httpx.AsyncClient(timeout=10) as client:
        res = await client.get(f"{OLLAMA_URL}/api/tags")
        res.raise_for_status()
    return {"models": [m["name"] for m in res.json().get("models", [])]}

RAG pipeline

api/app/rag.pyPYTHON
import os
import time
import httpx
import chromadb
from fastapi import APIRouter, Depends, HTTPException
from langchain.text_splitter import RecursiveCharacterTextSplitter
from .auth import require_api_key
from .schemas import RagRequest, IngestRequest, StandardResponse

router = APIRouter(tags=["rag"])

OLLAMA_URL   = os.getenv("OLLAMA_BASE_URL", "http://ollama:11434")
LLM_MODEL    = os.getenv("LLM_MODEL", "llama3.1:8b")
EMBED_MODEL  = os.getenv("EMBEDDING_MODEL", "nomic-embed-text")
COLLECTION   = os.getenv("RAG_COLLECTION", "private_docs")
TIMEOUT      = int(os.getenv("MODEL_TIMEOUT_SECONDS", "120"))

def _chroma():
    return chromadb.HttpClient(
        host=os.getenv("CHROMA_HOST", "chroma"),
        port=int(os.getenv("CHROMA_PORT", "8000")),
    )

async def _embed(text: str) -> list[float]:
    """Call the Ollama embedding API."""
    async with httpx.AsyncClient(timeout=30) as client:
        res = await client.post(
            f"{OLLAMA_URL}/api/embed",
            json={"model": EMBED_MODEL, "input": text},
        )
        res.raise_for_status()
    return res.json()["embeddings"][0]

async def _llm(prompt: str) -> str:
    """Call the Ollama LLM."""
    async with httpx.AsyncClient(timeout=TIMEOUT) as client:
        res = await client.post(
            f"{OLLAMA_URL}/api/chat",
            json={
                "model": LLM_MODEL,
                "messages": [{"role": "user", "content": prompt}],
                "stream": False,
            },
        )
        res.raise_for_status()
    return res.json()["message"]["content"]

@router.post("/documents/ingest", dependencies=[Depends(require_api_key)])
async def ingest(req: IngestRequest):
    """Chunk a document and store it in Chroma."""
    splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)
    chunks = splitter.split_text(req.text)
    if not chunks:
        raise HTTPException(400, "The document is empty.")

    collection_name = req.collection or COLLECTION
    db = _chroma()
    col = db.get_or_create_collection(collection_name)

    embeddings = [await _embed(chunk) for chunk in chunks]
    ids = [f"doc-{i}-{hash(chunk) & 0xFFFFFF:06x}" for i, chunk in enumerate(chunks)]
    metadatas = [{**req.metadata, "chunk_index": i} for i in range(len(chunks))]

    col.add(documents=chunks, embeddings=embeddings, ids=ids, metadatas=metadatas)
    return {"ingested_chunks": len(chunks), "collection": collection_name}

@router.post("/rag/query", response_model=StandardResponse, dependencies=[Depends(require_api_key)])
async def rag_query(req: RagRequest):
    """Chroma retrieval + LLM answer generation."""
    collection_name = req.collection or COLLECTION
    db = _chroma()

    try:
        col = db.get_collection(collection_name)
    except Exception:
        raise HTTPException(404, f"Collection '{collection_name}' does not exist. Ingest documents first.")

    query_vec = await _embed(req.query)
    results = col.query(query_embeddings=[query_vec], n_results=req.top_k)

    docs = results["documents"][0]
    metas = results["metadatas"][0]

    if not docs:
        return StandardResponse(answer="No relevant documents found.")

    context = "\n\n---\n\n".join(docs)
    prompt = f"""Answer the question using the documents below.
If the documents do not contain the answer, reply "I could not find relevant content in the documents".

[Reference documents]
{context}

[Question]
{req.query}

[Answer]"""

    start = time.monotonic()
    answer = await _llm(prompt)
    elapsed = int((time.monotonic() - start) * 1000)

    citations = [{"text": d[:120] + "...", "metadata": m} for d, m in zip(docs, metas)]
    return StandardResponse(answer=answer, model=LLM_MODEL, latency_ms=elapsed, citations=citations)

LangGraph Agent

api/app/agent.pyPYTHON
import os
import time
from typing import TypedDict, Annotated
from fastapi import APIRouter, Depends
from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage, AIMessage, BaseMessage
from langgraph.graph import StateGraph, END
from langgraph.graph.message import add_messages
from .auth import require_api_key
from .schemas import AgentRequest, StandardResponse

router = APIRouter(tags=["agent"])

OLLAMA_URL = os.getenv("OLLAMA_BASE_URL", "http://ollama:11434")
LLM_MODEL  = os.getenv("LLM_MODEL", "llama3.1:8b")

# ── State definition ──
class AgentState(TypedDict):
    messages: Annotated[list[BaseMessage], add_messages]
    task: str
    result: str

# ── Node functions ──
def _plan_node(state: AgentState) -> dict:
    """Analyze the task and build an execution plan."""
    llm = ChatOllama(model=LLM_MODEL, base_url=OLLAMA_URL, temperature=0)
    plan_prompt = f"""Analyze the following task and plan the execution in at most 3 steps.
Task: {state['task']}
Plan (as a numbered list):"""
    response = llm.invoke([HumanMessage(content=plan_prompt)])
    return {"messages": [response]}

def _execute_node(state: AgentState) -> dict:
    """Execute the plan and produce the result."""
    llm = ChatOllama(model=LLM_MODEL, base_url=OLLAMA_URL, temperature=0)
    plan = state["messages"][-1].content if state["messages"] else ""
    exec_prompt = f"""Produce the final result based on the task and the plan.
Task: {state['task']}
Plan: {plan}
Context: {state.get('context', '')}
Result:"""
    response = llm.invoke([HumanMessage(content=exec_prompt)])
    return {"messages": [response], "result": response.content}

def _build_graph():
    builder = StateGraph(AgentState)
    builder.add_node("plan",    _plan_node)
    builder.add_node("execute", _execute_node)
    builder.set_entry_point("plan")
    builder.add_edge("plan", "execute")
    builder.add_edge("execute", END)
    return builder.compile()

_graph = _build_graph()

@router.post("/agent/run", response_model=StandardResponse, dependencies=[Depends(require_api_key)])
async def run_agent(req: AgentRequest):
    """Run the LangGraph Agent."""
    start = time.monotonic()
    result = _graph.invoke({
        "messages": [],
        "task": req.task,
        "context": req.context or "",
        "result": "",
    })
    elapsed = int((time.monotonic() - start) * 1000)
    return StandardResponse(answer=result["result"], model=LLM_MODEL, latency_ms=elapsed)

Step 3 β€” Install Ollama models

Start Chroma and Ollama first, then download the models.

BASH
# 1. Start Chroma + Ollama first
docker compose up -d ollama chroma

# 2. Check container status (wait until healthy)
docker compose ps

# 3. Download the main model (~5GB, takes 5-10 minutes)
docker exec -it local-ollama ollama pull llama3.1:8b

# 4. Download the embedding model (~300MB)
docker exec -it local-ollama ollama pull nomic-embed-text

# 5. (Optional) lightweight model for fast responses
docker exec -it local-ollama ollama pull llama3.2:3b

# 6. Verify the install
docker exec -it local-ollama ollama list
BASH
# Test the Ollama API directly
curl http://localhost:11434/api/tags

# Sample response:
# {"models":[
#   {"name":"llama3.1:8b","size":4661224960},
#   {"name":"nomic-embed-text","size":274302480}
# ]}
πŸ’‘

Memory management: on 16GB of RAM, Docker Desktop needs 8GB and Ollama (llama3.1:8b) needs 5-7GB. Keep the IDE and browser tabs to a minimum. If that is too tight, switch to llama3.2:3b (~2GB).

Step 4 β€” Start the API server & ingest documents

BASH
# Start the API server
docker compose up -d api

# Check the status of all services
docker compose ps

# EXPECTED:
# local-ollama     running (healthy)
# local-chroma     running (healthy)
# local-agent-api  running (healthy)

# Health check
curl http://localhost:8000/health
# {"status":"ok","version":"1.0.0"}

# Request without an API key β†’ expect 401
curl -X POST http://localhost:8000/v1/chat \
  -H "content-type: application/json" \
  -d '{"message":"test"}'
# {"detail":"Invalid API key"}

# Request with an API key β†’ OK
curl -X POST http://localhost:8000/v1/chat \
  -H "content-type: application/json" \
  -H "x-api-key: change-this-to-a-random-secret-key-32chars" \
  -d '{"message":"Hello. Please introduce yourself briefly."}'
documents/sample.mdMARKDOWN
# About the company

## Service overview
AIDevOps is a technology platform for building and operating AI Agents.
Main services: Finance AI Agent, system performance optimization, LLMOps guides

## Key features
- Real-time market data analysis
- RAG search over filings
- Automated portfolio rebalancing

## Tech stack
Python, FastAPI, LangChain, LangGraph, ChromaDB, Ollama
BASH
# Test document ingestion
curl -X POST http://localhost:8000/v1/documents/ingest \
  -H "content-type: application/json" \
  -H "x-api-key: change-this-to-a-random-secret-key-32chars" \
  -d '{
    "text": "AIDevOps is an AI Agent development platform. Its key features are financial analysis, RAG search, and LangGraph orchestration.",
    "metadata": {"source": "sample", "date": "2026-06-06"}
  }'
# {"ingested_chunks":1,"collection":"private_docs"}

# Test a RAG query
curl -X POST http://localhost:8000/v1/rag/query \
  -H "content-type: application/json" \
  -H "x-api-key: change-this-to-a-random-secret-key-32chars" \
  -d '{"query": "What are the key features of AIDevOps?"}'
BASH
# Bulk-ingest every document in a folder (Python script)
python - <<'EOF'
import httpx, pathlib, os

API = "http://localhost:8000"
KEY = os.getenv("LOCAL_API_KEY", "change-this-to-a-random-secret-key-32chars")

for path in pathlib.Path("documents").rglob("*.md"):
    text = path.read_text(encoding="utf-8")
    r = httpx.post(
        f"{API}/v1/documents/ingest",
        headers={"x-api-key": KEY},
        json={"text": text, "metadata": {"filename": path.name}},
    )
    print(f"{path.name}: {r.json()}")
EOF

Step 5 β€” Cloudflare Tunnel (cloud integration)

The cloud AI Agent needs a secure tunnel to call FastAPI on the laptop.

⚠️

Never do this: do not expose the FastAPI port (8000) directly to the internet. Pair Cloudflare Tunnel with a Cloudflare Access policy so that only server-side clients, not browsers, can reach it.

BASH
# 1. Install cloudflared (Windows)
winget install --id Cloudflare.cloudflared

# 2. Log in to Cloudflare
cloudflared tunnel login

# 3. Create the tunnel
cloudflared tunnel create laptop-ai-agent

# 4. Create the config file
# C:Users[user].cloudflaredconfig.yml

# 5. Temporary test tunnel (works right away, no domain needed)
cloudflared tunnel --url http://localhost:8000
~/.cloudflared/config.ymlYAML
tunnel: laptop-ai-agent
credentials-file: C:\Users\[user]\.cloudflared\[tunnel-id].json

ingress:
  - hostname: laptop-api.your-domain.com
    service: http://localhost:8000
    originRequest:
      noTLSVerify: false
  - service: http_status:404
BASH
# Run the tunnel
cloudflared tunnel run laptop-ai-agent

# An Access Policy must be set in the Cloudflare Dashboard:
# - Service Auth: create a Service Token
# - Allow only the cloud server IP (Bypass policy)

# Test the connection
curl https://laptop-api.your-domain.com/health \
  -H "CF-Access-Client-Id: your-service-token-id" \
  -H "CF-Access-Client-Secret: your-service-token-secret"

Step 6 β€” Next.js / TypeScript client

Server-side code for calling the laptop FastAPI from the cloud AI Agent (Next.js). It is never called directly from the browser.

lib/laptop-agent.tsTYPESCRIPT
const LAPTOP_API_URL = process.env.LAPTOP_AGENT_API_URL!;
const LAPTOP_API_KEY = process.env.LAPTOP_AGENT_API_KEY!;

interface AgentResponse {
  answer: string;
  model: string;
  latency_ms: number;
  citations: { text: string; metadata: Record<string, unknown> }[];
}

async function callLaptopApi(path: string, body: unknown): Promise<AgentResponse> {
  const url = LAPTOP_API_URL + path;
  const res = await fetch(url, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "x-api-key": LAPTOP_API_KEY,
    },
    body: JSON.stringify(body),
    signal: AbortSignal.timeout(130_000),
  });

  if (!res.ok) {
    const err = await res.text();
    throw new Error("Laptop API " + res.status + ": " + err);
  }
  return res.json();
}

export async function chat(message: string, system?: string): Promise<AgentResponse> {
  return callLaptopApi("/v1/chat", { message, system });
}

export async function ragQuery(query: string, collection?: string): Promise<AgentResponse> {
  return callLaptopApi("/v1/rag/query", { query, collection });
}

export async function runAgent(task: string, context?: string): Promise<AgentResponse> {
  return callLaptopApi("/v1/agent/run", { task, context });
}
app/api/ask/route.tsTYPESCRIPT
import { NextRequest, NextResponse } from "next/server";
import { ragQuery } from "@/lib/laptop-agent";
import { getServerSession } from "next-auth";   // auth middleware

export async function POST(req: NextRequest) {
  const session = await getServerSession();
  if (!session) {
    return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
  }

  const { question } = await req.json();
  if (!question?.trim()) {
    return NextResponse.json({ error: "question is required" }, { status: 400 });
  }

  try {
    const result = await ragQuery(question);
    return NextResponse.json(result);
  } catch (err) {
    const msg = err instanceof Error ? err.message : "Unknown error";
    return NextResponse.json({ error: msg }, { status: 502 });
  }
}
.env.local (cloud server)BASH
# Cloud server environment variables (never expose to the browser)
LAPTOP_AGENT_API_URL=https://laptop-api.your-domain.com
LAPTOP_AGENT_API_KEY=change-this-to-a-random-secret-key-32chars

Step 7 β€” k6 performance check

k6/laptop-api-test.jsJAVASCRIPT
import http from "k6/http";
import { check, sleep } from "k6";
import { Trend } from "k6/metrics";

const llmLatency = new Trend("llm_latency_ms", true);

export const options = {
  scenarios: {
    // Scenario 1: health check (quick verification)
    health: {
      executor: "constant-vus",
      vus: 1,
      duration: "10s",
      exec: "healthCheck",
    },
    // Scenario 2: LLM Chat (sequential β€” the model handles one request at a time)
    chat: {
      executor: "constant-vus",
      vus: 1,
      duration: "2m",
      startTime: "15s",
      exec: "chatTest",
    },
    // Scenario 3: RAG query
    rag: {
      executor: "constant-vus",
      vus: 1,
      duration: "2m",
      startTime: "15s",
      exec: "ragTest",
    },
  },
  thresholds: {
    "http_req_failed":  ["rate<0.05"],
    "llm_latency_ms":   ["p(95)<60000"],   // within 60 seconds for AI
  },
};

const BASE  = "http://localhost:8000";
const HEADERS = {
  "content-type": "application/json",
  "x-api-key": __ENV.API_KEY || "change-this-to-a-random-secret-key-32chars",
};

export function healthCheck() {
  const res = http.get(`${BASE}/health`, { headers: HEADERS });
  check(res, { "health ok": (r) => r.status === 200 });
  sleep(1);
}

export function chatTest() {
  const start = Date.now();
  const res = http.post(
    `${BASE}/v1/chat`,
    JSON.stringify({ message: "Give me three characteristics of the stock market" }),
    { headers: HEADERS },
  );
  check(res, {
    "chat 200":    (r) => r.status === 200,
    "has answer":  (r) => JSON.parse(r.body)?.answer?.length > 0,
  });
  llmLatency.add(Date.now() - start);
  sleep(2);
}

export function ragTest() {
  const start = Date.now();
  const res = http.post(
    `${BASE}/v1/rag/query`,
    JSON.stringify({ query: "What are the key features of AIDevOps?" }),
    { headers: HEADERS },
  );
  check(res, {
    "rag 200":     (r) => r.status === 200,
    "has answer":  (r) => JSON.parse(r.body)?.answer?.length > 0,
  });
  llmLatency.add(Date.now() - start);
  sleep(2);
}
BASH
# Install k6 (Windows)
choco install k6

# Run the test
k6 run k6/laptop-api-test.js

# Inject the API key through an environment variable
k6 run -e API_KEY=your-key k6/laptop-api-test.js

# Expected result (CPU only):
# llm_latency_ms  p(95)=18,420ms  βœ“ (18s, normal)
# http_req_failed rate=0.00%      βœ“
ℹ️

On CPU only (llama3.1:8b), 5-30 seconds is the normal range for an LLM response, depending on request length. Set the k6 threshold generously at 60 seconds. Concurrent requests (vus=2+) can run out of memory, so run sequentially with vus=1.

Final verification checklist

BASH
# βœ… 1. Check that every container is healthy
docker compose ps
# All 3 services must be "running (healthy)"

# βœ… 2. Check the Ollama model list
curl http://localhost:11434/api/tags
# Confirm it includes llama3.1:8b and nomic-embed-text

# βœ… 3. Check Chroma status
curl http://localhost:8001/api/v1/heartbeat
# {"nanosecond heartbeat": ...}

# βœ… 4. Confirm requests without an API key are blocked (401 response)
curl -X POST http://localhost:8000/v1/chat \
  -H "content-type: application/json" \
  -d '{"message":"test"}'

# βœ… 5. Confirm LLM Chat works
curl -X POST http://localhost:8000/v1/chat \
  -H "content-type: application/json" \
  -H "x-api-key: change-this-to-a-random-secret-key-32chars" \
  -d '{"message":"1+1=?"}'

# βœ… 6. Confirm a RAG query works (after ingesting documents)
curl -X POST http://localhost:8000/v1/rag/query \
  -H "content-type: application/json" \
  -H "x-api-key: change-this-to-a-random-secret-key-32chars" \
  -d '{"query":"What are the key features of AIDevOps?"}'

# βœ… 7. Confirm the Agent runs
curl -X POST http://localhost:8000/v1/agent/run \
  -H "content-type: application/json" \
  -H "x-api-key: change-this-to-a-random-secret-key-32chars" \
  -d '{"task":"Analyze what AAPL and Samsung Electronics have in common"}'

Troubleshooting

SymptomCauseFix
Ollama not responding / timing outOut of memory, model still loadingCheck docker logs local-ollama, then switch to llama3.2:3b
OOM killed16GB of RAM is not enoughLower the Docker Desktop memory allocation, use llama3.2:3b
Chroma connection failureContainer not starteddocker compose up -d chroma
401 UnauthorizedAPI key mismatch in .envRecheck the .env file, docker compose restart api
Empty RAG answerDocuments not ingestedRe-run the Step 4 document ingestion
Port conflictAnother process is using 8000Check with netstat -ano | findstr 8000
BASH
# Check the logs
docker logs local-agent-api --tail 50
docker logs local-ollama --tail 20

# Restart everything
docker compose down && docker compose up -d

# Open a shell inside the container (debugging)
docker exec -it local-agent-api bash
docker exec -it local-ollama bash