Contents 15 sections
- Technical guide
- Development environment
- AI model selection
- Model install and run
- Training and tuning
- Vector DB setup
- RAG development
- System integration
- Validation and operations
- Model selection by domain
- Overall architecture
- Project ABC
- Data and tools
- Milestones/deliverables
- Tech stack
Technical Blueprint
Hands-on AI Agent Development Checklist
Executable development documents come before explanations of the idea. An AI Agent only moves from demo to real service once local execution, the API contract, RAG data, tool calls, the validation routine, and the operational fallback are fixed first.
1. Development environment and run baseline
- Put `npm install`, `docker compose up`, `npm run build`, and the health check command in order at the top of the README.
- Fix the default development environment: Node.js 22 LTS, Python 3.11 or later, Git, Docker Desktop, and VS Code or Cursor.
- Start with Next.js for the frontend, FastAPI or a Cloudflare Worker for the Agent API, and either Chroma or pgvector for the Vector DB.
- In `.env.example`, keep the model name, API key, DB URL, vector collection, timeout, and observability endpoint under their real variable names.
- From the first commit, treat the build, API unit tests, representative curl calls, and a k6 smoke test as the pass criteria.
2. Model and cost criteria
- First classify the task as one of: conversation, document Q&A, code generation, data analysis, or tool calling.
- Compare candidate production models not on quality alone but also on monthly cost, p95 latency, JSON schema compliance rate, and how well they can fall back on failure.
- For local-first projects, write down the actual model names to install and their expected sizes, such as `llama3.1:8b`, `llama3.2:3b`, `gemma2:9b`, and `nomic-embed-text`.
- When also using API models, add a provider adapter so you can switch through `MODEL_PROVIDER`, `MODEL_NAME`, and `MODEL_TIMEOUT_SECONDS`.
- Evaluate models against a golden set of 30-50 representative questions and tabulate accuracy, groundedness, hallucination rate, latency, and failure types.
3. API contract and local execution
- Four endpoints are enough to fix at first: `/health`, `/v1/chat`, `/v1/rag/query`, and `/v1/agent/run`.
- Every response returns `answer`, `citations`, `model`, `latency_ms`, `trace_id`, and `warnings` in the same shape.
- Design it so the browser never calls the model provider or the laptop API directly, only the server-side API facade.
- Run local models with Ollama or Docker Compose, and include the model list call and a first chat completion curl in the docs.
- Do not hard-code timeout, retry, max tokens, temperature, or streaming; move them into environment variables.
4. Prompts and response policy
- Keep the system prompt short and fixed: the role, forbidden behavior, how to answer when evidence is thin, and the conditions for using tools.
- Early on, improve the prompt template, RAG document quality, tool schema, and JSON response format before reaching for fine-tuning.
- Include examples of forbidden answers, insufficient-evidence answers, and answers that need user confirmation in the golden set.
- Manage per-task output not as free text but as a schema such as `summary`, `evidence`, `next_actions`, `confidence`, and `needs_review`.
- When a prompt changes, compare the golden set pass rate and failure types against the previous version.
5. Building the RAG data
- Store the source documents, chunks, embeddings, metadata, and ingestion version separately, and put the re-index command in the README.
- Start with 10 documents and inspect the chunk size, overlap, metadata, and search results by eye.
- The search API takes explicit `query`, `top_k`, `filters`, and `min_score` and returns the source id, title, page, score, and chunk text.
- Make citations mandatory in RAG answers, and fall back to an “insufficient evidence” response when there is no citation.
- Decide in advance how to detect document changes, index incrementally, tombstone deleted documents, and re-index when the embedding version changes.
6. Implementing the Agent workflow
- Split the workflow into intent classification, deciding whether retrieval is needed, tool calls, answer generation, verification, and fallback, in that order.
- If you use LangGraph, document the input/output schema of each node and the next node on failure.
- Start the tools the Agent can call as read-only, and put write actions behind a human approval step.
- Attach a timeout, retry, rate limit, audit log, and user permission check to every tool call by default.
- Return the final answer with evidence, next actions, and whether review is needed as separate fields so the UI can render it as-is.
7. System integration and security
- The Agent does not query the DB directly; it calls only restricted tool APIs such as `stock_search`, `portfolio_summary`, `log_search`, and `policy_check`.
- Give external APIs and internal DB adapters permissions, rate limits, timeouts, retries, and audit logs by default.
- Clearly divide where the API key, JWT, session id, user id, and request id are handled between the cloud and the local API.
- Do not leave sensitive data as-is in prompts and logs; mask it or move it to a separate secure store.
- Link the user request, tool calls, model response, and final answer by trace id and use it for incident analysis and quality improvement.
8. Validation and moving to production
- Measure correctness, groundedness, policy violations, and output format compliance with the golden set, and compare results before and after each change.
- The minimum bar before production is the health check, smoke test, k6 p95 latency, API error rate, RAG hit rate, and fallback rate.
- In high-risk areas such as finance, education, and operations automation, always separate LLM judgment from deterministic rule checks.
- Observe latency, token usage, tool failures, and retrieval hit rate with OpenTelemetry, Prometheus, Grafana, or at least structured logs.
- Start the first release as a read-only Agent with a human-approval workflow, and widen the scope of automatic execution after enough validation.
Model Selection
AI model selection and install/usage by domain
So that you can start a POC right away on a AI Ops project, this section lays out the recommended API models, local models, embedding models, and install/usage guidelines.
Recommended API models
GPT-4.1 mini or Gemini 2.5 Flash
Local/on-premises alternatives
Llama 3.3 70B Instruct or the DeepSeek/Qwen coder family
Embedding models
text-embedding-3-large or bge-m3
Chosen models and Docker install baseline
Choose Qwen3-32B-Instruct as the default local model for the AIOps AI Agent. It handles log/metric/runbook summarization and the tool calling flow in a balanced way, with a coder model attached as a helper when needed.
Start by standing up an OpenAI-compatible endpoint with vLLM Docker, and connect the Prometheus/Loki/GitHub Actions adapters with read-only tokens only.
Recommended server spec
- Development/demo: GPU with 24GB VRAM, 64GB RAM, NVMe SSD. Suitable for single-user diagnosis including recent logs and runbook search.
- Production POC: one L40S 48GB or A100 80GB or better. Plan for burst requests and long log summaries during incidents.
- Do not pile log ingestion and model inference onto one server; split them into at least 2 containers.
Capacity and storage
- About 18-24GB for the Qwen3 32B Q4 model, 65GB or more for the FP16 cache, and at least a 150GB SSD recommended including log samples and the vector index.
- Keep the Prometheus/Loki source data in the external systems and store only incident snapshots, runbook chunks, and evaluation results in the AI Agent DB.
- Give logs kept for incident reproduction a 7-30 day retention policy, and mask sensitive tokens before ingestion.
Docker install commands
- vLLM: `docker run --runtime nvidia --gpus all -v ~/.cache/huggingface:/root/.cache/huggingface -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model Qwen/Qwen3-32B --max-model-len 32768`
- Quick local check: `docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama`, then `docker exec -it ollama ollama pull qwen3:32b`
- App settings: `LLM_BASE_URL=http://localhost:8000/v1`, `LLM_MODEL=Qwen/Qwen3-32B`, `MODEL_TIMEOUT_SECONDS=30`
Post-install verification
- Measure the match rate between candidate causes and evidence logs over 10 incident alerts, 10 log bundles, and 10 runbooks.
- Confirm that write actions such as restart, delete, rollback, and scale changes are exposed only through the approval workflow, never through the model answer.
- Track p95 response time, tool failure rate, and the rate of hallucinated service names as observability metrics.
Why these were chosen
- An AI Ops Agent has to bring together logs, metrics, deployment history, and runbooks, so tool calling reliability and long-context summarization matter.
- When code or stack traces need interpreting, attaching a coder-family local model as a helper improves the quality of the candidate-cause explanations.
- Before automatic recovery, it is safer to start with read-only diagnosis and a suggestion mode based on human approval.
Install and configuration
- For the API setup, split cause summaries from short alert classification with `LLM_MODEL=gpt-4.1-mini` and `FAST_MODEL=gpt-4.1-nano`.
- For the local setup, put the vLLM/Ollama endpoint behind a FastAPI adapter and connect the Prometheus, Loki, and deployment history APIs with read-only tokens.
- Store runbooks in the Vector DB with title, service name, severity, symptoms, recovery steps, and forbidden-command metadata.
How to use them
- Call `metric_query`, `log_search`, `deploy_lookup`, and `runbook_retriever` in order to gather candidate causes and evidence.
- Fix the model output to `incident_summary`, `likely_causes`, `evidence`, `recommended_runbook`, and `requires_human_approval`.
- Leave recovery command execution out of the AI Agent scope and build the approval UI and audit log first.
Cautions
- Never let the model directly run commands such as delete, restart, scale changes, or deployment rollback.
- Add a fallback answer so that, without runbook evidence, it does not generate speculative recovery procedures.
Reference Architecture
Overall architecture design
An AI Agent is easier to change and validate when the UI, orchestration, tool integration, data retrieval, and evaluation/operations layers are separated.
Presentation Layer
- Next.js UI
- Agent run status
- Evidence/source display
- User feedback collection
Agent Orchestration
- Prompt policy
- Tool routing
- Memory/session
- Structured output
Tool & Integration
- Business API adapter
- DB query tool
- Document search tool
- Policy check tool
Data & Retrieval
- PostgreSQL
- pgvector
- Document chunking
- embedding/reranking
Evaluation & Ops
- Golden set
- Playwright
- k6
- OpenTelemetry/Prometheus
Project ABC
The ABCs of running the project for real
A is architecture design, B is the AI Agent implementation, and C is the validation criteria. Following this order makes it easy to grow from a demo into an Agent you can operate.
A. Architecture
- Start with metric queries, log search, deployment history lookup, and runbook search as read-only tools.
- Leave recovery command execution out of the AI Agent and design it as an approval-based suggestion mode.
- Split the Agent state into alert intake, candidate causes, evidence gathering, runbook recommendation, and human approval.
B. Build
- Create 10 sample incidents and 10 response runbooks and define how the metrics, logs, and runbooks connect.
- Make the time range, service name, and severity required parameters for Prometheus queries and log search.
- Lay out the operations UI in this order: alert card, evidence logs, metric snapshot, recommended runbook.
C. Check
- Test wrong service names, excessive time ranges, and unauthorized tool calls as failure cases.
- Confirm that it does not invent speculative recovery procedures for incidents with no runbook.
- Treat MTTR, candidate-cause accuracy, runbook recommendation accuracy, and p95 response time as the core metrics.
Project Spec
Data and Agent tools
Data
- Prometheus metrics
- Application logs
- Deployment history
- Incident response runbooks
Agent Tools
metric_querylog_searchdeploy_lookuprunbook_retriever
AI Agent scope
- List candidate causes of the alert
- Check the impact of recent deployments
- Summarize related logs
- Suggest a recovery procedure
Validation criteria
- Blocking of wrong command execution
- Runbook evidence included
- MTTR reduction
- Tool restrictions by permission
Execution
Milestones and deliverables
Milestones
- Design incident samples
- Implement the metric/log tools
- Connect runbook search
- Approval-based UI
- Operational safety validation
Deliverables
- AIOps Agent API
- Sample runbook set
- Prometheus query templates
- Operations dashboard
- Safety policy tests
Stack
Tech stack
FastAPIPrometheusGrafanaOpenTelemetryLangGraphk6