Skip to main content
AIDevOps
  • Learn
  • Learning Paths
  • Practice
  • Open Source
  • Books
  • Engineering

    AI DevOpsThe full map of building and operating AI servicesLLMOpsLLM deployment · evaluation · observabilityHands-on ProjectsBuild AI Agent projects

    Knowledge

    DocsTechnical documentationBlogEngineering articlesPloggerDevelopment log feed

    Validate

    Certification3-level skills certification · coming soon
AI Models
LlamaTutorial|MistralTutorial|GemmaTutorial|DeepSeekTutorial|QwenTutorial
🧠 AI Agent Development
Finance AI AgentLLM API ServerStock Investing AgentAIOps AI AgentEducation AI AgentCoding AI Agent
🤖 AI Applied Development
Applied AI Intro & RoadmapHugging FaceLangChainLlamaIndexLLMOps|LangGraphMCPMulti-AgentAgent Evaluation
🧠 AI Core
AI Intro & RoadmapML FundamentalsLLM Fundamentals|Python AIC++|PyTorchTensorFlowJAX
🌱 Spring Cloud
Spring Intro & RoadmapSpring Cloud GatewaySpring BootJava|Spring AISpring SecuritySpring BatchSpring JPA
🐳 DevOps
DevOps Intro & RoadmapLinuxDockerCI/CD|Kubernetes BasicsK8s AdvancedPrometheusGrafana
🧱 Infrastructure
Infrastructure Intro & RoadmapNginxRedis
☁️ Cloud
Cloud Intro & RoadmapAWSGCPAzureNCPCloudflare
🎨 Frontend
Frontend Intro & RoadmapJavaScriptTypeScript|ReactNext.js|VueNuxt|Electron
📱 Mobile
Mobile Intro & RoadmapKotlinAndroidFlutter
⚙️ Backend
Backend Intro & RoadmapPython BasicsFastAPIDjangoFlask|CGoGinNode.js
💾 Database
DB Intro & RoadmapCore SQLOracleMySQLPostgreSQL|MongoDBVector DB
🧪 Testing
k6JMeternGrinder
AIDevOps

Engineering AI. From Code to Production.
An engineering learning platform for building and operating AI and AI Agents

Learn

  • All Guides
  • Learning Paths
  • AI Tutorial
  • Practice
  • Books

Resources

  • AI DevOps
  • LLMOps
  • Hands-on Projects
  • Docs
  • Blog
  • Plogger
  • Open Source
  • Certification (coming soon)

Start Here

  • AI Core Roadmap
  • AI Applied Development Roadmap
  • Spring Cloud Roadmap
  • DevOps Roadmap
  • Infrastructure Roadmap

 

  • Cloud Roadmap
  • Frontend Roadmap
  • Mobile Roadmap
  • Backend Roadmap
  • Database Roadmap
© 2026 AIDevOps. All rights reserved.
Terms of ServicePrivacy PolicySitemaptestforge.kr
AIOps AI Agent

OPS AIOps Runbook Agent AI Agent Project

Visitors

An incident response Agent that connects alerts, logs, metrics, and runbooks. When an operations alert fires, summarizes the related logs and metrics and presents a safe recovery procedure and the points the owner needs to confirm.

← Back to projects
OPS

Contents

0 / 15
  1. Technical guide
  2. Development environment
  3. AI model selection
  4. Model install and run
  5. Training and tuning
  6. Vector DB setup
  7. RAG development
  8. System integration
  9. Validation and operations
  10. Model selection by domain
  11. Overall architecture
  12. Project ABC
  13. Data and tools
  14. Milestones/deliverables
  15. Tech stack
Contents 15 sections
  1. Technical guide
  2. Development environment
  3. AI model selection
  4. Model install and run
  5. Training and tuning
  6. Vector DB setup
  7. RAG development
  8. System integration
  9. Validation and operations
  10. Model selection by domain
  11. Overall architecture
  12. Project ABC
  13. Data and tools
  14. Milestones/deliverables
  15. Tech stack

Technical Blueprint

Hands-on AI Agent Development Checklist

Executable development documents come before explanations of the idea. An AI Agent only moves from demo to real service once local execution, the API contract, RAG data, tool calls, the validation routine, and the operational fallback are fixed first.

1. Development environment and run baseline

  • Put `npm install`, `docker compose up`, `npm run build`, and the health check command in order at the top of the README.
  • Fix the default development environment: Node.js 22 LTS, Python 3.11 or later, Git, Docker Desktop, and VS Code or Cursor.
  • Start with Next.js for the frontend, FastAPI or a Cloudflare Worker for the Agent API, and either Chroma or pgvector for the Vector DB.
  • In `.env.example`, keep the model name, API key, DB URL, vector collection, timeout, and observability endpoint under their real variable names.
  • From the first commit, treat the build, API unit tests, representative curl calls, and a k6 smoke test as the pass criteria.

2. Model and cost criteria

  • First classify the task as one of: conversation, document Q&A, code generation, data analysis, or tool calling.
  • Compare candidate production models not on quality alone but also on monthly cost, p95 latency, JSON schema compliance rate, and how well they can fall back on failure.
  • For local-first projects, write down the actual model names to install and their expected sizes, such as `llama3.1:8b`, `llama3.2:3b`, `gemma2:9b`, and `nomic-embed-text`.
  • When also using API models, add a provider adapter so you can switch through `MODEL_PROVIDER`, `MODEL_NAME`, and `MODEL_TIMEOUT_SECONDS`.
  • Evaluate models against a golden set of 30-50 representative questions and tabulate accuracy, groundedness, hallucination rate, latency, and failure types.

3. API contract and local execution

  • Four endpoints are enough to fix at first: `/health`, `/v1/chat`, `/v1/rag/query`, and `/v1/agent/run`.
  • Every response returns `answer`, `citations`, `model`, `latency_ms`, `trace_id`, and `warnings` in the same shape.
  • Design it so the browser never calls the model provider or the laptop API directly, only the server-side API facade.
  • Run local models with Ollama or Docker Compose, and include the model list call and a first chat completion curl in the docs.
  • Do not hard-code timeout, retry, max tokens, temperature, or streaming; move them into environment variables.

4. Prompts and response policy

  • Keep the system prompt short and fixed: the role, forbidden behavior, how to answer when evidence is thin, and the conditions for using tools.
  • Early on, improve the prompt template, RAG document quality, tool schema, and JSON response format before reaching for fine-tuning.
  • Include examples of forbidden answers, insufficient-evidence answers, and answers that need user confirmation in the golden set.
  • Manage per-task output not as free text but as a schema such as `summary`, `evidence`, `next_actions`, `confidence`, and `needs_review`.
  • When a prompt changes, compare the golden set pass rate and failure types against the previous version.

5. Building the RAG data

  • Store the source documents, chunks, embeddings, metadata, and ingestion version separately, and put the re-index command in the README.
  • Start with 10 documents and inspect the chunk size, overlap, metadata, and search results by eye.
  • The search API takes explicit `query`, `top_k`, `filters`, and `min_score` and returns the source id, title, page, score, and chunk text.
  • Make citations mandatory in RAG answers, and fall back to an “insufficient evidence” response when there is no citation.
  • Decide in advance how to detect document changes, index incrementally, tombstone deleted documents, and re-index when the embedding version changes.

6. Implementing the Agent workflow

  • Split the workflow into intent classification, deciding whether retrieval is needed, tool calls, answer generation, verification, and fallback, in that order.
  • If you use LangGraph, document the input/output schema of each node and the next node on failure.
  • Start the tools the Agent can call as read-only, and put write actions behind a human approval step.
  • Attach a timeout, retry, rate limit, audit log, and user permission check to every tool call by default.
  • Return the final answer with evidence, next actions, and whether review is needed as separate fields so the UI can render it as-is.

7. System integration and security

  • The Agent does not query the DB directly; it calls only restricted tool APIs such as `stock_search`, `portfolio_summary`, `log_search`, and `policy_check`.
  • Give external APIs and internal DB adapters permissions, rate limits, timeouts, retries, and audit logs by default.
  • Clearly divide where the API key, JWT, session id, user id, and request id are handled between the cloud and the local API.
  • Do not leave sensitive data as-is in prompts and logs; mask it or move it to a separate secure store.
  • Link the user request, tool calls, model response, and final answer by trace id and use it for incident analysis and quality improvement.

8. Validation and moving to production

  • Measure correctness, groundedness, policy violations, and output format compliance with the golden set, and compare results before and after each change.
  • The minimum bar before production is the health check, smoke test, k6 p95 latency, API error rate, RAG hit rate, and fallback rate.
  • In high-risk areas such as finance, education, and operations automation, always separate LLM judgment from deterministic rule checks.
  • Observe latency, token usage, tool failures, and retrieval hit rate with OpenTelemetry, Prometheus, Grafana, or at least structured logs.
  • Start the first release as a read-only Agent with a human-approval workflow, and widen the scope of automatic execution after enough validation.

Model Selection

AI model selection and install/usage by domain

So that you can start a POC right away on a AI Ops project, this section lays out the recommended API models, local models, embedding models, and install/usage guidelines.

Recommended API models

GPT-4.1 mini or Gemini 2.5 Flash

Local/on-premises alternatives

Llama 3.3 70B Instruct or the DeepSeek/Qwen coder family

Embedding models

text-embedding-3-large or bge-m3

Chosen models and Docker install baseline

Choose Qwen3-32B-Instruct as the default local model for the AIOps AI Agent. It handles log/metric/runbook summarization and the tool calling flow in a balanced way, with a coder model attached as a helper when needed.

Start by standing up an OpenAI-compatible endpoint with vLLM Docker, and connect the Prometheus/Loki/GitHub Actions adapters with read-only tokens only.

Recommended server spec

  • Development/demo: GPU with 24GB VRAM, 64GB RAM, NVMe SSD. Suitable for single-user diagnosis including recent logs and runbook search.
  • Production POC: one L40S 48GB or A100 80GB or better. Plan for burst requests and long log summaries during incidents.
  • Do not pile log ingestion and model inference onto one server; split them into at least 2 containers.

Capacity and storage

  • About 18-24GB for the Qwen3 32B Q4 model, 65GB or more for the FP16 cache, and at least a 150GB SSD recommended including log samples and the vector index.
  • Keep the Prometheus/Loki source data in the external systems and store only incident snapshots, runbook chunks, and evaluation results in the AI Agent DB.
  • Give logs kept for incident reproduction a 7-30 day retention policy, and mask sensitive tokens before ingestion.

Docker install commands

  • vLLM: `docker run --runtime nvidia --gpus all -v ~/.cache/huggingface:/root/.cache/huggingface -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model Qwen/Qwen3-32B --max-model-len 32768`
  • Quick local check: `docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama`, then `docker exec -it ollama ollama pull qwen3:32b`
  • App settings: `LLM_BASE_URL=http://localhost:8000/v1`, `LLM_MODEL=Qwen/Qwen3-32B`, `MODEL_TIMEOUT_SECONDS=30`

Post-install verification

  • Measure the match rate between candidate causes and evidence logs over 10 incident alerts, 10 log bundles, and 10 runbooks.
  • Confirm that write actions such as restart, delete, rollback, and scale changes are exposed only through the approval workflow, never through the model answer.
  • Track p95 response time, tool failure rate, and the rate of hallucinated service names as observability metrics.

Why these were chosen

  • An AI Ops Agent has to bring together logs, metrics, deployment history, and runbooks, so tool calling reliability and long-context summarization matter.
  • When code or stack traces need interpreting, attaching a coder-family local model as a helper improves the quality of the candidate-cause explanations.
  • Before automatic recovery, it is safer to start with read-only diagnosis and a suggestion mode based on human approval.

Install and configuration

  • For the API setup, split cause summaries from short alert classification with `LLM_MODEL=gpt-4.1-mini` and `FAST_MODEL=gpt-4.1-nano`.
  • For the local setup, put the vLLM/Ollama endpoint behind a FastAPI adapter and connect the Prometheus, Loki, and deployment history APIs with read-only tokens.
  • Store runbooks in the Vector DB with title, service name, severity, symptoms, recovery steps, and forbidden-command metadata.

How to use them

  • Call `metric_query`, `log_search`, `deploy_lookup`, and `runbook_retriever` in order to gather candidate causes and evidence.
  • Fix the model output to `incident_summary`, `likely_causes`, `evidence`, `recommended_runbook`, and `requires_human_approval`.
  • Leave recovery command execution out of the AI Agent scope and build the approval UI and audit log first.

Cautions

  • Never let the model directly run commands such as delete, restart, scale changes, or deployment rollback.
  • Add a fallback answer so that, without runbook evidence, it does not generate speculative recovery procedures.

Reference Architecture

Overall architecture design

An AI Agent is easier to change and validate when the UI, orchestration, tool integration, data retrieval, and evaluation/operations layers are separated.

Presentation Layer

  • Next.js UI
  • Agent run status
  • Evidence/source display
  • User feedback collection

Agent Orchestration

  • Prompt policy
  • Tool routing
  • Memory/session
  • Structured output

Tool & Integration

  • Business API adapter
  • DB query tool
  • Document search tool
  • Policy check tool

Data & Retrieval

  • PostgreSQL
  • pgvector
  • Document chunking
  • embedding/reranking

Evaluation & Ops

  • Golden set
  • Playwright
  • k6
  • OpenTelemetry/Prometheus

Project ABC

The ABCs of running the project for real

A is architecture design, B is the AI Agent implementation, and C is the validation criteria. Following this order makes it easy to grow from a demo into an Agent you can operate.

A. Architecture

  • Start with metric queries, log search, deployment history lookup, and runbook search as read-only tools.
  • Leave recovery command execution out of the AI Agent and design it as an approval-based suggestion mode.
  • Split the Agent state into alert intake, candidate causes, evidence gathering, runbook recommendation, and human approval.

B. Build

  • Create 10 sample incidents and 10 response runbooks and define how the metrics, logs, and runbooks connect.
  • Make the time range, service name, and severity required parameters for Prometheus queries and log search.
  • Lay out the operations UI in this order: alert card, evidence logs, metric snapshot, recommended runbook.

C. Check

  • Test wrong service names, excessive time ranges, and unauthorized tool calls as failure cases.
  • Confirm that it does not invent speculative recovery procedures for incidents with no runbook.
  • Treat MTTR, candidate-cause accuracy, runbook recommendation accuracy, and p95 response time as the core metrics.

Project Spec

Data and Agent tools

Data

  • Prometheus metrics
  • Application logs
  • Deployment history
  • Incident response runbooks

Agent Tools

  • metric_query
  • log_search
  • deploy_lookup
  • runbook_retriever

AI Agent scope

  • List candidate causes of the alert
  • Check the impact of recent deployments
  • Summarize related logs
  • Suggest a recovery procedure

Validation criteria

  • Blocking of wrong command execution
  • Runbook evidence included
  • MTTR reduction
  • Tool restrictions by permission

Execution

Milestones and deliverables

Milestones

  • Design incident samples
  • Implement the metric/log tools
  • Connect runbook search
  • Approval-based UI
  • Operational safety validation

Deliverables

  • AIOps Agent API
  • Sample runbook set
  • Prometheus query templates
  • Operations dashboard
  • Safety policy tests

Stack

Tech stack

FastAPIPrometheusGrafanaOpenTelemetryLangGraphk6