Skip to main content
AIDevOps
  • Learn
  • Learning Paths
  • Practice
  • Open Source
  • Books
  • Engineering

    AI DevOpsThe full map of building and operating AI servicesLLMOpsLLM deployment Β· evaluation Β· observabilityHands-on ProjectsBuild AI Agent projects

    Knowledge

    DocsTechnical documentationBlogEngineering articlesPloggerDevelopment log feed

    Validate

    Certification3-level skills certification Β· coming soon
AI Models
LlamaTutorial|MistralTutorial|GemmaTutorial|DeepSeekTutorial|QwenTutorial
🐳 DevOps
DevOps Intro & RoadmapLinuxDockerCI/CD|Kubernetes BasicsK8s AdvancedPrometheusGrafana
πŸ€– AI Applied Development
Applied AI Intro & RoadmapHugging FaceLangChainLlamaIndexLLMOps|LangGraphMCPMulti-AgentAgent Evaluation
🧠 AI Core
AI Intro & RoadmapML FundamentalsLLM Fundamentals|Python AIC++|PyTorchTensorFlowJAX
🧠 AI Agent Development
Finance AI AgentLLM API ServerStock Investing AgentAIOps AI AgentEducation AI AgentCoding AI Agent
🌱 Spring Cloud
Spring Intro & RoadmapSpring Cloud GatewaySpring BootJava|Spring AISpring SecuritySpring BatchSpring JPA
🧱 Infrastructure
Infrastructure Intro & RoadmapNginxRedis
☁️ Cloud
Cloud Intro & RoadmapAWSGCPAzureNCPCloudflare
🎨 Frontend
Frontend Intro & RoadmapJavaScriptTypeScript|ReactNext.js|VueNuxt|Electron
πŸ“± Mobile
Mobile Intro & RoadmapKotlinAndroidFlutter
βš™οΈ Backend
Backend Intro & RoadmapPython BasicsFastAPIDjangoFlask|CGoGinNode.js
πŸ’Ύ Database
DB Intro & RoadmapCore SQLOracleMySQLPostgreSQL|MongoDBVector DB
πŸ§ͺ Testing
k6JMeternGrinder
AIDevOps

Engineering AI. From Code to Production.
An engineering learning platform for building and operating AI and AI Agents

Learn

  • All Guides
  • Learning Paths
  • AI Tutorial
  • Practice
  • Books

Resources

  • AI DevOps
  • LLMOps
  • Hands-on Projects
  • Docs
  • Blog
  • Plogger
  • Open Source
  • Certification (coming soon)

Start Here

  • AI Core Roadmap
  • AI Applied Development Roadmap
  • Spring Cloud Roadmap
  • DevOps Roadmap
  • Infrastructure Roadmap

Β 

  • Cloud Roadmap
  • Frontend Roadmap
  • Mobile Roadmap
  • Backend Roadmap
  • Database Roadmap
Β© 2026 AIDevOps. All rights reserved.
Terms of ServicePrivacy PolicySitemaptestforge.kr
  1. Home
  2. Learn
  3. DevOps
  4. Prometheus
Guide to monitoring & observability

πŸ“Š Prometheus Complete Guide

Visitors

Monitor servers and applications with Prometheus + Grafana. A complete guide to metric collection, the Exporter ecosystem, PromQL, provisioning Grafana dashboards, AlertManager alert routing, and installing, extending through CRDs, and operating kube-prometheus-stack.

  • Advanced
  • Updated Oct 1, 2026
  • ~21 min read
  • 15 sections
  • 35 code examples
  • Web IDE lab included
πŸ“Š

Prometheus Web IDE

Run code in the browser with nothing to install and learn through step-by-step examples.

Open the Web IDE β†’
Metric collection & running ExportersPromQL queries & recording rulesGrafana dashboards as codeAlertManager alert routingInstalling kube-prometheus-stack & running it in production

Related frameworks & environments

πŸ“ˆGrafanaβ†’β˜ΈοΈKubernetes Basicsβ†’β˜ΈοΈK8s Advancedβ†’πŸ³Dockerβ†’CICI/CDβ†’

Contents

0 / 18
  1. How to use this guide
  2. Architecture diagrams
  3. What is observability?
  4. Installing with Docker Compose
  5. The Exporter ecosystem & custom metrics
  6. Health checks vs metrics β€” Probes and Actuator
  7. PromQL basics & recording rules
  8. Grafana integration
  9. AlertManager alerts
  10. What is a CRD? β€” the basics of extending the Kubernetes API
  11. Before installing β€” prerequisites & cluster sizing
  12. Kubernetes integration β€” kube-prometheus-stack
  13. CRD extensions β€” ServiceMonitor Β· PodMonitor Β· Probe
  14. Verification Β· upgrades Β· troubleshooting
  15. Prometheus design
  16. Operating standards
  17. Verification strategy
  18. Frequently asked questions
Contents 18 sections
  1. How to use this guide
  2. Architecture diagrams
  3. What is observability?
  4. Installing with Docker Compose
  5. The Exporter ecosystem & custom metrics
  6. Health checks vs metrics β€” Probes and Actuator
  7. PromQL basics & recording rules
  8. Grafana integration
  9. AlertManager alerts
  10. What is a CRD? β€” the basics of extending the Kubernetes API
  11. Before installing β€” prerequisites & cluster sizing
  12. Kubernetes integration β€” kube-prometheus-stack
  13. CRD extensions β€” ServiceMonitor Β· PodMonitor Β· Probe
  14. Verification Β· upgrades Β· troubleshooting
  15. Prometheus design
  16. Operating standards
  17. Verification strategy
  18. Frequently asked questions

How to use this guide

How to read it

Understanding Prometheus as a real-world workflow

Monitor servers and applications with Prometheus + Grafana. A complete guide to metric collection, the Exporter ecosystem, PromQL, provisioning Grafana dashboards, AlertManager alert routing, and installing, extending through CRDs, and operating kube-prometheus-stack. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.

Key perspective

Infrastructure / operations

Rather than memorizing install commands, understand the order in which traffic, runtime, observability, and incident response connect.

Metric collection & running ExportersPromQL queries & recording rulesGrafana dashboards as codeAlertManager alert routingInstalling kube-prometheus-stack & running it in production

Architecture diagrams

To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning Prometheus.

Learning flow

Rendering diagram…

Architecture view

Rendering diagram…

The three pillars of observability

When you first open Prometheus, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.

Confirming that a service is working normally takes three signals that answer different questions. No single one shows the whole picture of a failure, so in practice all three are collected together and linked to each other.
SignalToolsQuestion
MetricsPrometheus, GrafanaHow many requests are coming in right now?
LogsLoki, ELKWhat went wrong?
TracesJaeger, TempoWhere is it slow?

Installing with Docker Compose

Here you look at Installing with Docker Compose alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

prometheus.yml is the core configuration that defines "what to read, how often, and from where". Register scrape targets with scrape_configs, load recording/alerting rule files with rule_files, and connect the AlertManager address with alerting.alertmanagers.
docker-compose.ymlYAML
services:
  prometheus:
    image: prom/prometheus:latest
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert_rules.yml:/etc/prometheus/alert_rules.yml
      - promdata:/prometheus
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --web.enable-lifecycle          # reload the config with curl -X POST, no restart

  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - grafdata:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning   # data sources and dashboards as code

  alertmanager:
    image: prom/alertmanager:latest
    ports: ["9093:9093"]
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml

  node-exporter:
    image: prom/node-exporter:latest
    ports: ["9100:9100"]

volumes:
  promdata:
  grafdata:
prometheus.ymlYAML
global:
  scrape_interval: 15s        # default scrape interval
  evaluation_interval: 15s    # recording/alerting rule evaluation interval

rule_files:
  - "alert_rules.yml"         # recording rules + alerting rules

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: myapp
    metrics_path: /metrics
    static_configs:
      - targets: ["myapp:8080"]
        labels: { env: production, team: backend }
BASH
docker compose up -d

# Check the state of the scrape targets β€” UP means healthy
curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health}'

# Reload only the configuration file (applied with no container restart; needs --web.enable-lifecycle)
curl -X POST localhost:9090/-/reload

# Validate rule file syntax (running it in CI before deploy is recommended)
docker run --rm -v $(pwd):/rules prom/prometheus:latest \
  promtool check rules /rules/alert_rules.yml

The Exporter ecosystem & custom metrics

Here you look at The Exporter ecosystem & custom metrics alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Prometheus uses a pull model. Prometheus periodically scrapes the /metrics endpoint of each target, so for every system you want to monitor you need to place a matching exporter that exposes the metrics in text format. Services you build yourself can expose whatever indicators you want as custom metrics through the client library for each language.
app_metrics.pyPYTHON
from prometheus_client import Counter, Histogram, start_http_server
import time

# Counter β€” a value that only ever accumulates (request count, error count)
REQUEST_COUNT = Counter(
    "http_requests_total", "Total HTTP requests",
    ["method", "endpoint", "status"]
)

# Histogram β€” measures a distribution (buckets for computing response time p50/p95/p99)
REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds", "Request latency",
    ["endpoint"], buckets=[0.05, 0.1, 0.3, 0.5, 1, 2, 5]
)

def handle_request(endpoint: str):
    start = time.time()
    try:
        ...  # the actual handling logic
        REQUEST_COUNT.labels("GET", endpoint, "200").inc()
    finally:
        REQUEST_LATENCY.labels(endpoint).observe(time.time() - start)

start_http_server(8080)   # exposes the /metrics endpoint automatically
blackbox-scrape.ymlYAML
# blackbox_exporter β€” checks endpoint availability from the outside (synthetic monitoring)
# blackbox.yml β€” configuration of the exporter itself
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      method: GET

---
# The scrape job to add to prometheus.yml β€” the real target is passed as params.target
scrape_configs:
  - job_name: blackbox
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://api.example.com/health
          - https://example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115   # reroute to the address of the exporter that performs the actual probe
ExporterWhat it collectsDefault port
node_exporterHost CPU Β· memory Β· disk Β· network9100
cAdvisorResource usage per container (Docker/K8s)8080
blackbox_exporterHTTP/TCP/ICMP probes β€” endpoint availability and response time9115
mysqld_exporter / postgres_exporterDB connections, slow queries, replication lag9104 / 9187
redis_exporterRedis memory, command throughput, hit rate9121
kube-state-metricsState of K8s objects (Pod/Deployment/PVC and so on)8080

Health checks vs metrics β€” Probes and Actuator

Here you look at Health checks vs metrics β€” Probes and Actuator alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

The Kubernetes Liveness/Readiness Probes are not something Prometheus scrapes. They are a binary (UP/DOWN) signal with which the kubelet itself calls the health check endpoint of the application and decides only "should it be restarted (liveness)" and "should it receive traffic (readiness)"; Prometheus, separately, scrapes numeric time series from the /metrics endpoint to analyze trends and alert. Because the same application often exposes both kinds of endpoint at once, it is easy to mistake "health checks" and "metrics" for the same thing.
Rendering diagram…
application.yml β€” Spring Boot ActuatorYAML
management:
  endpoints:
    web:
      exposure:
        include: health, prometheus, metrics   # expose only the endpoints you need, explicitly
  endpoint:
    health:
      probes:
        enabled: true   # splits out /actuator/health/liveness and /actuator/health/readiness for K8s
  prometheus:
    metrics:
      export:
        enabled: true    # exposes micrometer metrics in Prometheus format (/actuator/prometheus)
deployment.yaml β€” wiring each endpoint to its own purposeYAML
spec:
  containers:
  - name: myapp
    livenessProbe:
      httpGet: { path: /actuator/health/liveness, port: 8080 }
      periodSeconds: 10
    readinessProbe:
      httpGet: { path: /actuator/health/readiness, port: 8080 }
      periodSeconds: 5
---
# prometheus.yml (or a ServiceMonitor) scrapes /actuator/prometheus separately
scrape_configs:
  - job_name: myapp
    metrics_path: /actuator/prometheus
    static_configs:
      - targets: ["myapp:8080"]
PromQL β€” observing probe failures (restarts) as a time seriesTEXT
# Containers restarted 3 or more times in the last 15 minutes β€” a sign the liveness probe keeps failing
increase(kube_pod_container_status_restarts_total[15m]) >= 3
ItemLiveness/Readiness ProbePrometheus metrics
Who calls itThe kubelet (inside K8s)The Prometheus server (periodic scrape from outside)
Shape of the valueBinary β€” success/failure (UP/DOWN)Numeric time series β€” trends, ratios, and distributions can be computed
Result of failureContainer restarted, or removed from the Service at onceNo immediate action β€” Alertmanager alerts according to the rules
Purpose"Is this instance healthy at this very moment?""How is the system changing over time?"

Tip

  • The Prometheus Operator "Probe" CRD in the k8s-crds section later on merely shares the name and is a completely different concept β€” it is not the Kubernetes liveness/readiness probe but a resource that remotely checks the availability of external URLs with blackbox_exporter. Whenever you see the word "Probe", check which context it is (kubelet health check vs remote blackbox check).
  • The Actuator /actuator/health can be heavy because it checks dependencies such as the DB and disk too, so splitting liveness/readiness with probes.enabled=true lets liveness check lightly only "is the process alive?" while readiness checks "is it ready to take traffic, dependencies included?".

PromQL basics & recording rules

Here you look at PromQL basics & recording rules alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

A counter can reset to 0 on restart, so rather than the value itself you should look at its rate of increase with rate(). Precomputing the complex queries your dashboards use often as recording rules reduces the load at query time and speeds up responses.
PromQL examplesTEXT
# HTTP requests per second (5-minute average)
rate(http_requests_total[5m])

# Error rate (4xx + 5xx)
sum(rate(http_requests_total{status=~"4..|5.."}[5m]))
/ sum(rate(http_requests_total[5m]))

# p95 response time per endpoint
histogram_quantile(0.95, sum by (le, endpoint) (rate(http_request_duration_seconds_bucket[5m])))

# CPU utilization
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Pods restarted within the last hour (K8s)
increase(kube_pod_container_status_restarts_total[1h]) > 0
alert_rules.yml (recording group)YAML
groups:
  - name: myapp.recording-rules
    interval: 30s
    rules:
      # Precompute heavy queries that are read often and store them under the job:metric:operation naming rule
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_errors:ratio5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          / sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_request_duration:p95_5m
        expr: histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))
Function/operationUseCaution
rate(v[5m])Average per-second rate of increase over a 5-minute windowCounters only. Corrects for counter resets automatically
irate(v[5m])Instantaneous rate computed from just the two most recent samplesNoisy graphs β€” fine for dashboards, unsuitable for alert thresholds
histogram_quantile()Approximates percentiles (p95/p99) from histogram bucketsThe design of the bucket boundaries decides the accuracy
sum by (label)(...)Sums grouped only by specific labelsConfusing by and without risks a cardinality explosion

Grafana integration

Here you look at Grafana integration alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

πŸ“ˆ
This covers only connecting the Prometheus data source and dashboard provisioning. To learn Grafana itself in depth β€” building panels, variables, Unified Alerting, and so on β€” see the Grafana guide.
Creating data sources and dashboards by clicking through the Grafana UI is not reproducible. Managing data sources and dashboards as code with provisioning (file-based automatic configuration) restores the same state automatically even when the Grafana container is recreated or redeployed. If you have several clusters, have each one identified through the Prometheus external_labels, and set things up so dashboards can switch between them by registering a data source per cluster or with a $cluster template variable.
provisioning/datasources/prometheus.ymlYAML
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 15s          # matching global.scrape_interval makes the graphs look natural
      httpMethod: POST
provisioning/dashboards/default.ymlYAML
apiVersion: 1
providers:
  - name: default
    folder: "Services"
    type: file
    updateIntervalSeconds: 30   # detects file changes periodically and reloads automatically
    options:
      path: /etc/grafana/provisioning/dashboards/files
      foldersFromFilesStructure: true
files/myapp-red-metrics.json (key fields excerpted)JSON
{
  "title": "MyApp - RED Metrics",
  "templating": {
    "list": [
      { "name": "job", "type": "query", "query": "label_values(http_requests_total, job)" }
    ]
  },
  "panels": [
    {
      "title": "Request Rate",
      "type": "timeseries",
      "targets": [
        { "expr": "sum(rate(http_requests_total{job=\"$job\"}[5m]))", "legendFormat": "req/s" }
      ]
    },
    {
      "title": "Error Ratio",
      "type": "timeseries",
      "targets": [
        { "expr": "job:http_errors:ratio5m{job=\"$job\"}", "legendFormat": "error %" }
      ],
      "fieldConfig": { "defaults": { "unit": "percentunit", "thresholds": { "steps": [
        { "color": "green", "value": 0 }, { "color": "red", "value": 0.05 }
      ]}}}
    },
    {
      "title": "P95 Duration",
      "type": "timeseries",
      "targets": [
        { "expr": "job:http_request_duration:p95_5m{job=\"$job\"}", "legendFormat": "p95" }
      ]
    }
  ]
}
BASH
# Check through the API that the dashboard loaded properly
curl -s -u admin:admin localhost:3000/api/search?query=RED | jq '.[].title'

# Export the dashboard JSON as code (commit it to version control)
curl -s -u admin:admin localhost:3000/api/dashboards/uid/<uid> | jq '.dashboard' > myapp-red-metrics.json
prometheus.yml (cluster identification label)YAML
# With several clusters, each Prometheus must mark which cluster it is,
# so Grafana or a central store can tell the clusters apart and filter by them later.
global:
  external_labels:
    cluster: prod-us-east     # a unique name per cluster (prod-us-east, prod-eu-west, staging ...)
    region: us-east-1
provisioning/datasources/multi-cluster.ymlYAML
# Approach A β€” register a separate Prometheus data source per cluster
# The cluster is switched with the data source dropdown at the top of the dashboard
apiVersion: 1
datasources:
  - name: Prometheus-Seoul
    uid: prom-seoul
    type: prometheus
    access: proxy
    url: http://prometheus-seoul.monitoring.svc:9090
    jsonData: { timeInterval: 15s }

  - name: Prometheus-Tokyo
    uid: prom-tokyo
    type: prometheus
    access: proxy
    url: http://prometheus-tokyo.monitoring.svc:9090
    jsonData: { timeInterval: 15s }
cluster-selector-panel.json (key fields excerpted)JSON
{
  "templating": {
    "list": [
      {
        "name": "datasource",
        "type": "datasource",
        "query": "prometheus",
        "current": { "text": "Prometheus-Seoul", "value": "prom-seoul" }
      },
      {
        "name": "cluster",
        "type": "query",
        "datasource": "${datasource}",
        "query": "label_values(up, cluster)",
        "refresh": 2
      }
    ]
  },
  "panels": [
    {
      "title": "Cluster Node Count",
      "type": "stat",
      "datasource": "${datasource}",
      "targets": [
        { "expr": "count(up{job=\"node\", cluster=\"$cluster\"} == 1)" }
      ]
    }
  ]
}
values.yaml (automatic import of community dashboards)YAML
# kube-prometheus-stack β€” loads proven cluster dashboards from grafana.com automatically by gnetId
# (add to the helm install -f values.yaml in the Kubernetes integration section)
grafana:
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: community
          orgId: 1
          folder: "Community"
          type: file
          options:
            path: /var/lib/grafana/dashboards/community
  dashboards:
    community:
      k8s-cluster-monitoring:
        gnetId: 315            # Kubernetes cluster monitoring (via Prometheus) β€” overview of the whole cluster
        revision: 3
        datasource: Prometheus
      node-exporter-full:
        gnetId: 1860           # Node Exporter Full β€” CPU/memory/disk/network detail per node
        revision: 37
        datasource: Prometheus
      # check the latest revision on the grafana.com/grafana/dashboards/<gnetId> page and update it
BASH
# When importing directly through the UI, as in a Docker Compose environment
# Grafana β†’ Dashboards β†’ New β†’ Import β†’ enter the ID (315 or 1860) β†’ choose the data source

# Check already installed community dashboards through the API
curl -s -u admin:admin localhost:3000/api/search?query=Kubernetes | jq '.[].title'

# List the clusters (every cluster carrying external_labels.cluster)
curl -s -u admin:admin --data-urlencode 'query=label_values(up, cluster)' \
  localhost:3000/api/datasources/proxy/uid/prom-seoul/api/v1/query
ComponentRoleFile location (default)
Data source provisioningRegisters data sources such as Prometheus automatically, with no UI clicks/etc/grafana/provisioning/datasources/*.yml
Dashboard provisioningLoads dashboard JSON automatically, by folder/etc/grafana/provisioning/dashboards/*.yml
Template variableParameterizes dashboards with $job, $instance, and so ontemplating.list in the dashboard JSON
Folders / permissionsSeparate dashboard folders per team and service; restrict edit rights with RBACGrafana organization settings
Multi-cluster datasourceRegisters the Prometheus of each cluster as a separate data source, switched from the dropdown at the topprovisioning/datasources/*.yml
Importing community dashboardsBring in proven community dashboards instantly by grafana.com IDDashboards β†’ New β†’ Import (or gnetId provisioning)

AlertManager alerts

Here you look at AlertManager alerts alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Prometheus only fires an alert when a rule meets its condition; deciding "to whom and how" that alert is sent is the job of AlertManager. The essentials are separating recipients by severity with the routing tree, and configuring grouping and inhibition so the same alert is not sent repeatedly.
alert_rules.yml (alerting group)YAML
groups:
  - name: myapp.alerts
    rules:
      - alert: HighErrorRate
        expr: job:http_errors:ratio5m > 0.05
        for: 5m                          # must persist for 5 minutes to fire β€” avoids false positives from momentary spikes
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} error rate {{ $value | humanizePercentage }}"
          runbook_url: "https://wiki.internal/runbooks/high-error-rate"

      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} memory usage above 90%"

      - alert: PodCrashLooping
        expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.namespace }}/{{ $labels.pod }} is restarting repeatedly"
alertmanager.ymlYAML
route:
  receiver: default-webhook
  group_by: [alertname, job]
  group_wait: 30s           # after a group is created, gather related alerts for 30 seconds before sending
  group_interval: 5m        # minimum interval before re-sending when new alerts join the same group
  repeat_interval: 4h       # re-send interval for unresolved alerts
  routes:
    - matchers: ["severity = critical"]
      receiver: slack-critical
      continue: false
    - matchers: ["severity = warning"]
      receiver: slack-warning

inhibit_rules:
  # While a critical alert is firing, warning alerts for the same job are noise, so suppress them
  - source_matchers: ["severity = critical"]
    target_matchers: ["severity = warning"]
    equal: [job]

receivers:
  - name: default-webhook
    webhook_configs:
      - url: http://ops-bot:9000/alert

  - name: slack-critical
    slack_configs:
      - api_url: "${SLACK_WEBHOOK_URL}"
        channel: "#alerts-critical"
        title: "{{ .CommonAnnotations.summary }}"
        send_resolved: true

  - name: slack-warning
    slack_configs:
      - api_url: "${SLACK_WEBHOOK_URL}"
        channel: "#alerts-warning"
        send_resolved: true
BASH
# Validate the configuration syntax (run in CI before deploy)
amtool check-config alertmanager.yml

# List the currently active alerts
amtool alert query --alertmanager.url=http://localhost:9093

# Mute a specific alert during a maintenance window (2 hours)
amtool silence add alertname=HighMemoryUsage instance=web-01 \
  --alertmanager.url=http://localhost:9093 \
  --duration=2h --comment="Scheduled maintenance"

# List active silences / expire one early
amtool silence query --alertmanager.url=http://localhost:9093
amtool silence expire <silence-id> --alertmanager.url=http://localhost:9093
ConceptRole
routeThe tree structure that decides which receiver an alert goes to, based on its labels
receiverThe channel that actually receives alerts: Slack, Email, PagerDuty, Webhook, and so on
group_by / group_waitCollects alerts of the same group and sends them together β€” prevents alert storms
repeat_intervalThe interval at which unresolved alerts are re-sent
inhibit_ruleSuppresses related lower alerts (warning) while a serious alert (critical) is active
silenceManually mutes specific alerts for a period in expected situations such as maintenance and deployments

What is a CRD (Custom Resource Definition)?

Here you look at What is a CRD (Custom Resource Definition)? alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

The resources the Kubernetes API provides out of the box, such as Pod, Deployment, and Service, are predefined in the K8s core code. A CRD is a "schema definition" with which users register additional resource kinds that are not on this list. Once a CRD is applied to the cluster, kube-apiserver can from that moment handle the new resource just as it handles a Pod, as in `kubectl get servicemonitor` and `kubectl apply -f servicemonitor.yaml` β€” storage (etcd), reads, and RBAC permission grants all work the same as for built-in resources.

However, a CRD is only a definition β€” "a resource with these fields" β€” and does nothing by itself. For anything to actually happen, a separate program called a Controller has to be running, watching the CR (Custom Resource) instances and reconciling them to the desired state. The combination "a CRD + a dedicated Controller that handles it" is called the Operator pattern β€” the Prometheus Operator of kube-prometheus-stack mentioned earlier is exactly this pattern: it creates a CRD called ServiceMonitor, and a Controller called Prometheus Operator reads those instances and applies them to the scrape configuration of the real Prometheus.
explore.shBASH
# List the CRDs kube-prometheus-stack actually registered
kubectl get crd | grep monitoring.coreos.com
# alertmanagerconfigs.monitoring.coreos.com
# alertmanagers.monitoring.coreos.com
# podmonitors.monitoring.coreos.com
# probes.monitoring.coreos.com
# prometheuses.monitoring.coreos.com
# prometheusrules.monitoring.coreos.com
# servicemonitors.monitoring.coreos.com
# thanosrulers.monitoring.coreos.com

# The schema (field structure) a CRD defines can be read with kubectl explain β€” works just as for a Pod
kubectl explain servicemonitor.spec
kubectl explain servicemonitor.spec.endpoints

# Looking at one CRD in detail, it is made up of group/version/kind and openAPIV3Schema (field validation rules)
kubectl get crd servicemonitors.monitoring.coreos.com -o yaml | head -30

# Check that the Controller behind this CRD (Prometheus Operator) is actually running
# β€” with no Controller, applying a CR only stores it in etcd and nothing happens
kubectl get deploy -n monitoring -l app=kube-prometheus-stack-operator
toy-crd-example.yamlYAML
# A minimal example for understanding the concept β€” registers a new, made-up resource kind called "Website"
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: websites.example.com          # <plural>.<group>
spec:
  group: example.com
  scope: Namespaced
  names:
    plural: websites
    singular: website
    kind: Website                     # from now on kubectl get website works
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:               # the field structure a CR must follow (spec.url is a required string)
          type: object
          properties:
            spec:
              type: object
              required: ["url"]
              properties:
                url:
                  type: string
---
# After applying the CRD above, you can create CR instances like the one below
apiVersion: example.com/v1
kind: Website
metadata:
  name: aidevops
spec:
  url: https://www.aidevops.us
# Note: applying this CR does not actually deploy/check a website.
# For that to happen, a Controller that watches Website CRs has to be built and deployed separately
# (in kube-prometheus-stack, the Prometheus Operator plays this Controller role for you)
TermMeaning
CRD (Custom Resource Definition)The schema definition itself that registers a new resource kind with the K8s API β€” the "specification"
CR (Custom Resource)An actual instance following the schema the CRD defines β€” each YAML you kubectl apply
ControllerA program that watches creation, changes, and deletion of CRs and reconciles them to the desired state
OperatorA CRD + dedicated Controller bundled to automate the operational knowledge of a specific piece of software (e.g. Prometheus Operator)
Difference from built-in resourcesPod/Service are resources hardcoded in the K8s core; a CR is a "plugin resource" anyone can add

Before installing β€” prerequisites & cluster sizing

Here you look at Before installing β€” prerequisites & cluster sizing alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

kube-prometheus-stack installs CRDs and several components at once, so unless you check the state of the cluster beforehand it often fails ambiguously partway through installation. In particular, the CRD manifests are large, which brings a specific error commonly seen when applying them directly with kubectl apply -f, and the resources allocated to Prometheus have to be sized differently from the start according to the cluster scale (number of nodes, number of monitored targets).
Rendering diagram…
preflight-check.shBASH
# 1) Check the cluster and Helm versions β€” recent kube-prometheus-stack requires Helm 3.8+
kubectl version --short
helm version

# 2) Check that the StorageClass really exists (must match storageClassName in values.yaml exactly)
kubectl get storageclass
#   NAME (default)   PROVISIONER             ...
#   fast-ssd          ebs.csi.aws.com          ...   ← use this name as-is in values.yaml

# 3) Check spare node resources β€” Prometheus alone claims at least 500m CPU / 2Gi memory
kubectl top nodes
kubectl describe nodes | grep -A5 "Allocated resources"

# 4) Check that no resource cap is set on the monitoring namespace
kubectl get resourcequota -n monitoring 2>/dev/null || echo "No quota (fine)"

# 5) The error commonly seen when applying the CRDs directly with kubectl apply -f β€”
#    "metadata.annotations: Too long: must have at most 262144 bytes"
#    The CRDs of kube-prometheus-stack are large and exceed the storage limit of the
#    last-applied-configuration annotation of kubectl apply. helm install registers the CRDs in a way
#    that avoids this problem, so if you really must apply them by hand, always use --server-side.
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/crds/crd-prometheusrules.yaml
argocd-application.yaml β€” installing the GitOps wayYAML
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kube-prometheus-stack
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://prometheus-community.github.io/helm-charts
    chart: kube-prometheus-stack
    targetRevision: 65.5.1        # always pin the version β€” managed separately from the Chart.yaml appVersion
    helm:
      valueFiles: ["values-production.yaml"]   # references a values file in the same Git repository
  destination:
    server: https://kubernetes.default.svc
    namespace: monitoring
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: ["CreateNamespace=true"]
Cluster scaleRecommended Prometheus resourcesretentionNotes
Small (up to 20 nodes, tens of ServiceMonitors)requests: 500m CPU / 2Gi15dNo need to stray far from the defaults
Medium (20-100 nodes)requests: 1-2 CPU / 4-8Gi30dSet retentionSize as well to head off running out of volume
Large (100+ nodes, high cardinality)requests: 4 CPU+ / 16Gi+15d + remoteWriteNear the limit of a single Prometheus β€” consider sharding and separate long-term storage with Thanos/Mimir

Tip

CRD-related errors mostly occur "when the CRDs were put in directly with kubectl apply -f" β€” if you install kube-prometheus-stack with helm install (or a GitOps tool that uses the Helm source, as in the ArgoCD example above), Helm handles the CRDs through a separate path and you never meet this problem.

Kubernetes integration β€” kube-prometheus-stack

Here you look at Kubernetes integration β€” kube-prometheus-stack alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

kube-prometheus-stack is an "opinionated" bundle that packages Prometheus, Alertmanager, and Grafana into one Helm chart instead of having you install and wire them up individually. The heart of it is the Prometheus Operator installed alongside β€” the Operator watches CRDs such as ServiceMonitor, PodMonitor, and PrometheusRule and, when a manifest is created/changed, applies it automatically to the Prometheus scrape configuration and alerting rules. In other words, instead of editing prometheus.yml by hand and restarting pods, you can add monitoring targets just by kubectl applying Kubernetes resources.
install.shBASH
# 1) Add the Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# 2) Check the versions β€” in production, always install a pinned version (never latest)
helm search repo prometheus-community/kube-prometheus-stack --versions | head -5

# 3) Install (the CRDs are included in the chart and installed with it)
helm install kube-prom prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --version 65.5.1 \
  -f values.yaml

# 4) Verify the install β€” Prometheus/Alertmanager/Grafana/Operator/node-exporter/kube-state-metrics all Running
kubectl get pods -n monitoring
kubectl get prometheus,alertmanager -n monitoring   # check the state of the CRs the Operator created

# 5) Access Grafana (initial password of the admin account)
kubectl get secret kube-prom-grafana -n monitoring \
  -o jsonpath="{.data.admin-password}" | base64 -d; echo
kubectl port-forward -n monitoring svc/kube-prom-grafana 3000:80
values.yamlYAML
prometheus:
  prometheusSpec:
    retention: 30d                        # local TSDB retention period
    retentionSize: "45GB"                 # also set a size cap, leaving headroom relative to the volume
    resources:
      requests: { cpu: 500m, memory: 2Gi }
      limits:   { memory: 4Gi }           # a CPU limit causes throttling β€” usually best left unset
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: fast-ssd
          resources: { requests: { storage: 50Gi } }
    # when set to false, ServiceMonitors/PodMonitors/Probes/Rules across the whole cluster are picked up regardless of the release label
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    probeSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    additionalScrapeConfigs: []           # for adding static targets that ServiceMonitors cannot cover
    # remoteWrite:                        # if long-term retention is needed, store remotely in Thanos/Mimir/Grafana Cloud and so on
    #   - url: https://mimir.example.com/api/v1/push

# Managed K8s (EKS/GKE/AKS) gives no access to control plane metrics β€” off by default is recommended (avoids Pending targets)
kubeControllerManager: { enabled: false }
kubeScheduler:         { enabled: false }
kubeEtcd:               { enabled: false }
kubeProxy:               { enabled: false }

alertmanager:
  alertmanagerSpec:
    resources:
      requests: { cpu: 100m, memory: 256Mi }
    storage:
      volumeClaimTemplate:
        spec:
          resources: { requests: { storage: 5Gi } }

grafana:
  admin:
    existingSecret: grafana-admin-credentials   # referencing a separate Secret is recommended over a plain-text password
  persistence: { enabled: true, size: 10Gi }
  sidecar:
    dashboards: { enabled: true, label: grafana_dashboard }   # dashboards loaded automatically by ConfigMap label
    datasources: { enabled: true }
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    hosts: ["grafana.aidevops.us"]
    tls:
      - secretName: grafana-tls
        hosts: ["grafana.aidevops.us"]

defaultRules:
  rules:
    etcd: false          # control plane rules are mostly ones to disable on managed K8s too
    kubeScheduler: false
ComponentKindRole
Prometheus OperatorDeploymentWatches the CRDs and creates/updates the Prometheus/Alertmanager StatefulSets and configuration automatically
PrometheusStatefulSet (created by the Operator)The metric storage engine β€” declared through the Prometheus CRD rather than created directly
AlertmanagerStatefulSet (created by the Operator)Alert grouping, deduplication, and routing β€” declared through the Alertmanager/AlertmanagerConfig CRDs
GrafanaDeploymentDashboard visualization. A sidecar container detects labeled ConfigMaps and loads them automatically
node-exporterDaemonSetExposes the CPU/memory/disk/network OS metrics of every node
kube-state-metricsDeploymentTurns the state of K8s API objects such as Pods/Deployments/PVCs into metrics
kubelet / cAdvisorBuilt into the existing kubeletContainer resource usage β€” the chart sets up the default scrape configuration automatically
Default PrometheusRules/dashboardsCRD + ConfigMapProven alerts and dashboards for kube-apiserver, etcd, kubelet, and more are included by default

Extending collection and alerting with CRDs β€” ServiceMonitor Β· PodMonitor Β· Probe

Here you look at Extending collection and alerting with CRDs β€” ServiceMonitor Β· PodMonitor Β· Probe alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

The Operator defines scrape targets in three ways. A ServiceMonitor goes through a Service resource (the most common way), a PodMonitor targets Pods directly when there is no stable Service, and a Probe remotely checks the availability of URLs inside/outside the cluster through blackbox-exporter. The most common mistake is that for all three CRDs the labels must match the selector of the Prometheus CR (the xxxSelectorNilUsesHelmValues settings in the values.yaml above) to be picked up. This Probe CRD shares only its name with the Kubernetes liveness/readiness probe explained earlier in the health-vs-metrics section and is an entirely different resource, so do not confuse them.
servicemonitor.yamlYAML
# The Operator sees this resource and adds myapp to the scrape targets automatically
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp
  namespace: production
  labels:
    release: kube-prom          # needed only when serviceMonitorSelectorNilUsesHelmValues: true
spec:
  selector:
    matchLabels:
      app: myapp                # the label of the target Service
  endpoints:
    - port: metrics              # the port name of the target Service
      path: /metrics
      interval: 15s
podmonitor.yamlYAML
# Scrape Pods directly, with no stable Service (e.g. Job/batch workloads, sidecar metrics)
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: myapp-batch
  namespace: production
spec:
  selector:
    matchLabels:
      app: myapp-batch
  podMetricsEndpoints:
    - port: metrics
      path: /metrics
      interval: 30s
probe-blackbox.yamlYAML
# Remote availability checks of external/internal endpoints β€” blackbox-exporter must be installed separately
# helm install blackbox-exporter prometheus-community/prometheus-blackbox-exporter -n monitoring
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  name: aidevops-uptime
  namespace: monitoring
  labels:
    release: kube-prom
spec:
  jobName: blackbox-http
  interval: 30s
  module: http_2xx
  prober:
    url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc:19115
  targets:
    staticConfig:
      static:
        - https://www.aidevops.us
        - https://www.aidevops.us/learn/prometheus/
prometheusrule.yamlYAML
# Declared as a CRD instead of mounting alert_rules.yml by hand β€” manageable with GitOps
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: myapp-alerts
  namespace: production
  labels:
    release: kube-prom
spec:
  groups:
    - name: myapp.recording
      rules:
        - record: job:http_errors:ratio5m
          expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job)
    - name: myapp.alerts
      rules:
        - alert: HighErrorRate
          expr: job:http_errors:ratio5m{namespace="production"} > 0.05
          for: 5m
          labels: { severity: critical }
          annotations:
            summary: "{{ $labels.job }} error rate above 5%"
        - alert: ProbeFailing
          expr: probe_success{job="blackbox-http"} == 0
          for: 5m
          labels: { severity: critical }
          annotations:
            summary: "{{ $labels.instance }} health check failing"
grafana-dashboard-configmap.yamlYAML
# With the grafana_dashboard label, the sidecar detects it in any namespace of the cluster and loads it automatically
apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  myapp-red-metrics.json: |
    { "title": "MyApp - RED Metrics", "panels": [] }

Verification Β· upgrades Β· troubleshooting

Here you look at Verification Β· upgrades Β· troubleshooting alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Because it is CRD-based, configuration errors are often silently ignored with no Prometheus restart. After deploying, you need the habit of always checking the Target state and the Conditions of the Prometheus CR the Operator created. Upgrades are safe with helm upgrade, but helm uninstall does not delete the CRDs β€” this is intended Helm 3 behavior to prevent data loss, so for a complete removal the CRDs must be deleted separately as well to avoid conflicts on reinstall.
verify.shBASH
# Check the Target state (all should be up)
kubectl port-forward -n monitoring svc/kube-prom-kube-prome-prometheus 9090:9090 &
curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health}'

# Check that the Prometheus CR the Operator created was applied properly
kubectl get prometheus -n monitoring -o jsonpath='{.items[0].status.conditions}' | jq

# Check which ServiceMonitors/PodMonitors/Rules the Operator actually picked up
kubectl get servicemonitors,podmonitors,prometheusrules -A

# Validate PrometheusRule syntax (running it in CI before deploy is recommended)
kubectl get prometheusrule myapp-alerts -n production -o jsonpath='{.spec}' > /tmp/rule.yaml
promtool check rules /tmp/rule.yaml
upgrade-uninstall.shBASH
# Upgrade β€” reapply after changing values.yaml
helm upgrade kube-prom prometheus-community/kube-prometheus-stack \
  --namespace monitoring -f values.yaml

# When upgrading to a specific version, check first whether the CRDs change (a major version bump can change the CRD schema)
helm show crds prometheus-community/kube-prometheus-stack --version 65.5.1 | grep "kind: CustomResourceDefinition" -A2

# Complete removal β€” Helm does not delete the CRDs, so delete them by hand to avoid conflicts on reinstall
helm uninstall kube-prom --namespace monitoring
kubectl delete crd -l app.kubernetes.io/part-of=kube-prometheus-stack
SymptomCauseFix
Created a ServiceMonitor but it does not show under TargetsThe label does not match the selector, or selectorNilUsesHelmValues is trueAdd the release label to the ServiceMonitor, or switch it to false in values.yaml
kube-controller-manager/kube-scheduler Targets stay downManaged K8s does not expose the control planekubeControllerManager/kubeScheduler.enabled: false in values.yaml
PVC stuck in PendingThe specified storageClassName does not exist in the clusterCheck the real name with kubectl get storageclass, then fix values.yaml
The Grafana sidecar cannot find the dashboardThe ConfigMap label does not match grafana.sidecar.dashboards.labelCheck the label value (default grafana_dashboard) and the namespaces being watched
Reinstall fails after helm uninstallCRDs/PVCs from the previous install remainClean up with kubectl delete crd and kubectl delete pvc -n monitoring, then reinstall

Prometheus practical design

Prometheus practical design is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

With Prometheus, decide what to measure first. Without RED/USE metrics and a label cardinality policy, query and storage costs grow.
Decision pointQuestion to askPractical guideline
BoundariesWhich parts of the Prometheus code are most likely to change?Separate input/output, configuration, external integrations, and the core rules.
StateWhere is state created and where does it go away?Make the owner and lifecycle of state visible in the code.
FailureWhat does the caller receive on failure?Decide the timeout, fallback, and error contract first.

Prometheus operating standards

This section covers Prometheus operating standards from a practical perspective. Instead of memorizing the concept, focus on the situations in which you would reach for it.

Manage the scrape interval, retention, recording rules, alert fatigue, and dashboard ownership.

Tip

  • label cardinality
  • recording rules
  • alert fatigue
  • runbook link

Prometheus verification strategy

Prometheus verification strategy is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

Include alert rule tests, dashboard reviews, synthetic metrics, and runbook links.
Quality axisHow to verifyDefinition of done
CorrectnessAutomate the normal/failure cases.The core scenarios pass reproducibly.
Regression preventionWhen fixing a bug, leave the same case behind as a test.The same failure is not deployed again.
OperabilityCheck the logs, metrics, and alerts.There is a path for tracing the cause when a problem occurs.

Prometheus: frequently asked questions

What do Prometheus and Grafana each do?

Prometheus periodically collects metrics from target systems, stores them as time series, and lets you query them. Grafana visualizes that data as dashboards. Prometheus alone can query and evaluate alerting rules, but Grafana is added when you want many metrics on one screen.

Why does Prometheus collect metrics by pulling?

Prometheus reads the /metrics endpoint of each target on a schedule. A target that does not respond is itself a failure signal, the scrape interval and targets are managed centrally, and applications do not need to know the address of the collector. For cases where pulling is hard, such as short-lived batch jobs, use the Pushgateway.

When do I use a Counter and when a Gauge?

A Counter is for values that only increase, such as request or error counts, and you query it with rate() to get the per-second increase. A Gauge is for current values that go up and down, such as memory usage or queued jobs. Values whose distribution matters, such as response time, are recorded as a Histogram so percentiles can be calculated.

How long can Prometheus keep data?

Retention is a setting, but Prometheus stores data on local disk, so it is not suited to long-term retention or to combining data from several instances. For months of retention or high availability, use remote write to long-term storage such as Thanos or Grafana Mimir.

← Previous guideKubernetes AdvancedNext guide β†’Grafana