Monitor servers and applications with Prometheus + Grafana. A complete guide to metric collection, the Exporter ecosystem, PromQL, provisioning Grafana dashboards, AlertManager alert routing, and installing, extending through CRDs, and operating kube-prometheus-stack.
Advanced
Updated Oct 1, 2026
~21 min read
15 sections
35 code examples
Web IDE lab included
π
Prometheus Web IDE
Run code in the browser with nothing to install and learn through step-by-step examples.
Metric collection & running ExportersPromQL queries & recording rulesGrafana dashboards as codeAlertManager alert routingInstalling kube-prometheus-stack & running it in production
Monitor servers and applications with Prometheus + Grafana. A complete guide to metric collection, the Exporter ecosystem, PromQL, provisioning Grafana dashboards, AlertManager alert routing, and installing, extending through CRDs, and operating kube-prometheus-stack. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.
Key perspective
Infrastructure / operations
Rather than memorizing install commands, understand the order in which traffic, runtime, observability, and incident response connect.
Metric collection & running ExportersPromQL queries & recording rulesGrafana dashboards as codeAlertManager alert routingInstalling kube-prometheus-stack & running it in production
Architecture diagrams
To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning Prometheus.
Learning flow
Rendering diagramβ¦
Architecture view
Rendering diagramβ¦
The three pillars of observability
When you first open Prometheus, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.
Confirming that a service is working normally takes three signals that answer different questions. No single one shows the whole picture of a failure, so in practice all three are collected together and linked to each other.
Signal
Tools
Question
Metrics
Prometheus, Grafana
How many requests are coming in right now?
Logs
Loki, ELK
What went wrong?
Traces
Jaeger, Tempo
Where is it slow?
Installing with Docker Compose
Here you look at Installing with Docker Compose alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
prometheus.yml is the core configuration that defines "what to read, how often, and from where". Register scrape targets with scrape_configs, load recording/alerting rule files with rule_files, and connect the AlertManager address with alerting.alertmanagers.
docker-compose.ymlYAML
services: prometheus: image: prom/prometheus:latest ports: ["9090:9090"] volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml - ./alert_rules.yml:/etc/prometheus/alert_rules.yml - promdata:/prometheus command: - --config.file=/etc/prometheus/prometheus.yml - --web.enable-lifecycle # reload the config with curl -X POST, no restart grafana: image: grafana/grafana:latest ports: ["3000:3000"] environment: GF_SECURITY_ADMIN_PASSWORD: admin volumes: - grafdata:/var/lib/grafana - ./grafana/provisioning:/etc/grafana/provisioning # data sources and dashboards as code alertmanager: image: prom/alertmanager:latest ports: ["9093:9093"] volumes: - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml node-exporter: image: prom/node-exporter:latest ports: ["9100:9100"]volumes: promdata: grafdata:
docker compose up -d# Check the state of the scrape targets β UP means healthycurl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health}'# Reload only the configuration file (applied with no container restart; needs --web.enable-lifecycle)curl -X POST localhost:9090/-/reload# Validate rule file syntax (running it in CI before deploy is recommended)docker run --rm -v $(pwd):/rules prom/prometheus:latest \ promtool check rules /rules/alert_rules.yml
The Exporter ecosystem & custom metrics
Here you look at The Exporter ecosystem & custom metrics alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Prometheus uses a pull model. Prometheus periodically scrapes the /metrics endpoint of each target, so for every system you want to monitor you need to place a matching exporter that exposes the metrics in text format. Services you build yourself can expose whatever indicators you want as custom metrics through the client library for each language.
app_metrics.pyPYTHON
from prometheus_client import Counter, Histogram, start_http_serverimport time# Counter β a value that only ever accumulates (request count, error count)REQUEST_COUNT = Counter( "http_requests_total", "Total HTTP requests", ["method", "endpoint", "status"])# Histogram β measures a distribution (buckets for computing response time p50/p95/p99)REQUEST_LATENCY = Histogram( "http_request_duration_seconds", "Request latency", ["endpoint"], buckets=[0.05, 0.1, 0.3, 0.5, 1, 2, 5])def handle_request(endpoint: str): start = time.time() try: ... # the actual handling logic REQUEST_COUNT.labels("GET", endpoint, "200").inc() finally: REQUEST_LATENCY.labels(endpoint).observe(time.time() - start)start_http_server(8080) # exposes the /metrics endpoint automatically
blackbox-scrape.ymlYAML
# blackbox_exporter β checks endpoint availability from the outside (synthetic monitoring)# blackbox.yml β configuration of the exporter itselfmodules: http_2xx: prober: http timeout: 5s http: valid_status_codes: [200] method: GET---# The scrape job to add to prometheus.yml β the real target is passed as params.targetscrape_configs: - job_name: blackbox metrics_path: /probe params: module: [http_2xx] static_configs: - targets: - https://api.example.com/health - https://example.com relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115 # reroute to the address of the exporter that performs the actual probe
Exporter
What it collects
Default port
node_exporter
Host CPU Β· memory Β· disk Β· network
9100
cAdvisor
Resource usage per container (Docker/K8s)
8080
blackbox_exporter
HTTP/TCP/ICMP probes β endpoint availability and response time
9115
mysqld_exporter / postgres_exporter
DB connections, slow queries, replication lag
9104 / 9187
redis_exporter
Redis memory, command throughput, hit rate
9121
kube-state-metrics
State of K8s objects (Pod/Deployment/PVC and so on)
8080
Health checks vs metrics β Probes and Actuator
Here you look at Health checks vs metrics β Probes and Actuator alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
The Kubernetes Liveness/Readiness Probes are not something Prometheus scrapes. They are a binary (UP/DOWN) signal with which the kubelet itself calls the health check endpoint of the application and decides only "should it be restarted (liveness)" and "should it receive traffic (readiness)"; Prometheus, separately, scrapes numeric time series from the /metrics endpoint to analyze trends and alert. Because the same application often exposes both kinds of endpoint at once, it is easy to mistake "health checks" and "metrics" for the same thing.
Rendering diagramβ¦
application.yml β Spring Boot ActuatorYAML
management: endpoints: web: exposure: include: health, prometheus, metrics # expose only the endpoints you need, explicitly endpoint: health: probes: enabled: true # splits out /actuator/health/liveness and /actuator/health/readiness for K8s prometheus: metrics: export: enabled: true # exposes micrometer metrics in Prometheus format (/actuator/prometheus)
deployment.yaml β wiring each endpoint to its own purposeYAML
PromQL β observing probe failures (restarts) as a time seriesTEXT
# Containers restarted 3 or more times in the last 15 minutes β a sign the liveness probe keeps failing
increase(kube_pod_container_status_restarts_total[15m]) >= 3
Item
Liveness/Readiness Probe
Prometheus metrics
Who calls it
The kubelet (inside K8s)
The Prometheus server (periodic scrape from outside)
Shape of the value
Binary β success/failure (UP/DOWN)
Numeric time series β trends, ratios, and distributions can be computed
Result of failure
Container restarted, or removed from the Service at once
No immediate action β Alertmanager alerts according to the rules
Purpose
"Is this instance healthy at this very moment?"
"How is the system changing over time?"
PromQL basics & recording rules
Here you look at PromQL basics & recording rules alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
A counter can reset to 0 on restart, so rather than the value itself you should look at its rate of increase with rate(). Precomputing the complex queries your dashboards use often as recording rules reduces the load at query time and speeds up responses.
PromQL examplesTEXT
# HTTP requests per second (5-minute average)
rate(http_requests_total[5m])
# Error rate (4xx + 5xx)
sum(rate(http_requests_total{status=~"4..|5.."}[5m]))
/ sum(rate(http_requests_total[5m]))
# p95 response time per endpoint
histogram_quantile(0.95, sum by (le, endpoint) (rate(http_request_duration_seconds_bucket[5m])))
# CPU utilization
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# Pods restarted within the last hour (K8s)
increase(kube_pod_container_status_restarts_total[1h]) > 0
alert_rules.yml (recording group)YAML
groups: - name: myapp.recording-rules interval: 30s rules: # Precompute heavy queries that are read often and store them under the job:metric:operation naming rule - record: job:http_requests:rate5m expr: sum by (job) (rate(http_requests_total[5m])) - record: job:http_errors:ratio5m expr: | sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (job) (rate(http_requests_total[5m])) - record: job:http_request_duration:p95_5m expr: histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))
Function/operation
Use
Caution
rate(v[5m])
Average per-second rate of increase over a 5-minute window
Counters only. Corrects for counter resets automatically
irate(v[5m])
Instantaneous rate computed from just the two most recent samples
Noisy graphs β fine for dashboards, unsuitable for alert thresholds
histogram_quantile()
Approximates percentiles (p95/p99) from histogram buckets
The design of the bucket boundaries decides the accuracy
sum by (label)(...)
Sums grouped only by specific labels
Confusing by and without risks a cardinality explosion
Grafana integration
Here you look at Grafana integration alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
π
This covers only connecting the Prometheus data source and dashboard provisioning. To learn Grafana itself in depth β building panels, variables, Unified Alerting, and so on β see the Grafana guide.
Creating data sources and dashboards by clicking through the Grafana UI is not reproducible. Managing data sources and dashboards as code with provisioning (file-based automatic configuration) restores the same state automatically even when the Grafana container is recreated or redeployed. If you have several clusters, have each one identified through the Prometheus external_labels, and set things up so dashboards can switch between them by registering a data source per cluster or with a $cluster template variable.
provisioning/datasources/prometheus.ymlYAML
apiVersion: 1datasources: - name: Prometheus type: prometheus access: proxy url: http://prometheus:9090 isDefault: true jsonData: timeInterval: 15s # matching global.scrape_interval makes the graphs look natural httpMethod: POST
# Check through the API that the dashboard loaded properlycurl -s -u admin:admin localhost:3000/api/search?query=RED | jq '.[].title'# Export the dashboard JSON as code (commit it to version control)curl -s -u admin:admin localhost:3000/api/dashboards/uid/<uid> | jq '.dashboard' > myapp-red-metrics.json
prometheus.yml (cluster identification label)YAML
# With several clusters, each Prometheus must mark which cluster it is,# so Grafana or a central store can tell the clusters apart and filter by them later.global: external_labels: cluster: prod-us-east # a unique name per cluster (prod-us-east, prod-eu-west, staging ...) region: us-east-1
provisioning/datasources/multi-cluster.ymlYAML
# Approach A β register a separate Prometheus data source per cluster# The cluster is switched with the data source dropdown at the top of the dashboardapiVersion: 1datasources: - name: Prometheus-Seoul uid: prom-seoul type: prometheus access: proxy url: http://prometheus-seoul.monitoring.svc:9090 jsonData: { timeInterval: 15s } - name: Prometheus-Tokyo uid: prom-tokyo type: prometheus access: proxy url: http://prometheus-tokyo.monitoring.svc:9090 jsonData: { timeInterval: 15s }
values.yaml (automatic import of community dashboards)YAML
# kube-prometheus-stack β loads proven cluster dashboards from grafana.com automatically by gnetId# (add to the helm install -f values.yaml in the Kubernetes integration section)grafana: dashboardProviders: dashboardproviders.yaml: apiVersion: 1 providers: - name: community orgId: 1 folder: "Community" type: file options: path: /var/lib/grafana/dashboards/community dashboards: community: k8s-cluster-monitoring: gnetId: 315 # Kubernetes cluster monitoring (via Prometheus) β overview of the whole cluster revision: 3 datasource: Prometheus node-exporter-full: gnetId: 1860 # Node Exporter Full β CPU/memory/disk/network detail per node revision: 37 datasource: Prometheus # check the latest revision on the grafana.com/grafana/dashboards/<gnetId> page and update it
BASH
# When importing directly through the UI, as in a Docker Compose environment# Grafana β Dashboards β New β Import β enter the ID (315 or 1860) β choose the data source# Check already installed community dashboards through the APIcurl -s -u admin:admin localhost:3000/api/search?query=Kubernetes | jq '.[].title'# List the clusters (every cluster carrying external_labels.cluster)curl -s -u admin:admin --data-urlencode 'query=label_values(up, cluster)' \ localhost:3000/api/datasources/proxy/uid/prom-seoul/api/v1/query
Component
Role
File location (default)
Data source provisioning
Registers data sources such as Prometheus automatically, with no UI clicks
/etc/grafana/provisioning/datasources/*.yml
Dashboard provisioning
Loads dashboard JSON automatically, by folder
/etc/grafana/provisioning/dashboards/*.yml
Template variable
Parameterizes dashboards with $job, $instance, and so on
templating.list in the dashboard JSON
Folders / permissions
Separate dashboard folders per team and service; restrict edit rights with RBAC
Grafana organization settings
Multi-cluster datasource
Registers the Prometheus of each cluster as a separate data source, switched from the dropdown at the top
provisioning/datasources/*.yml
Importing community dashboards
Bring in proven community dashboards instantly by grafana.com ID
Dashboards β New β Import (or gnetId provisioning)
AlertManager alerts
Here you look at AlertManager alerts alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Prometheus only fires an alert when a rule meets its condition; deciding "to whom and how" that alert is sent is the job of AlertManager. The essentials are separating recipients by severity with the routing tree, and configuring grouping and inhibition so the same alert is not sent repeatedly.
route: receiver: default-webhook group_by: [alertname, job] group_wait: 30s # after a group is created, gather related alerts for 30 seconds before sending group_interval: 5m # minimum interval before re-sending when new alerts join the same group repeat_interval: 4h # re-send interval for unresolved alerts routes: - matchers: ["severity = critical"] receiver: slack-critical continue: false - matchers: ["severity = warning"] receiver: slack-warninginhibit_rules: # While a critical alert is firing, warning alerts for the same job are noise, so suppress them - source_matchers: ["severity = critical"] target_matchers: ["severity = warning"] equal: [job]receivers: - name: default-webhook webhook_configs: - url: http://ops-bot:9000/alert - name: slack-critical slack_configs: - api_url: "${SLACK_WEBHOOK_URL}" channel: "#alerts-critical" title: "{{ .CommonAnnotations.summary }}" send_resolved: true - name: slack-warning slack_configs: - api_url: "${SLACK_WEBHOOK_URL}" channel: "#alerts-warning" send_resolved: true
BASH
# Validate the configuration syntax (run in CI before deploy)amtool check-config alertmanager.yml# List the currently active alertsamtool alert query --alertmanager.url=http://localhost:9093# Mute a specific alert during a maintenance window (2 hours)amtool silence add alertname=HighMemoryUsage instance=web-01 \ --alertmanager.url=http://localhost:9093 \ --duration=2h --comment="Scheduled maintenance"# List active silences / expire one earlyamtool silence query --alertmanager.url=http://localhost:9093amtool silence expire <silence-id> --alertmanager.url=http://localhost:9093
Concept
Role
route
The tree structure that decides which receiver an alert goes to, based on its labels
receiver
The channel that actually receives alerts: Slack, Email, PagerDuty, Webhook, and so on
group_by / group_wait
Collects alerts of the same group and sends them together β prevents alert storms
repeat_interval
The interval at which unresolved alerts are re-sent
inhibit_rule
Suppresses related lower alerts (warning) while a serious alert (critical) is active
silence
Manually mutes specific alerts for a period in expected situations such as maintenance and deployments
What is a CRD (Custom Resource Definition)?
Here you look at What is a CRD (Custom Resource Definition)? alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
The resources the Kubernetes API provides out of the box, such as Pod, Deployment, and Service, are predefined in the K8s core code. A CRD is a "schema definition" with which users register additional resource kinds that are not on this list. Once a CRD is applied to the cluster, kube-apiserver can from that moment handle the new resource just as it handles a Pod, as in `kubectl get servicemonitor` and `kubectl apply -f servicemonitor.yaml` β storage (etcd), reads, and RBAC permission grants all work the same as for built-in resources.
However, a CRD is only a definition β "a resource with these fields" β and does nothing by itself. For anything to actually happen, a separate program called a Controller has to be running, watching the CR (Custom Resource) instances and reconciling them to the desired state. The combination "a CRD + a dedicated Controller that handles it" is called the Operator pattern β the Prometheus Operator of kube-prometheus-stack mentioned earlier is exactly this pattern: it creates a CRD called ServiceMonitor, and a Controller called Prometheus Operator reads those instances and applies them to the scrape configuration of the real Prometheus.
explore.shBASH
# List the CRDs kube-prometheus-stack actually registeredkubectl get crd | grep monitoring.coreos.com# alertmanagerconfigs.monitoring.coreos.com# alertmanagers.monitoring.coreos.com# podmonitors.monitoring.coreos.com# probes.monitoring.coreos.com# prometheuses.monitoring.coreos.com# prometheusrules.monitoring.coreos.com# servicemonitors.monitoring.coreos.com# thanosrulers.monitoring.coreos.com# The schema (field structure) a CRD defines can be read with kubectl explain β works just as for a Podkubectl explain servicemonitor.speckubectl explain servicemonitor.spec.endpoints# Looking at one CRD in detail, it is made up of group/version/kind and openAPIV3Schema (field validation rules)kubectl get crd servicemonitors.monitoring.coreos.com -o yaml | head -30# Check that the Controller behind this CRD (Prometheus Operator) is actually running# β with no Controller, applying a CR only stores it in etcd and nothing happenskubectl get deploy -n monitoring -l app=kube-prometheus-stack-operator
toy-crd-example.yamlYAML
# A minimal example for understanding the concept β registers a new, made-up resource kind called "Website"apiVersion: apiextensions.k8s.io/v1kind: CustomResourceDefinitionmetadata: name: websites.example.com # <plural>.<group>spec: group: example.com scope: Namespaced names: plural: websites singular: website kind: Website # from now on kubectl get website works versions: - name: v1 served: true storage: true schema: openAPIV3Schema: # the field structure a CR must follow (spec.url is a required string) type: object properties: spec: type: object required: ["url"] properties: url: type: string---# After applying the CRD above, you can create CR instances like the one belowapiVersion: example.com/v1kind: Websitemetadata: name: aidevopsspec: url: https://www.aidevops.us# Note: applying this CR does not actually deploy/check a website.# For that to happen, a Controller that watches Website CRs has to be built and deployed separately# (in kube-prometheus-stack, the Prometheus Operator plays this Controller role for you)
Term
Meaning
CRD (Custom Resource Definition)
The schema definition itself that registers a new resource kind with the K8s API β the "specification"
CR (Custom Resource)
An actual instance following the schema the CRD defines β each YAML you kubectl apply
Controller
A program that watches creation, changes, and deletion of CRs and reconciles them to the desired state
Operator
A CRD + dedicated Controller bundled to automate the operational knowledge of a specific piece of software (e.g. Prometheus Operator)
Difference from built-in resources
Pod/Service are resources hardcoded in the K8s core; a CR is a "plugin resource" anyone can add
Before installing β prerequisites & cluster sizing
Here you look at Before installing β prerequisites & cluster sizing alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
kube-prometheus-stack installs CRDs and several components at once, so unless you check the state of the cluster beforehand it often fails ambiguously partway through installation. In particular, the CRD manifests are large, which brings a specific error commonly seen when applying them directly with kubectl apply -f, and the resources allocated to Prometheus have to be sized differently from the start according to the cluster scale (number of nodes, number of monitored targets).
Rendering diagramβ¦
preflight-check.shBASH
# 1) Check the cluster and Helm versions β recent kube-prometheus-stack requires Helm 3.8+kubectl version --shorthelm version# 2) Check that the StorageClass really exists (must match storageClassName in values.yaml exactly)kubectl get storageclass# NAME (default) PROVISIONER ...# fast-ssd ebs.csi.aws.com ... β use this name as-is in values.yaml# 3) Check spare node resources β Prometheus alone claims at least 500m CPU / 2Gi memorykubectl top nodeskubectl describe nodes | grep -A5 "Allocated resources"# 4) Check that no resource cap is set on the monitoring namespacekubectl get resourcequota -n monitoring 2>/dev/null || echo "No quota (fine)"# 5) The error commonly seen when applying the CRDs directly with kubectl apply -f β# "metadata.annotations: Too long: must have at most 262144 bytes"# The CRDs of kube-prometheus-stack are large and exceed the storage limit of the# last-applied-configuration annotation of kubectl apply. helm install registers the CRDs in a way# that avoids this problem, so if you really must apply them by hand, always use --server-side.kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/crds/crd-prometheusrules.yaml
argocd-application.yaml β installing the GitOps wayYAML
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata: name: kube-prometheus-stack namespace: argocdspec: project: default source: repoURL: https://prometheus-community.github.io/helm-charts chart: kube-prometheus-stack targetRevision: 65.5.1 # always pin the version β managed separately from the Chart.yaml appVersion helm: valueFiles: ["values-production.yaml"] # references a values file in the same Git repository destination: server: https://kubernetes.default.svc namespace: monitoring syncPolicy: automated: { prune: true, selfHeal: true } syncOptions: ["CreateNamespace=true"]
Cluster scale
Recommended Prometheus resources
retention
Notes
Small (up to 20 nodes, tens of ServiceMonitors)
requests: 500m CPU / 2Gi
15d
No need to stray far from the defaults
Medium (20-100 nodes)
requests: 1-2 CPU / 4-8Gi
30d
Set retentionSize as well to head off running out of volume
Large (100+ nodes, high cardinality)
requests: 4 CPU+ / 16Gi+
15d + remoteWrite
Near the limit of a single Prometheus β consider sharding and separate long-term storage with Thanos/Mimir
Kubernetes integration β kube-prometheus-stack
Here you look at Kubernetes integration β kube-prometheus-stack alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
kube-prometheus-stack is an "opinionated" bundle that packages Prometheus, Alertmanager, and Grafana into one Helm chart instead of having you install and wire them up individually. The heart of it is the Prometheus Operator installed alongside β the Operator watches CRDs such as ServiceMonitor, PodMonitor, and PrometheusRule and, when a manifest is created/changed, applies it automatically to the Prometheus scrape configuration and alerting rules. In other words, instead of editing prometheus.yml by hand and restarting pods, you can add monitoring targets just by kubectl applying Kubernetes resources.
install.shBASH
# 1) Add the Helm repohelm repo add prometheus-community https://prometheus-community.github.io/helm-chartshelm repo update# 2) Check the versions β in production, always install a pinned version (never latest)helm search repo prometheus-community/kube-prometheus-stack --versions | head -5# 3) Install (the CRDs are included in the chart and installed with it)helm install kube-prom prometheus-community/kube-prometheus-stack \ --namespace monitoring --create-namespace \ --version 65.5.1 \ -f values.yaml# 4) Verify the install β Prometheus/Alertmanager/Grafana/Operator/node-exporter/kube-state-metrics all Runningkubectl get pods -n monitoringkubectl get prometheus,alertmanager -n monitoring # check the state of the CRs the Operator created# 5) Access Grafana (initial password of the admin account)kubectl get secret kube-prom-grafana -n monitoring \ -o jsonpath="{.data.admin-password}" | base64 -d; echokubectl port-forward -n monitoring svc/kube-prom-grafana 3000:80
values.yamlYAML
prometheus: prometheusSpec: retention: 30d # local TSDB retention period retentionSize: "45GB" # also set a size cap, leaving headroom relative to the volume resources: requests: { cpu: 500m, memory: 2Gi } limits: { memory: 4Gi } # a CPU limit causes throttling β usually best left unset storageSpec: volumeClaimTemplate: spec: storageClassName: fast-ssd resources: { requests: { storage: 50Gi } } # when set to false, ServiceMonitors/PodMonitors/Probes/Rules across the whole cluster are picked up regardless of the release label serviceMonitorSelectorNilUsesHelmValues: false podMonitorSelectorNilUsesHelmValues: false probeSelectorNilUsesHelmValues: false ruleSelectorNilUsesHelmValues: false additionalScrapeConfigs: [] # for adding static targets that ServiceMonitors cannot cover # remoteWrite: # if long-term retention is needed, store remotely in Thanos/Mimir/Grafana Cloud and so on # - url: https://mimir.example.com/api/v1/push# Managed K8s (EKS/GKE/AKS) gives no access to control plane metrics β off by default is recommended (avoids Pending targets)kubeControllerManager: { enabled: false }kubeScheduler: { enabled: false }kubeEtcd: { enabled: false }kubeProxy: { enabled: false }alertmanager: alertmanagerSpec: resources: requests: { cpu: 100m, memory: 256Mi } storage: volumeClaimTemplate: spec: resources: { requests: { storage: 5Gi } }grafana: admin: existingSecret: grafana-admin-credentials # referencing a separate Secret is recommended over a plain-text password persistence: { enabled: true, size: 10Gi } sidecar: dashboards: { enabled: true, label: grafana_dashboard } # dashboards loaded automatically by ConfigMap label datasources: { enabled: true } ingress: enabled: true ingressClassName: nginx annotations: cert-manager.io/cluster-issuer: letsencrypt-prod hosts: ["grafana.aidevops.us"] tls: - secretName: grafana-tls hosts: ["grafana.aidevops.us"]defaultRules: rules: etcd: false # control plane rules are mostly ones to disable on managed K8s too kubeScheduler: false
Component
Kind
Role
Prometheus Operator
Deployment
Watches the CRDs and creates/updates the Prometheus/Alertmanager StatefulSets and configuration automatically
Prometheus
StatefulSet (created by the Operator)
The metric storage engine β declared through the Prometheus CRD rather than created directly
Alertmanager
StatefulSet (created by the Operator)
Alert grouping, deduplication, and routing β declared through the Alertmanager/AlertmanagerConfig CRDs
Grafana
Deployment
Dashboard visualization. A sidecar container detects labeled ConfigMaps and loads them automatically
node-exporter
DaemonSet
Exposes the CPU/memory/disk/network OS metrics of every node
kube-state-metrics
Deployment
Turns the state of K8s API objects such as Pods/Deployments/PVCs into metrics
kubelet / cAdvisor
Built into the existing kubelet
Container resource usage β the chart sets up the default scrape configuration automatically
Default PrometheusRules/dashboards
CRD + ConfigMap
Proven alerts and dashboards for kube-apiserver, etcd, kubelet, and more are included by default
Extending collection and alerting with CRDs β ServiceMonitor Β· PodMonitor Β· Probe
Here you look at Extending collection and alerting with CRDs β ServiceMonitor Β· PodMonitor Β· Probe alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
The Operator defines scrape targets in three ways. A ServiceMonitor goes through a Service resource (the most common way), a PodMonitor targets Pods directly when there is no stable Service, and a Probe remotely checks the availability of URLs inside/outside the cluster through blackbox-exporter. The most common mistake is that for all three CRDs the labels must match the selector of the Prometheus CR (the xxxSelectorNilUsesHelmValues settings in the values.yaml above) to be picked up. This Probe CRD shares only its name with the Kubernetes liveness/readiness probe explained earlier in the health-vs-metrics section and is an entirely different resource, so do not confuse them.
servicemonitor.yamlYAML
# The Operator sees this resource and adds myapp to the scrape targets automaticallyapiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata: name: myapp namespace: production labels: release: kube-prom # needed only when serviceMonitorSelectorNilUsesHelmValues: truespec: selector: matchLabels: app: myapp # the label of the target Service endpoints: - port: metrics # the port name of the target Service path: /metrics interval: 15s
# Declared as a CRD instead of mounting alert_rules.yml by hand β manageable with GitOpsapiVersion: monitoring.coreos.com/v1kind: PrometheusRulemetadata: name: myapp-alerts namespace: production labels: release: kube-promspec: groups: - name: myapp.recording rules: - record: job:http_errors:ratio5m expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job) - name: myapp.alerts rules: - alert: HighErrorRate expr: job:http_errors:ratio5m{namespace="production"} > 0.05 for: 5m labels: { severity: critical } annotations: summary: "{{ $labels.job }} error rate above 5%" - alert: ProbeFailing expr: probe_success{job="blackbox-http"} == 0 for: 5m labels: { severity: critical } annotations: summary: "{{ $labels.instance }} health check failing"
grafana-dashboard-configmap.yamlYAML
# With the grafana_dashboard label, the sidecar detects it in any namespace of the cluster and loads it automaticallyapiVersion: v1kind: ConfigMapmetadata: name: myapp-dashboard namespace: monitoring labels: grafana_dashboard: "1"data: myapp-red-metrics.json: | { "title": "MyApp - RED Metrics", "panels": [] }
Verification Β· upgrades Β· troubleshooting
Here you look at Verification Β· upgrades Β· troubleshooting alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Because it is CRD-based, configuration errors are often silently ignored with no Prometheus restart. After deploying, you need the habit of always checking the Target state and the Conditions of the Prometheus CR the Operator created. Upgrades are safe with helm upgrade, but helm uninstall does not delete the CRDs β this is intended Helm 3 behavior to prevent data loss, so for a complete removal the CRDs must be deleted separately as well to avoid conflicts on reinstall.
verify.shBASH
# Check the Target state (all should be up)kubectl port-forward -n monitoring svc/kube-prom-kube-prome-prometheus 9090:9090 &curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health}'# Check that the Prometheus CR the Operator created was applied properlykubectl get prometheus -n monitoring -o jsonpath='{.items[0].status.conditions}' | jq# Check which ServiceMonitors/PodMonitors/Rules the Operator actually picked upkubectl get servicemonitors,podmonitors,prometheusrules -A# Validate PrometheusRule syntax (running it in CI before deploy is recommended)kubectl get prometheusrule myapp-alerts -n production -o jsonpath='{.spec}' > /tmp/rule.yamlpromtool check rules /tmp/rule.yaml
upgrade-uninstall.shBASH
# Upgrade β reapply after changing values.yamlhelm upgrade kube-prom prometheus-community/kube-prometheus-stack \ --namespace monitoring -f values.yaml# When upgrading to a specific version, check first whether the CRDs change (a major version bump can change the CRD schema)helm show crds prometheus-community/kube-prometheus-stack --version 65.5.1 | grep "kind: CustomResourceDefinition" -A2# Complete removal β Helm does not delete the CRDs, so delete them by hand to avoid conflicts on reinstallhelm uninstall kube-prom --namespace monitoringkubectl delete crd -l app.kubernetes.io/part-of=kube-prometheus-stack
Symptom
Cause
Fix
Created a ServiceMonitor but it does not show under Targets
The label does not match the selector, or selectorNilUsesHelmValues is true
Add the release label to the ServiceMonitor, or switch it to false in values.yaml
kube-controller-manager/kube-scheduler Targets stay down
Managed K8s does not expose the control plane
kubeControllerManager/kubeScheduler.enabled: false in values.yaml
PVC stuck in Pending
The specified storageClassName does not exist in the cluster
Check the real name with kubectl get storageclass, then fix values.yaml
The Grafana sidecar cannot find the dashboard
The ConfigMap label does not match grafana.sidecar.dashboards.label
Check the label value (default grafana_dashboard) and the namespaces being watched
Reinstall fails after helm uninstall
CRDs/PVCs from the previous install remain
Clean up with kubectl delete crd and kubectl delete pvc -n monitoring, then reinstall
Prometheus practical design
Prometheus practical design is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
With Prometheus, decide what to measure first. Without RED/USE metrics and a label cardinality policy, query and storage costs grow.
Decision point
Question to ask
Practical guideline
Boundaries
Which parts of the Prometheus code are most likely to change?
Separate input/output, configuration, external integrations, and the core rules.
State
Where is state created and where does it go away?
Make the owner and lifecycle of state visible in the code.
Failure
What does the caller receive on failure?
Decide the timeout, fallback, and error contract first.
Prometheus operating standards
This section covers Prometheus operating standards from a practical perspective. Instead of memorizing the concept, focus on the situations in which you would reach for it.
Manage the scrape interval, retention, recording rules, alert fatigue, and dashboard ownership.
Prometheus verification strategy
Prometheus verification strategy is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
Include alert rule tests, dashboard reviews, synthetic metrics, and runbook links.
Quality axis
How to verify
Definition of done
Correctness
Automate the normal/failure cases.
The core scenarios pass reproducibly.
Regression prevention
When fixing a bug, leave the same case behind as a test.
The same failure is not deployed again.
Operability
Check the logs, metrics, and alerts.
There is a path for tracing the cause when a problem occurs.
Prometheus: frequently asked questions
What do Prometheus and Grafana each do?
Prometheus periodically collects metrics from target systems, stores them as time series, and lets you query them. Grafana visualizes that data as dashboards. Prometheus alone can query and evaluate alerting rules, but Grafana is added when you want many metrics on one screen.
Why does Prometheus collect metrics by pulling?
Prometheus reads the /metrics endpoint of each target on a schedule. A target that does not respond is itself a failure signal, the scrape interval and targets are managed centrally, and applications do not need to know the address of the collector. For cases where pulling is hard, such as short-lived batch jobs, use the Pushgateway.
When do I use a Counter and when a Gauge?
A Counter is for values that only increase, such as request or error counts, and you query it with rate() to get the per-second increase. A Gauge is for current values that go up and down, such as memory usage or queued jobs. Values whose distribution matters, such as response time, are recorded as a Histogram so percentiles can be calculated.
How long can Prometheus keep data?
Retention is a setting, but Prometheus stores data on local disk, so it is not suited to long-term retention or to combining data from several instances. For months of retention or high availability, use remote write to long-term storage such as Thanos or Grafana Mimir.