Skip to main content
AIDevOps
  • Learn
  • Learning Paths
  • Practice
  • Open Source
  • Books
  • Engineering

    AI DevOpsThe full map of building and operating AI servicesLLMOpsLLM deployment Β· evaluation Β· observabilityHands-on ProjectsBuild AI Agent projects

    Knowledge

    DocsTechnical documentationBlogEngineering articlesPloggerDevelopment log feed

    Validate

    Certification3-level skills certification Β· coming soon
AI Models
LlamaTutorial|MistralTutorial|GemmaTutorial|DeepSeekTutorial|QwenTutorial
🐳 DevOps
DevOps Intro & RoadmapLinuxDockerCI/CD|Kubernetes BasicsK8s AdvancedPrometheusGrafana
πŸ€– AI Applied Development
Applied AI Intro & RoadmapHugging FaceLangChainLlamaIndexLLMOps|LangGraphMCPMulti-AgentAgent Evaluation
🧠 AI Core
AI Intro & RoadmapML FundamentalsLLM Fundamentals|Python AIC++|PyTorchTensorFlowJAX
🧠 AI Agent Development
Finance AI AgentLLM API ServerStock Investing AgentAIOps AI AgentEducation AI AgentCoding AI Agent
🌱 Spring Cloud
Spring Intro & RoadmapSpring Cloud GatewaySpring BootJava|Spring AISpring SecuritySpring BatchSpring JPA
🧱 Infrastructure
Infrastructure Intro & RoadmapNginxRedis
☁️ Cloud
Cloud Intro & RoadmapAWSGCPAzureNCPCloudflare
🎨 Frontend
Frontend Intro & RoadmapJavaScriptTypeScript|ReactNext.js|VueNuxt|Electron
πŸ“± Mobile
Mobile Intro & RoadmapKotlinAndroidFlutter
βš™οΈ Backend
Backend Intro & RoadmapPython BasicsFastAPIDjangoFlask|CGoGinNode.js
πŸ’Ύ Database
DB Intro & RoadmapCore SQLOracleMySQLPostgreSQL|MongoDBVector DB
πŸ§ͺ Testing
k6JMeternGrinder
AIDevOps

Engineering AI. From Code to Production.
An engineering learning platform for building and operating AI and AI Agents

Learn

  • All Guides
  • Learning Paths
  • AI Tutorial
  • Practice
  • Books

Resources

  • AI DevOps
  • LLMOps
  • Hands-on Projects
  • Docs
  • Blog
  • Plogger
  • Open Source
  • Certification (coming soon)

Start Here

  • AI Core Roadmap
  • AI Applied Development Roadmap
  • Spring Cloud Roadmap
  • DevOps Roadmap
  • Infrastructure Roadmap

Β 

  • Cloud Roadmap
  • Frontend Roadmap
  • Mobile Roadmap
  • Backend Roadmap
  • Database Roadmap
Β© 2026 AIDevOps. All rights reserved.
Terms of ServicePrivacy PolicySitemaptestforge.kr
  1. Home
  2. Learn
  3. DevOps
  4. Kubernetes Advanced
Production operations & advanced guide

☸️ Kubernetes Advanced Complete Guide

Visitors

HPA/VPA autoscaling, RBAC security, StatefulSet, DaemonSet, and CronJob workloads, AI model operations strategies such as deploying GPU-based LLM inference servers, canary rollouts, and queue-based autoscaling, upgrades and failure recovery, advanced Helm covering chart structure, hooks, and subcharts, moving from Ingress to Contour/Gateway API, Proxy Protocol settings per LB and Contour version and handling internal calls, and monitoring β€” the advanced topics needed to run Kubernetes in production.

  • Advanced
  • Updated Oct 1, 2026
  • ~53 min read
  • 22 sections
  • 77 code examples

Learning Paths that include this guide

This guide is one step in the paths below. Study it together with the steps before and after.

  • Cloud Native Engineer β†’
  • AI Platform Engineer β†’
Autoscaling & zero-downtime deploymentRBAC-based permission managementOperating StatefulSets/DaemonSets/CronJobsLLM inference servers on GPU nodes & AI model operations strategyAdvanced Helm charts & GitOps integrationMoving from Ingress to Contour/Gateway API & policy settingsPreserving the client IP with Proxy Protocol & handling internal calls

Related frameworks & environments

☸️Kubernetes Basicsβ†’πŸ³Dockerβ†’CICI/CDβ†’πŸ“ŠPrometheusβ†’πŸ“ˆGrafanaβ†’

Contents

0 / 25
  1. How to use this guide
  2. Architecture diagrams
  3. HPA & VPA autoscaling
  4. RBAC & security
  5. StatefulSet vs Stateless
  6. Complete guide to StatefulSets
  7. StatefulSet deployment workflow
  8. Real-world samples
  9. Advanced StatefulSets β€” Parallel Β· volume expansion Β· snapshots
  10. Complete guide to DaemonSets
  11. Complete guide to CronJobs & Jobs
  12. Deploying a Java Agent StatefulSet
  13. GPU workloads & deploying LLM inference servers
  14. AI model deployment strategy & operations
  15. Operations & upgrades
  16. Helm charts
  17. Contour & Gateway API β€” replacing NGINX Ingress
  18. Proxy Protocol β€” preserving the client IP
  19. Proxy Protocol settings per LB
  20. Envoy settings per Contour version (1.28-1.33)
  21. Internal calls with no PROXY header & alternatives
  22. Proxy Protocol verification & troubleshooting
  23. Contour/Envoy monitoring β€” Prometheus & Grafana
  24. Frequently asked questions
  25. Next steps
Contents 25 sections
  1. How to use this guide
  2. Architecture diagrams
  3. HPA & VPA autoscaling
  4. RBAC & security
  5. StatefulSet vs Stateless
  6. Complete guide to StatefulSets
  7. StatefulSet deployment workflow
  8. Real-world samples
  9. Advanced StatefulSets β€” Parallel Β· volume expansion Β· snapshots
  10. Complete guide to DaemonSets
  11. Complete guide to CronJobs & Jobs
  12. Deploying a Java Agent StatefulSet
  13. GPU workloads & deploying LLM inference servers
  14. AI model deployment strategy & operations
  15. Operations & upgrades
  16. Helm charts
  17. Contour & Gateway API β€” replacing NGINX Ingress
  18. Proxy Protocol β€” preserving the client IP
  19. Proxy Protocol settings per LB
  20. Envoy settings per Contour version (1.28-1.33)
  21. Internal calls with no PROXY header & alternatives
  22. Proxy Protocol verification & troubleshooting
  23. Contour/Envoy monitoring β€” Prometheus & Grafana
  24. Frequently asked questions
  25. Next steps

How to use this guide

How to read it

Understanding Kubernetes Advanced as a real-world workflow

HPA/VPA autoscaling, RBAC security, StatefulSet, DaemonSet, and CronJob workloads, AI model operations strategies such as deploying GPU-based LLM inference servers, canary rollouts, and queue-based autoscaling, upgrades and failure recovery, advanced Helm covering chart structure, hooks, and subcharts, moving from Ingress to Contour/Gateway API, Proxy Protocol settings per LB and Contour version and handling internal calls, and monitoring β€” the advanced topics needed to run Kubernetes in production. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.

Key perspective

Infrastructure / operations

Rather than memorizing install commands, understand the order in which traffic, runtime, observability, and incident response connect.

Autoscaling & zero-downtime deploymentRBAC-based permission managementOperating StatefulSets/DaemonSets/CronJobsLLM inference servers on GPU nodes & AI model operations strategyAdvanced Helm charts & GitOps integrationMoving from Ingress to Contour/Gateway API & policy settingsPreserving the client IP with Proxy Protocol & handling internal calls

Architecture diagrams

To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning Kubernetes Advanced.

Learning flow

Rendering diagram…

Architecture view

Rendering diagram…

HPA & VPA autoscaling

When you first open Kubernetes Advanced, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.

A service with uneven traffic should use autoscalers rather than fixed replicas, adjusting the number of Pods (HPA) or the container resources (VPA) automatically to match the load. Turning both on for the same resource with CPU/memory targets makes them conflict, so it is safer to run them separately: the HPA on CPU/custom metrics and the VPA for request recommendations only (updateMode: Off).
Rendering diagram…
hpa.yamlYAML
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60        # scale out when CPU exceeds 60%
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageValue: 400Mi
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300 # scale in after observing for 5 minutes β€” prevents scale flapping
      policies:
      - type: Pods
        value: 1
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
BASH
# Install Metrics Server (a required HPA dependency)
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Check the status
kubectl get hpa myapp-hpa --watch
kubectl describe hpa myapp-hpa          # current metric values and scaling events
kubectl top pods                        # live CPU/memory per Pod
vpa.yamlYAML
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: myapp-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  updatePolicy:
    updateMode: "Off"   # only computes recommendations, applied by hand β€” safe for production
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      minAllowed: { cpu: 50m, memory: 64Mi }
      maxAllowed: { cpu: 2, memory: 4Gi }
AutoscalerWhat it adjustsPrerequisiteCaution
HPANumber of Pod replicas (horizontal scaling)Metrics Server or Prometheus AdapterWith custom metrics, a scale-down delay (stabilizationWindow) needs to be set
VPAContainer requests/limits (vertical scaling)The vertical-pod-autoscaler componentupdateMode: Auto causes Pod restarts β€” risky for stateful workloads
Cluster AutoscalerNumber of worker nodesIntegration with the Auto Scaling Group of the cloud providerAdding nodes is triggered only when Pods are unschedulable (Pending)

RBAC & security

Here you look at RBAC & security alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Role-Based Access Control (RBAC) separates API access rights per team and per service. ServiceAccounts are used to control the permissions of processes inside Pods, and Roles/ClusterRoles to separate user and team permissions.
team-rbac.yamlYAML
# Per-team RBAC β€” the dev team gets read-only access to the production namespace
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dev-team-reader
  namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-only
  namespace: production
rules:
- apiGroups: [""]
  resources: ["pods", "services", "endpoints", "configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets", "statefulsets"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-team-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: dev-team-reader
  namespace: production
roleRef:
  kind: Role
  name: read-only
  apiGroup: rbac.authorization.k8s.io
ops-clusterrole.yamlYAML
# Ops team ClusterRole β€” management rights across all namespaces
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ops-admin
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
- nonResourceURLs: ["*"]
  verbs: ["*"]
---
# ServiceAccount dedicated to the CI/CD system (updates Deployments only)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cicd-deployer
rules:
- apiGroups: ["apps"]
  resources: ["deployments", "statefulsets"]
  verbs: ["get", "list", "patch", "update"]
- apiGroups: [""]
  resources: ["services", "configmaps"]
  verbs: ["get", "list", "create", "update", "patch"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cicd-deployer
  namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cicd-deployer-binding
subjects:
- kind: ServiceAccount
  name: cicd-deployer
  namespace: production
roleRef:
  kind: ClusterRole
  name: cicd-deployer
  apiGroup: rbac.authorization.k8s.io
kubeconfig-user.shBASH
# Create a kubeconfig for an external user (developer)
# 1. Generate the private key & CSR
openssl genrsa -out devuser.key 4096
openssl req -new -key devuser.key -out devuser.csr \
  -subj "/CN=devuser/O=dev-team"

# 2. Submit a K8s CertificateSigningRequest
cat > devuser-csr.yaml << EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: devuser
spec:
  request: $(base64 -w 0 devuser.csr)
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 7776000   # 90 days
  usages:
  - client auth
EOF
kubectl apply -f devuser-csr.yaml

# 3. Approve the CSR
kubectl certificate approve devuser

# 4. Extract the certificate & build the kubeconfig
kubectl get csr devuser -o jsonpath='{.status.certificate}' | base64 -d > devuser.crt

kubectl config set-credentials devuser \
  --client-key=devuser.key \
  --client-certificate=devuser.crt \
  --embed-certs=true

kubectl config set-context devuser-context \
  --cluster=kubernetes \
  --user=devuser \
  --namespace=production

# 5. Grant permissions with a RoleBinding
kubectl create rolebinding devuser-read \
  --role=read-only \
  --user=devuser \
  --namespace=production

StatefulSet vs stateless (Deployment)

Here you look at StatefulSet vs stateless (Deployment) alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

The most important criterion when choosing a workload type in Kubernetes is "does the Pod hold state (data) itself?". A Deployment treats Pods as interchangeable replicas, while a StatefulSet gives each Pod a unique ID, a stable network name, and independent storage.
Rendering diagram…
AspectDeployment (Stateless)StatefulSet (Stateful)
Pod nameRandom suffix (app-7d4f9c-xkz)Fixed ordinal (app-0, app-1, app-2)
Pod replacementCreated and deleted in any orderOrdered (created 0β†’1β†’2, deleted 2β†’1β†’0)
Network IDShared Service ClusterIP, changing Pod IPsFixed DNS such as app-0.svc, app-1.svc through a Headless Service
StorageData is lost when the Pod is deleted (no PVC guarantee)volumeClaimTemplates β†’ a dedicated PVC kept for each Pod
Scale outImmediate parallel expansion, no orderingSequential expansion in the order 0, 1, 2
Rolling updateRandom replacement (maxSurge/maxUnavailable)Sequential replacement in reverse order (2β†’1β†’0)
Main usesAPI servers, web apps, AI inference servers, and so onDatabases (PostgreSQL, MySQL), message brokers (Kafka), distributed caches (Redis Cluster)
PodDisruptionBudgetOptionalStrongly recommended in production

Complete guide to StatefulSets

Here you look at Complete guide to StatefulSets alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Using the example of deploying PostgreSQL as a StatefulSet, this section goes step by step through the Headless Service, volumeClaimTemplates, stable network IDs, and operational patterns.
postgres-statefulset.yamlYAML
# 1. Headless Service β€” stable DNS per Pod (postgres-0.postgres-svc, postgres-1.postgres-svc)
apiVersion: v1
kind: Service
metadata:
  name: postgres-svc
  labels:
    app: postgres
spec:
  clusterIP: None          # Headless: registers Pod IPs directly in DNS
  selector:
    app: postgres
  ports:
  - port: 5432
    name: postgres
---
# 2. StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-svc   # matches the Headless Service name
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:16-alpine
        ports:
        - containerPort: 5432
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        resources:
          requests: { cpu: 250m, memory: 512Mi }
          limits:   { cpu: 1,    memory: 2Gi   }
        volumeMounts:
        - name: postgres-data
          mountPath: /var/lib/postgresql/data
        readinessProbe:
          exec:
            command: ["pg_isready", "-U", "postgres"]
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          exec:
            command: ["pg_isready", "-U", "postgres"]
          initialDelaySeconds: 30
          periodSeconds: 20
  # 3. volumeClaimTemplates β€” an independent PVC created automatically per Pod
  volumeClaimTemplates:
  - metadata:
      name: postgres-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: standard
      resources:
        requests:
          storage: 10Gi
ops-commands.shBASH
# Deploy
kubectl apply -f postgres-statefulset.yaml

# Check the Pod status (ordered: postgres-1 is created after postgres-0 is Running)
kubectl get pods -l app=postgres -w

# Check the fixed DNS per Pod
# postgres-0.postgres-svc.default.svc.cluster.local
# postgres-1.postgres-svc.default.svc.cluster.local
kubectl exec -it postgres-0 -- psql -U postgres -c "SELECT inet_server_addr();"

# The PVC remains even after the Pod is deleted
kubectl delete pod postgres-2
kubectl get pvc | grep postgres      # postgres-data-postgres-2 is still there

# Scale down (deleted in reverse order 2β†’1β†’0)
kubectl scale statefulset postgres --replicas=1

# Rolling update (updateStrategy: RollingUpdate, reverse order 2β†’1β†’0)
kubectl set image statefulset/postgres postgres=postgres:17-alpine
kubectl rollout status statefulset/postgres

# Restart one specific Pod only (PVC kept)
kubectl delete pod postgres-1
pdb.yamlYAML
# PodDisruptionBudget β€” keep at least 2 during voluntary disruptions such as node drains
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: postgres-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: postgres
kafka-statefulset.yamlYAML
# Kafka StatefulSet example β€” a unique ID and independent storage per broker
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka-svc
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - name: kafka
        image: bitnami/kafka:3.7
        env:
        - name: KAFKA_BROKER_ID            # the broker ID is derived from the Pod ordinal
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['statefulset.kubernetes.io/pod-name']
        - name: KAFKA_ADVERTISED_LISTENERS
          value: "PLAINTEXT://$(POD_NAME).kafka-svc:9092"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
        - name: kafka-data
          mountPath: /bitnami/kafka
  volumeClaimTemplates:
  - metadata:
      name: kafka-data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 20Gi

StatefulSet deployment workflow

Here you look at StatefulSet deployment workflow alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

A 6-step procedure for deploying a StatefulSet to production safely. Proceed in the order Namespace β†’ StorageClass β†’ Secret β†’ Headless Service β†’ StatefulSet β†’ PodDisruptionBudget.
01-namespace.shBASH
# 1. Set up the Namespace and ResourceQuota
kubectl create namespace database

kubectl apply -f - <<EOF
apiVersion: v1
kind: ResourceQuota
metadata:
  name: db-quota
  namespace: database
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    persistentvolumeclaims: "10"
EOF
02-storageclass.yamlYAML
# 2. StorageClass β€” SSD-based dynamic provisioning
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/gce-pd   # GKE: gce-pd | EKS: ebs.csi.aws.com | NCP: nks-block-storage
parameters:
  type: pd-ssd
  replication-type: regional-pd
reclaimPolicy: Retain               # keep the disk even after the Pod/PVC is deleted
allowVolumeExpansion: true          # allow online volume expansion
volumeBindingMode: WaitForFirstConsumer
03-secret.shBASH
# 3. Create the Secret (base64 encoding handled automatically)
kubectl create secret generic postgres-secret \
  --from-literal=POSTGRES_PASSWORD="$(openssl rand -base64 24)" \
  --from-literal=REPLICATION_PASSWORD="$(openssl rand -base64 24)" \
  --namespace=database

# Verify
kubectl get secret postgres-secret -n database -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d
04-deploy.shBASH
# 4. Deploy in the order Headless Service β†’ StatefulSet (the order is mandatory)
kubectl apply -f postgres-headless-svc.yaml -n database
kubectl apply -f postgres-statefulset.yaml   -n database

# 5. Monitor the Pod startup order (postgres-1 starts after postgres-0 is Running)
kubectl get pods -n database -l app=postgres -w

# Check the Ready state
kubectl rollout status statefulset/postgres -n database

# 6. Apply the PodDisruptionBudget
kubectl apply -f postgres-pdb.yaml -n database

# Check all resources
kubectl get all,pvc,pdb -n database -l app=postgres
05-scale-update.shBASH
# ── Scale up/down ─────────────────────────────────────────────
# Scale up: 3 β†’ 5 (created sequentially in the order 3, 4)
kubectl scale statefulset postgres --replicas=5 -n database
kubectl get pods -n database -l app=postgres -w

# Scale down: 5 β†’ 3 (deleted in the reverse order 4, 3; PVCs are kept)
kubectl scale statefulset postgres --replicas=3 -n database

# PVCs are not deleted β€” if manual cleanup is needed
kubectl delete pvc postgres-data-postgres-3 -n database
kubectl delete pvc postgres-data-postgres-4 -n database

# ── Rolling update ────────────────────────────────────────────
# Update the image (reverse order: pod-2 β†’ pod-1 β†’ pod-0)
kubectl set image statefulset/postgres \
  postgres=postgres:17-alpine -n database

# Update progress
kubectl rollout status statefulset/postgres -n database

# Pause the update (canary deployment pattern)
kubectl patch statefulset postgres -n database \
  -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'
# partition=2 β†’ only pod-2 gets the new version; pod-0 and pod-1 stay
# After verifying, update everything with partition=0

# ── Rollback ─────────────────────────────────────────────────
kubectl rollout undo statefulset/postgres -n database
kubectl rollout history statefulset/postgres -n database
06-disaster-recovery.shBASH
# ── Failure recovery scenarios ───────────────────────────────
# Force-restart a Pod (PVC kept, data preserved)
kubectl delete pod postgres-1 -n database
kubectl get pods -n database -w    # confirm automatic recreation

# Diagnose when a Pod is Pending
kubectl describe pod postgres-0 -n database
kubectl get events -n database --sort-by='.lastTimestamp'

# Check the PVC volume status
kubectl get pvc -n database
kubectl describe pvc postgres-data-postgres-0 -n database

# Force Pod rescheduling on node failure (when the taint cannot be removed)
kubectl delete pod postgres-0 -n database --grace-period=0 --force

# View the logs of a specific Pod (including the previous container logs)
kubectl logs postgres-0 -n database --previous
kubectl logs postgres-0 -n database -f

Real-world samples

Here you look at Real-world samples alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Three production patterns: Redis Sentinel, MySQL Primary-Replica, and Elasticsearch. Each sample is built on the Headless Service + StatefulSet + volumeClaimTemplates structure.
redis-sentinel.yamlYAML
# Redis Sentinel β€” HA layout (1 Primary + 2 Replicas)
# Headless Service
apiVersion: v1
kind: Service
metadata:
  name: redis-svc
  namespace: cache
spec:
  clusterIP: None
  selector:
    app: redis
  ports:
  - port: 6379
    name: redis
  - port: 26379
    name: sentinel
---
# ClusterIP Service for clients (connects to the Primary only)
apiVersion: v1
kind: Service
metadata:
  name: redis-primary
  namespace: cache
spec:
  selector:
    app: redis
    role: primary
  ports:
  - port: 6379
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  namespace: cache
spec:
  serviceName: redis-svc
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
      - name: init-redis
        image: redis:7-alpine
        command:
        - bash
        - -c
        - |
          # pod-0 is configured as the Primary, the rest as Replicas
          [[ $(hostname) == *-0 ]] && ROLE=primary || ROLE=replica
          if [[ $ROLE == primary ]]; then
            cp /mnt/config/redis-primary.conf /etc/redis/redis.conf
          else
            # inject the primary hostname
            sed "s/PRIMARY_HOST/redis-0.redis-svc/" \
              /mnt/config/redis-replica.conf > /etc/redis/redis.conf
          fi
        volumeMounts:
        - name: config-template
          mountPath: /mnt/config
        - name: redis-config
          mountPath: /etc/redis
      containers:
      - name: redis
        image: redis:7-alpine
        command: ["redis-server", "/etc/redis/redis.conf"]
        ports:
        - containerPort: 6379
        resources:
          requests: { cpu: 100m, memory: 256Mi }
          limits:   { cpu: 500m, memory: 1Gi }
        volumeMounts:
        - name: redis-data
          mountPath: /data
        - name: redis-config
          mountPath: /etc/redis
        readinessProbe:
          exec:
            command: ["redis-cli", "ping"]
          initialDelaySeconds: 5
          periodSeconds: 5
      - name: sentinel
        image: redis:7-alpine
        command: ["redis-sentinel", "/etc/sentinel/sentinel.conf"]
        ports:
        - containerPort: 26379
        volumeMounts:
        - name: sentinel-config
          mountPath: /etc/sentinel
      volumes:
      - name: config-template
        configMap:
          name: redis-config
      - name: redis-config
        emptyDir: {}
      - name: sentinel-config
        configMap:
          name: sentinel-config
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 5Gi
mysql-primary-replica.yamlYAML
# MySQL Primary-Replica β€” roles branch automatically in an initContainer
apiVersion: v1
kind: Service
metadata:
  name: mysql-svc
  namespace: database
spec:
  clusterIP: None
  selector:
    app: mysql
  ports:
  - port: 3306
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-read
  namespace: database
spec:
  selector:
    app: mysql
  ports:
  - port: 3306
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: database
spec:
  serviceName: mysql-svc
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      initContainers:
      - name: init-mysql
        image: mysql:8.4
        command:
        - bash
        - -c
        - |
          # server-id is derived from the Pod ordinal; pod-0 is the Primary
          ORDINAL=$(hostname | awk -F- '{print $NF}')
          echo [mysqld] > /mnt/conf.d/server-id.cnf
          echo server-id=$((100 + $ORDINAL)) >> /mnt/conf.d/server-id.cnf
          if [[ $ORDINAL -eq 0 ]]; then
            cp /mnt/config/primary.cnf /mnt/conf.d/
          else
            cp /mnt/config/replica.cnf /mnt/conf.d/
          fi
        volumeMounts:
        - name: conf
          mountPath: /mnt/conf.d
        - name: config-map
          mountPath: /mnt/config
      - name: clone-mysql
        image: gcr.io/google-samples/xtrabackup:1.0
        command:
        - bash
        - -c
        - |
          # if this is not pod-0, clone the data from the previous Pod
          ORDINAL=$(hostname | awk -F- '{print $NF}')
          [[ $ORDINAL -eq 0 ]] && exit 0
          PEER=mysql-$(($ORDINAL - 1)).mysql-svc
          ncat --recv-only $PEER 3307 | xbstream -x -C /var/lib/mysql
          xtrabackup --prepare --target-dir=/var/lib/mysql
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
      containers:
      - name: mysql
        image: mysql:8.4
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: ROOT_PASSWORD
        ports:
        - containerPort: 3306
        resources:
          requests: { cpu: 500m, memory: 1Gi }
          limits:   { cpu: 2,    memory: 4Gi }
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
        readinessProbe:
          exec:
            command: ["mysqladmin", "ping", "-uroot", "-p$(MYSQL_ROOT_PASSWORD)"]
          initialDelaySeconds: 30
          periodSeconds: 10
      - name: xtrabackup
        image: gcr.io/google-samples/xtrabackup:1.0
        ports:
        - containerPort: 3307
        command:
        - bash
        - -c
        - |
          # wait for Replica clone requests (ncat server)
          cd /var/lib/mysql
          if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then
            cat xtrabackup_slave_info | sed -E 's/;*s*$//; s/MASTER/SOURCE/' \
              | mysql -uroot -p${MYSQL_ROOT_PASSWORD}
            rm -f xtrabackup_slave_info xtrabackup_binlog_info
          fi
          exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
            "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root --password=${MYSQL_ROOT_PASSWORD}"
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
      volumes:
      - name: conf
        emptyDir: {}
      - name: config-map
        configMap:
          name: mysql-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 20Gi
elasticsearch.yamlYAML
# Elasticsearch StatefulSet β€” 3-node cluster
apiVersion: v1
kind: Service
metadata:
  name: es-svc
  namespace: search
spec:
  clusterIP: None
  selector:
    app: elasticsearch
  ports:
  - port: 9200
    name: http
  - port: 9300
    name: transport
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: search
spec:
  serviceName: es-svc
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: es-data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
        env:
        - name: cluster.name
          value: "k8s-es-cluster"
        - name: node.name
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: discovery.seed_hosts
          value: "elasticsearch-0.es-svc,elasticsearch-1.es-svc,elasticsearch-2.es-svc"
        - name: cluster.initial_master_nodes
          value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"
        - name: ES_JAVA_OPTS
          value: "-Xms2g -Xmx2g"
        - name: xpack.security.enabled
          value: "true"
        - name: ELASTIC_PASSWORD
          valueFrom:
            secretKeyRef:
              name: es-secret
              key: password
        ports:
        - containerPort: 9200
          name: http
        - containerPort: 9300
          name: transport
        resources:
          requests: { cpu: 500m, memory: 3Gi }
          limits:   { cpu: 2,    memory: 5Gi }
        volumeMounts:
        - name: es-data
          mountPath: /usr/share/elasticsearch/data
        readinessProbe:
          httpGet:
            path: /_cluster/health?local=true
            port: 9200
            httpHeaders:
            - name: Authorization
              value: "Basic ZWxhc3RpYzpjaGFuZ2VtZQ=="
          initialDelaySeconds: 30
          periodSeconds: 10
  volumeClaimTemplates:
  - metadata:
      name: es-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 30Gi
verify-all.shBASH
# ── Common deployment verification script ────────────────────

# 1. Confirm all Pods are Running
kubectl get pods -A -l app in (postgres,redis,mysql,elasticsearch) \
  --field-selector=status.phase!=Running

# 2. Confirm the PVCs are Bound
kubectl get pvc -A | grep -v Bound

# 3. Confirm the StatefulSet rollout finished
for sts in postgres redis mysql elasticsearch; do
  echo "=== $sts ==="
  kubectl rollout status statefulset/$sts --timeout=120s 2>/dev/null || true
done

# 4. Check endpoint connectivity
# PostgreSQL
kubectl exec -it postgres-0 -n database -- \
  psql -U postgres -c "SELECT version();"

# Redis
kubectl exec -it redis-0 -n cache -- redis-cli ping
kubectl exec -it redis-0 -n cache -- \
  redis-cli info replication | grep -E "role|connected_slaves"

# Elasticsearch
kubectl exec -it elasticsearch-0 -n search -- \
  curl -s -u elastic:$ELASTIC_PASSWORD \
  http://localhost:9200/_cluster/health?pretty | jq '.status'

# 5. Resource usage
kubectl top pods -A -l app in (postgres,redis,mysql,elasticsearch)

Advanced StatefulSets β€” Parallel Β· volume expansion Β· snapshots

Here you look at Advanced StatefulSets β€” Parallel Β· volume expansion Β· snapshots alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Once you are comfortable with basic StatefulSet operations, you need the practical patterns of controlling startup order with podManagementPolicy, expanding PVCs with no downtime, and backing up and restoring data with VolumeSnapshots. In particular, the constraint that volumeClaimTemplates cannot be modified directly after the StatefulSet is created is a frequent source of mistakes.
statefulset-parallel.yamlYAML
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
spec:
  serviceName: elasticsearch-svc
  replicas: 5
  podManagementPolicy: Parallel   # start all 5 Pods at once β€” faster scaling with no waiting on order
  selector:
    matchLabels: { app: elasticsearch }
  template:
    metadata:
      labels: { app: elasticsearch }
    spec:
      containers:
      - name: elasticsearch
        image: elasticsearch:8.13.0
        env:
        - name: discovery.seed_hosts
          value: "elasticsearch-0.elasticsearch-svc,elasticsearch-1.elasticsearch-svc"
        - name: cluster.initial_master_nodes
          value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"
        volumeMounts:
        - name: es-data
          mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata: { name: es-data }
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources: { requests: { storage: 100Gi } }
pvc-online-expand.shBASH
# ── Online PVC expansion (requires allowVolumeExpansion: true on the StorageClass) ──
# ⚠️ volumeClaimTemplates cannot be patched/edited directly after the StatefulSet is created (immutable field)
# β†’ the PVCs already created have to be patched individually

# 1. Expand the capacity of each PVC directly (10Gi β†’ 50Gi)
for i in 0 1 2; do
  kubectl patch pvc postgres-data-postgres-$i -n database \
    -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'
done

# 2. Check the expansion progress (FileSystemResizePending β†’ disappears when done)
kubectl get pvc -n database -w
kubectl describe pvc postgres-data-postgres-0 -n database | grep -A3 Conditions

# 3. Bring the volumeClaimTemplates capacity in the StatefulSet manifest into line as well
#    (so new PVCs are created at the same size on the next recreation/new replica)
#    The spec.volumeClaimTemplates field itself cannot be modified, so when a change is
#    needed, recreate only the StatefulSet while preserving the Pods with --cascade=orphan
kubectl delete statefulset postgres -n database --cascade=orphan
kubectl apply -f postgres-statefulset.yaml -n database   # reapply after changing the capacity to 50Gi
volumesnapshot-backup.yamlYAML
# ── VolumeSnapshot-based backup & restore (requires a CSI driver + VolumeSnapshotClass) ──
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-snapclass
driver: ebs.csi.aws.com          # GKE: pd.csi.storage.gke.io | AKS: disk.csi.azure.com
deletionPolicy: Retain            # keep the snapshot even if the original PVC is deleted
---
# 1. Create a snapshot β€” a backup before a deployment / large migration
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-snap-before-upgrade
  namespace: database
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: postgres-data-postgres-0
---
# 2. Restore a new PVC from the snapshot β€” on failure, restore under a separate name, verify the data, then swap
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data-restored
  namespace: database
spec:
  storageClassName: fast-ssd
  dataSource:
    name: postgres-snap-before-upgrade
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 50Gi
snapshot-ops.shBASH
# Check the snapshot status (wait until readyToUse: true)
kubectl get volumesnapshot -n database -w
kubectl describe volumesnapshot postgres-snap-before-upgrade -n database

# Mount the restored PVC in a verification Pod to check the data, then swap it for the existing PVC if all is well
kubectl run pg-verify --image=postgres:16-alpine --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"pg-verify","image":"postgres:16-alpine","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}],"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"postgres-data-restored"}}]}}' \
  -n database
podManagementPolicyBehaviorSuitable workloads
OrderedReady (default)Pod-1 is created only after Pod-0 is Running & Ready. Deletion is one at a time too, in reverse orderWhere startup order affects data consistency, as with PostgreSQL Primary-Replica
ParallelCreates/deletes all Pods at the same time. No ordering guarantee, but far fasterWhere each node joins the cluster independently, as with Elasticsearch, Cassandra, and Kafka

Complete guide to DaemonSets

Here you look at Complete guide to DaemonSets alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

A DaemonSet deploys exactly one Pod on every node in the cluster (or every node selected by label). When a new node is added a Pod is created automatically, and when a node is removed the Pod is cleaned up with it. It suits infrastructure components that "need one per node", such as log collectors, monitoring agents, and CNI/CSI plugins.
Rendering diagram…
fluent-bit-daemonset.yamlYAML
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels: { app: fluent-bit }
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1        # restart only 1 node at a time β€” minimizes the window of lost logs
  template:
    metadata:
      labels: { app: fluent-bit }
    spec:
      tolerations:              # deploy to control-plane nodes too, to collect the master logs as well
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      serviceAccountName: fluent-bit
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:3.0
        resources:
          requests: { cpu: 100m, memory: 128Mi }   # multiplied by the node count, so keep it minimal
          limits:   { cpu: 300m, memory: 256Mi }
        volumeMounts:
        - name: varlog
          mountPath: /var/log            # mount the log directory of the node directly
          readOnly: true
        - name: dockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath: { path: /var/log }
      - name: dockercontainers
        hostPath: { path: /var/lib/docker/containers }
daemonset-ops.shBASH
# Check the rollout per node β€” READY must equal the number of nodes
kubectl get daemonset fluent-bit -n logging
kubectl get pods -n logging -l app=fluent-bit -o wide

# Restrict it to run only on certain nodes (e.g. deploy the device plugin only on GPU nodes)
kubectl label node gpu-node-1 workload=gpu
# add spec.template.spec.nodeSelector: { workload: gpu } to the manifest

# Rolling update
kubectl set image daemonset/fluent-bit fluent-bit=fluent/fluent-bit:3.1 -n logging
kubectl rollout status daemonset/fluent-bit -n logging
kubectl rollout history daemonset/fluent-bit -n logging

# Confirm automatic scheduling when a new node is added
kubectl get nodes -w
TraitDaemonSetDeployment
Setting replicasNot possible β€” decided automatically by the number of target nodesSpecified explicitly
SchedulingExactly 1 per node; target nodes can be restricted with nodeSelector/affinityThe scheduler places Pods freely on nodes with spare resources
Deploying to control-plane nodesDeployed only if tolerations allow the NoSchedule taintExcluded by default
Update strategyRollingUpdate (maxUnavailable) or OnDeleteRollingUpdate (maxSurge/maxUnavailable) or Recreate
Typical examplesFluent Bit, Node Exporter, Calico/Cilium, NVIDIA device pluginGeneral applications such as web servers and API servers

Complete guide to CronJobs & Jobs

Here you look at Complete guide to CronJobs & Jobs alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

A Job runs a one-off task whose completion must be guaranteed (batch jobs, migrations), and a CronJob creates that Job periodically on a cron schedule. Unless you clearly define the retry count on failure (backoffLimit), the maximum run time (activeDeadlineSeconds), and the concurrency policy (concurrencyPolicy), batch jobs end up in incidents where they retry forever or run in duplicate.
db-migration-job.yamlYAML
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration-v2
  namespace: production
spec:
  backoffLimit: 3                 # give up after 3 failures (prevents endless retries)
  activeDeadlineSeconds: 600       # force-terminate after 10 minutes
  ttlSecondsAfterFinished: 3600    # clean up automatically 1 hour after completion
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: migrate
        image: myapp-migrator:2.1.0
        command: ["./migrate.sh", "--target=v2"]
        resources:
          requests: { cpu: 200m, memory: 256Mi }
          limits:   { cpu: 500m, memory: 512Mi }
        envFrom:
        - secretRef: { name: db-credentials }
nightly-backup-cronjob.yamlYAML
apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-nightly-backup
  namespace: database
spec:
  schedule: "0 3 * * *"            # every day at 3 AM (in the time zone of the cluster kube-controller-manager)
  timeZone: "America/New_York"     # K8s 1.27+ β€” the time zone can be set on the CronJob itself
  concurrencyPolicy: Forbid        # skip this run if the previous backup is still running
  startingDeadlineSeconds: 300     # give up on this run if it cannot start within 5 minutes
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5        # keep failure history a little longer for root cause analysis
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 1800
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: backup
            image: postgres:16-alpine
            command:
            - sh
            - -c
            - |
              pg_dump -h postgres-svc -U postgres mydb | \
              gzip > /backup/mydb-$(date +%Y%m%d).sql.gz
            volumeMounts:
            - name: backup-storage
              mountPath: /backup
          volumes:
          - name: backup-storage
            persistentVolumeClaim: { claimName: backup-pvc }
job-cronjob-ops.shBASH
# Run once manually, right now, from the CronJob (test without waiting for the schedule)
kubectl create job --from=cronjob/postgres-nightly-backup manual-backup-test -n database

# Check the run history
kubectl get cronjob postgres-nightly-backup -n database
kubectl get jobs -n database -l job-name  --sort-by=.metadata.creationTimestamp

# View the Pod logs of a specific Job
kubectl logs -n database -l job-name=postgres-nightly-backup-28912345 --tail=100

# Suspend/resume the CronJob (during a deployment or an incident investigation)
kubectl patch cronjob postgres-nightly-backup -n database -p '{"spec":{"suspend":true}}'
kubectl patch cronjob postgres-nightly-backup -n database -p '{"spec":{"suspend":false}}'

# Clean up finished Jobs (manual cleanup when ttlSecondsAfterFinished is not set)
kubectl delete job --field-selector status.successful=1 -n database
FieldApplies toRole
backoffLimitJobMaximum number of retries on failure (default 6) β€” when exceeded, the Job ends as Failed
activeDeadlineSecondsJobTime limit for the whole run β€” force-terminated when exceeded, prevents endless runs
restartPolicyJob PodNever (a new Pod is created on retry) or OnFailure (the same Pod is restarted) β€” Always is not allowed
ttlSecondsAfterFinishedJobDeletes the Job/Pod automatically N seconds after completion β€” stops finished Jobs from piling up
concurrencyPolicyCronJobAllow (default, overlap permitted) / Forbid (skip if the previous run is still going) / Replace (cancel the previous run and start anew)
startingDeadlineSecondsCronJobHow many seconds of delayed start to allow when the scheduled time is missed (controller down and so on)
successfulJobsHistoryLimit / failedJobsHistoryLimitCronJobNumber of successful/failed Jobs to keep in history β€” only the last few are kept for debugging

Deploying a Java Agent StatefulSet

Here you look at Deploying a Java Agent StatefulSet alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

A hands-on guide to deploying a Spring Boot based Java AI Agent as a StatefulSet. Two PVCs are provisioned for each Pod: one storing the Agent state and vector index, and one keeping the audit logs separately.
DockerfileDOCKERFILE
# ── Multi-stage build β€” Spring Boot AI Agent ──────────────────
FROM eclipse-temurin:21-jdk-alpine AS builder
WORKDIR /app
COPY gradlew build.gradle.kts settings.gradle.kts ./
COPY gradle ./gradle
RUN ./gradlew dependencies --no-daemon -q

COPY src ./src
RUN ./gradlew bootJar --no-daemon -q

# ── Runtime image ─────────────────────────────────────────────
FROM eclipse-temurin:21-jre-alpine
WORKDIR /app

# Create the data and log directories (PVC mount points)
RUN mkdir -p /app/data /app/logs && \
    addgroup -S agent && adduser -S agent -G agent && \
    chown -R agent:agent /app

COPY --from=builder /app/build/libs/*.jar agent.jar
USER agent

# Declare the PVC volumes (mounted by K8s)
VOLUME ["/app/data", "/app/logs"]

EXPOSE 8080
ENTRYPOINT ["java", \
  "-XX:MaxRAMPercentage=75.0", \
  "-XX:+UseG1GC", \
  "-Dspring.profiles.active=prod", \
  "-Dlogging.file.path=/app/logs", \
  "-jar", "agent.jar"]
configmap.yamlYAML
apiVersion: v1
kind: ConfigMap
metadata:
  name: java-agent-config
  namespace: ai-agent
data:
  application-prod.yml: |
    spring:
      datasource:
        url: jdbc:h2:file:/app/data/agentdb;AUTO_SERVER=TRUE
        driver-class-name: org.h2.Driver
        username: agent
      jpa:
        hibernate:
          ddl-auto: update
        show-sql: false
      ai:
        openai:
          base-url: http://llm-gateway-svc:8080   # internal LLM Gateway
          api-key: ${OPENAI_API_KEY}

    agent:
      vector-store:
        path: /app/data/faiss-index          # local FAISS index
        dimension: 1536
      memory:
        persist-path: /app/data/memory.db   # Agent memory store
        max-tokens: 4096

    logging:
      file:
        name: /app/logs/agent.log
        max-size: 100MB
        max-history: 90                       # kept for 90 days
      pattern:
        file: "%d{yyyy-MM-dd HH:mm:ss} [%X{traceId}] %-5level %logger{36} - %msg%n"

    management:
      endpoints:
        web:
          exposure:
            include: health,metrics,prometheus
      metrics:
        export:
          prometheus:
            enabled: true
secret.shBASH
# Create the Secret (API key, DB password)
kubectl create namespace ai-agent

kubectl create secret generic java-agent-secret \
  --from-literal=OPENAI_API_KEY="sk-xxxx" \
  --from-literal=SPRING_DATASOURCE_PASSWORD="$(openssl rand -base64 16)" \
  --namespace=ai-agent

# Container registry credentials (private registry)
kubectl create secret docker-registry regcred \
  --docker-server=registry.aidevops.us \
  --docker-username=deploy \
  --docker-password="<token>" \
  --namespace=ai-agent
statefulset.yamlYAML
# ── Headless Service (DNS between Pods) ──────────────────────
apiVersion: v1
kind: Service
metadata:
  name: java-agent-svc
  namespace: ai-agent
spec:
  clusterIP: None
  selector:
    app: java-agent
  ports:
  - port: 8080
    name: http
---
# ── ClusterIP Service (for internal API calls) ───────────────
apiVersion: v1
kind: Service
metadata:
  name: java-agent-api
  namespace: ai-agent
spec:
  selector:
    app: java-agent
  ports:
  - port: 80
    targetPort: 8080
    name: http
---
# ── StatefulSet ───────────────────────────────────────────────
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: java-agent
  namespace: ai-agent
spec:
  serviceName: java-agent-svc       # Headless Service name
  replicas: 2
  selector:
    matchLabels:
      app: java-agent
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0                  # full rolling update
  template:
    metadata:
      labels:
        app: java-agent
    spec:
      terminationGracePeriodSeconds: 60
      imagePullSecrets:
      - name: regcred
      initContainers:
      # Initialize permissions on the data directory (avoids the root ownership problem on PVC mount)
      - name: init-permissions
        image: busybox:1.36
        command: ["sh", "-c", "chown -R 1000:1000 /app/data /app/logs"]
        volumeMounts:
        - name: agent-state
          mountPath: /app/data
        - name: agent-logs
          mountPath: /app/logs
      containers:
      - name: java-agent
        image: registry.aidevops.us/java-agent:1.0.0
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: java-agent-secret
              key: OPENAI_API_KEY
        - name: SPRING_DATASOURCE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: java-agent-secret
              key: SPRING_DATASOURCE_PASSWORD
        # Use the unique Pod ID as the Agent ID (java-agent-0, java-agent-1)
        - name: AGENT_ID
          value: "$(POD_NAME)"
        envFrom:
        - configMapRef:
            name: java-agent-config   # separate ConfigMap for mounting application-prod.yml
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 2
            memory: 3Gi
        volumeMounts:
        - name: agent-state           # PVC 1 β€” agent state and vector index
          mountPath: /app/data
        - name: agent-logs            # PVC 2 β€” audit/operational logs
          mountPath: /app/logs
        - name: app-config
          mountPath: /app/config
          readOnly: true
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 20
          failureThreshold: 4
        lifecycle:
          preStop:
            exec:
              # Graceful shutdown β€” wait for in-flight Agent tasks to finish
              command: ["sh", "-c", "sleep 10"]
      volumes:
      - name: app-config
        configMap:
          name: java-agent-config
  # ── 2 PVCs β€” volumeClaimTemplates ─────────────────────────
  volumeClaimTemplates:
  - metadata:
      name: agent-state
      labels:
        app: java-agent
        volume: state
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 20Gi
  - metadata:
      name: agent-logs
      labels:
        app: java-agent
        volume: logs
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: nfs-client     # logs can be centralized on NFS
      resources:
        requests:
          storage: 10Gi
ingress.yamlYAML
# Ingress β€” access to the Java Agent API from outside
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: java-agent-ingress
  namespace: ai-agent
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"    # wait for the Agent response
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - agent-api.aidevops.us
    secretName: agent-api-tls
  rules:
  - host: agent-api.aidevops.us
    http:
      paths:
      - path: /api/agent
        pathType: Prefix
        backend:
          service:
            name: java-agent-api
            port:
              number: 80
pdb.yamlYAML
# PodDisruptionBudget β€” always keep at least 1 Pod
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: java-agent-pdb
  namespace: ai-agent
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: java-agent
deploy-and-verify.shBASH
# ── Full deployment order ─────────────────────────────────────
kubectl apply -f configmap.yaml
kubectl apply -f statefulset.yaml    # Headless SVC + ClusterIP SVC + StatefulSet
kubectl apply -f ingress.yaml
kubectl apply -f pdb.yaml

# Check Pod startup (java-agent-1 after java-agent-0 is Running)
kubectl get pods -n ai-agent -l app=java-agent -w

# Confirm 2 PVCs each were created (agent-state + agent-logs per Pod)
kubectl get pvc -n ai-agent
# Expected output:
# agent-state-java-agent-0   Bound   20Gi
# agent-logs-java-agent-0    Bound   10Gi
# agent-state-java-agent-1   Bound   20Gi
# agent-logs-java-agent-1    Bound   10Gi

# Check Actuator health
kubectl exec -it java-agent-0 -n ai-agent -- \
  curl -s http://localhost:8080/actuator/health | jq .

# Check the data directory (FAISS index, H2 DB)
kubectl exec -it java-agent-0 -n ai-agent -- ls -lh /app/data/
kubectl exec -it java-agent-0 -n ai-agent -- ls -lh /app/logs/

# Follow the logs in real time
kubectl logs java-agent-0 -n ai-agent -f

# ── Rolling update ────────────────────────────────────────────
kubectl set image statefulset/java-agent \
  java-agent=registry.aidevops.us/java-agent:1.1.0 \
  -n ai-agent
kubectl rollout status statefulset/java-agent -n ai-agent

# ── Restart a Pod (PVC data preserved) ────────────────────────
kubectl delete pod java-agent-1 -n ai-agent
kubectl get pods -n ai-agent -w   # confirm automatic recreation

# ── Check PVC capacity ────────────────────────────────────────
kubectl exec -it java-agent-0 -n ai-agent -- df -h /app/data /app/logs
PVC nameMount pathPurposeSize
agent-state/app/dataH2 embedded DB, vector index (FAISS), agent memory, file uploads20 Gi
agent-logs/app/logsAudit logs, trace logs (Logback archive, kept 90 days)10 Gi

GPU workloads & deploying LLM inference servers

Here you look at GPU workloads & deploying LLM inference servers alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

AI Agent/LLM serving workloads must request GPU resources precisely and be isolated so they are not scheduled on the same nodes as ordinary CPU Pods. Once the NVIDIA device plugin exposes GPUs as a node resource (nvidia.com/gpu), they can be specified in requests/limits only in whole numbers β€” unlike CPU/memory, fractional GPU requests are not possible.
BASH
# Install the NVIDIA device plugin β€” exposes GPU nodes as the nvidia.com/gpu resource
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/main/deployments/static/nvidia-device-plugin.yml

# Check the GPU nodes
kubectl get nodes -o json | jq '.items[].status.capacity."nvidia.com/gpu"'

# Put a dedicated taint on GPU nodes β€” stops ordinary Pods from being scheduled there by mistake
kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule
llm-inference-deploy.yamlYAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
  namespace: ai-agent
spec:
  replicas: 2
  selector:
    matchLabels: { app: llm-inference }
  template:
    metadata:
      labels: { app: llm-inference }
    spec:
      nodeSelector:
        gpu-type: a10g                 # separate node pools per GPU type and label them
      tolerations:
      - key: nvidia.com/gpu
        operator: Equal
        value: present
        effect: NoSchedule
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args: ["--model", "/models/llama-3-8b", "--gpu-memory-utilization", "0.9"]
        resources:
          requests:
            cpu: "4"
            memory: 24Gi
            nvidia.com/gpu: "1"        # GPUs can only be requested in whole numbers β€” no fractions
          limits:
            cpu: "8"
            memory: 32Gi
            nvidia.com/gpu: "1"
        volumeMounts:
        - name: model-cache
          mountPath: /models
        readinessProbe:                 # block traffic until model loading (tens of seconds to minutes) finishes
          httpGet: { path: /health, port: 8000 }
          initialDelaySeconds: 60
          periodSeconds: 10
          failureThreshold: 30
        livenessProbe:
          httpGet: { path: /health, port: 8000 }
          initialDelaySeconds: 120
          periodSeconds: 30
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: llm-model-pvc      # model weights are mounted from a PVC, not the image
llm-inference-hpa.yamlYAML
# Autoscaling GPU Pods β€” custom metrics based on queue length/concurrent requests are recommended over CPU
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-inference-hpa
  namespace: ai-agent
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-inference
  minReplicas: 1               # GPUs are expensive β€” keep the minimum low
  maxReplicas: 6               # cap at the number of GPUs obtainable in the cluster
  metrics:
  - type: Pods
    pods:
      metric:
        name: inference_queue_length   # custom metric exposed through the Prometheus Adapter
      target:
        type: AverageValue
        averageValue: "5"
ComponentRoleNotes
NVIDIA device plugin (DaemonSet)Exposes GPU nodes as the schedulable resource nvidia.com/gpuDeployed automatically to GPU nodes only
nodeSelector / taint-tolerationSchedules GPU Pods only on GPU nodes and keeps ordinary Pods offPutting a NoSchedule taint on GPU nodes is the usual practice
Model weight storageMounted from a PVC/object storage instead of baked into the container imageSmaller images and shorter deploys; easy to swap model versions
Readiness probeBlocks incoming traffic until model loading finishesLLMs take tens of seconds to minutes to load β€” be generous with initialDelaySeconds

AI model deployment strategy & operations

Here you look at AI model deployment strategy & operations alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

An LLM inference server carries different operational risks from an ordinary web service. GPUs are expensive, so the minimum replica count has to stay low; changing one model means revalidating tens of GB of weights; and CPU/memory metrics alone cannot tell you how many requests are backed up in the queue right now. Progressive model rollout, queue-based autoscaling, and GPU node failure isolation have to be designed separately to suit these traits.
Rendering diagram…
model-canary-route.yaml β€” splitting traffic between two model versions with the Gateway APIYAML
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-inference-route
spec:
  parentRefs:
  - name: main-gateway
  rules:
  - backendRefs:
    - name: llm-inference-stable
      port: 8000
      weight: 95        # verified existing model
    - name: llm-inference-canary
      port: 8000
      weight: 5         # new model β€” raise the weight gradually
keda-scaledobject.yaml β€” autoscaling on queue lengthYAML
# Scale directly on external queue metrics such as Redis/SQS, even without the Prometheus Adapter
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-inference-scaler
  namespace: ai-agent
spec:
  scaleTargetRef:
    name: llm-inference
  minReplicaCount: 1        # GPUs are expensive β€” going down to 0 when idle is worth considering
  maxReplicaCount: 6
  cooldownPeriod: 300        # wait before scaling down β€” keep it generous, since shrinking and reacquiring GPU Pods is costly
  triggers:
  - type: redis
    metadata:
      address: redis.ai-agent.svc:6379
      listName: inference-queue
      listLength: "5"        # scale out when 5 or more pile up in the queue
gpu-pdb.yaml β€” preventing simultaneous eviction of GPU PodsYAML
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: llm-inference-pdb
  namespace: ai-agent
spec:
  minAvailable: 1            # keep at least 1 able to serve even during node maintenance
  selector:
    matchLabels: { app: llm-inference }
Operational itemOrdinary web serviceLLM inference server
Deployment riskSwapping one imageWhen the model weights change, response quality itself can change β€” needs canary verification first
Scaling signalCPU/memory utilization is often enoughScaling on external metrics such as queue length and concurrent requests is more accurate
Recovery cost on failureSeconds for a Pod restartMinutes to reacquire a GPU node and reload the model β€” preventing simultaneous eviction with a PodDisruptionBudget is essential

Tip

  • When verifying a canary model on 5% of real traffic, compare not just response latency but also response quality (the LLM-as-Judge of the agent-evaluation guide) and cost per token, so you do not miss "a model that is faster but answers worse".
  • Using spot/preemptible instances for GPU nodes cuts costs considerably, but they can be reclaimed without notice β€” along with a PodDisruptionBudget, configure node affinity (preferredDuringScheduling) with an on-demand GPU node pool as the fallback, and the service will not go down completely even when spot capacity is reclaimed.

Operations & upgrades

Here you look at Operations & upgrades alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

A kubectl cheat sheet, node maintenance, cluster version upgrades, etcd backup and restore, and failure diagnosis patterns. Day-to-day operations last many times longer than building the cluster, so having practiced in advance the procedure for safely emptying and returning a node, and means of rolling back on failure such as etcd backups, is what determines how fast you respond to a real incident.
kubectl-cheatsheet.shBASH
# ── Essential kubectl commands ───────────────────────────────

# Switch context
kubectl config get-contexts
kubectl config use-context production

# Set the default namespace
kubectl config set-context --current --namespace=production

# Watch Pod status in real time
kubectl get pods -n production -w
kubectl get pods -n production -o wide          # includes node placement

# Logs
kubectl logs <pod> -f                           # live
kubectl logs <pod> --previous                   # previous container logs
kubectl logs <pod> -c <container> --tail=100    # multi-container

# Debugging
kubectl describe pod <pod> -n production
kubectl exec -it <pod> -- /bin/sh
kubectl debug <pod> -it --image=busybox         # ephemeral debug container

# Port forwarding (local testing)
kubectl port-forward svc/postgres-svc 5432:5432 -n database
kubectl port-forward pod/redis-0 6379:6379 -n cache

# Resource usage
kubectl top nodes
kubectl top pods -n production --sort-by=cpu

# View events
kubectl get events -n production --sort-by='.lastTimestamp'

# Force a restart (rolling restart of a Deployment)
kubectl rollout restart deployment/<name> -n production

# Detailed resource output
kubectl get all -n production
kubectl get all,pvc,ingress,cm,secret -n production
node-maintenance.shBASH
# ── Node maintenance (cordon + drain) ────────────────────────

# 1. Block new Pods from being scheduled (existing Pods stay)
kubectl cordon k8s-worker-01

# 2. Move existing Pods to other nodes (respects PodDisruptionBudgets)
kubectl drain k8s-worker-01 \
  --ignore-daemonsets \    # exclude DaemonSet Pods from the drain
  --delete-emptydir-data \  # allow deleting Pods with emptyDir volumes
  --grace-period=30 \      # termination grace period
  --timeout=300s            # overall timeout

# 3. Carry out the maintenance: OS patches, kernel upgrade, disk replacement, and so on

# 4. Resume scheduling once the node is back
kubectl uncordon k8s-worker-01

# 5. Check the node status
kubectl get node k8s-worker-01 -o wide
kubectl describe node k8s-worker-01 | grep -A5 "Conditions:"
cluster-upgrade.shBASH
# ── Cluster version upgrade (1.35 β†’ 1.36, example) ─────────────────
# kubeadm can only move up one minor version at a time β€” no going straight from 1.34 to 1.36; you must pass through 1.35
# Always in the order Control Plane β†’ Worker

# 0. Pre-checks
kubeadm upgrade plan

# 1. Upgrade the Control Plane
# Update kubeadm
apt-mark unhold kubeadm
apt-get install -y kubeadm=1.36.0-00
apt-mark hold kubeadm

# Run the upgrade
kubeadm upgrade apply v1.36.0

# Update kubelet and kubectl
apt-mark unhold kubelet kubectl
apt-get install -y kubelet=1.36.0-00 kubectl=1.36.0-00
apt-mark hold kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet

# 2. Upgrade the Worker nodes (repeat for each node)
# On the Control Plane
kubectl cordon k8s-worker-01
kubectl drain k8s-worker-01 --ignore-daemonsets --delete-emptydir-data

# On the Worker node
apt-mark unhold kubeadm kubelet kubectl
apt-get install -y kubeadm=1.36.0-00 kubelet=1.36.0-00 kubectl=1.36.0-00
apt-mark hold kubeadm kubelet kubectl
kubeadm upgrade node
systemctl daemon-reload && systemctl restart kubelet

# On the Control Plane
kubectl uncordon k8s-worker-01

# 3. Confirm the upgrade finished
kubectl get nodes
etcd-backup.shBASH
# ── etcd backup & restore ────────────────────────────────────
# etcd is the store for all cluster state β€” regular backups are essential

# Backup (on the Control Plane)
ETCD_BACKUP_DIR=/backup/etcd/$(date +%Y%m%d-%H%M%S)
mkdir -p $ETCD_BACKUP_DIR

ETCDCTL_API=3 etcdctl snapshot save $ETCD_BACKUP_DIR/snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify the backup
ETCDCTL_API=3 etcdctl snapshot status $ETCD_BACKUP_DIR/snapshot.db \
  --write-out=table

# Scheduled backup with cron (every day at 2 AM)
echo "0 2 * * * root ETCDCTL_API=3 etcdctl snapshot save /backup/etcd/$(date +%Y%m%d).db   --endpoints=https://127.0.0.1:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key" >> /etc/crontab

# ── Restore procedure ────────────────────────────────────────
# 1. Stop the kubelet on every Control Plane node
systemctl stop kubelet

# 2. Back up the etcd data directory, then delete it
mv /var/lib/etcd /var/lib/etcd.bak

# 3. Restore the snapshot
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd/20240101.db \
  --data-dir=/var/lib/etcd \
  --name=k8s-master-01 \
  --initial-cluster=k8s-master-01=https://192.168.1.10:2380 \
  --initial-cluster-token=etcd-cluster-1 \
  --initial-advertise-peer-urls=https://192.168.1.10:2380

# 4. Restart the kubelet
systemctl start kubelet
kubectl get nodes
troubleshoot.shBASH
# ── Failure diagnosis patterns ───────────────────────────────

# When a Pod is Pending
kubectl describe pod <pod> -n <ns>
# β†’ Check the Events section:
#   "Insufficient cpu"    β†’ ResourceQuota or not enough node resources
#   "No nodes available"  β†’ every node is tainted/cordoned
#   "PVC not found"       β†’ a StorageClass or PV problem

# When a Pod is in CrashLoopBackOff
kubectl logs <pod> --previous -n <ns>
kubectl describe pod <pod> -n <ns>
# β†’ Check the exit code:
#   Exit Code 1: application error (check the logs)
#   Exit Code 137: OOM Kill β†’ raise limits.memory

# When a node is NotReady
kubectl describe node k8s-worker-01
# β†’ Check Conditions: MemoryPressure, DiskPressure, PIDPressure
ssh k8s-worker-01
systemctl status kubelet
journalctl -u kubelet -f

# When a service is unreachable
kubectl get endpoints <service> -n <ns>    # whether Pod IPs are registered
kubectl exec <pod> -- curl http://<service>.<ns>.svc.cluster.local
kubectl exec <pod> -- nslookup <service>   # check DNS

# Image pull failures
kubectl describe pod <pod> | grep -A3 "Failed"
# β†’ ErrImagePull / ImagePullBackOff
#   private registry: check the imagePullSecrets setting
kubectl create secret docker-registry regcred \
  --docker-server=registry.aidevops.us \
  --docker-username=<user> \
  --docker-password=<pass>

# Health check script for the whole cluster
echo "=== Nodes ===" && kubectl get nodes
echo "=== System Pods ===" && kubectl get pods -n kube-system | grep -v Running
echo "=== Failed Pods ===" && kubectl get pods -A | grep -E "Error|CrashLoop|Pending|OOMKilled"
echo "=== PVC ===" && kubectl get pvc -A | grep -v Bound
echo "=== Recent Events ===" && kubectl get events -A --sort-by='.lastTimestamp' | tail -20

Helm charts

Here you look at Helm charts alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Helm bundles several YAML manifests into one deployable unit (a chart) and lets you reuse it by changing only the values that differ per environment (image tag, replica count, number of GPUs, and so on) in a values file. What makes it fundamentally different from repeating kubectl apply is the revision history β€” every time helm upgrade runs, the full set of manifests and values at that moment is recorded as a snapshot, so a single helm rollback line returns everything to the state of an earlier revision.
Rendering diagram…
Chart directory structureTEXT
llm-inference-chart/
β”œβ”€β”€ Chart.yaml           # chart name, version, and dependency (subchart) declarations
β”œβ”€β”€ values.yaml          # defaults β€” the starting point for every environment
β”œβ”€β”€ values-staging.yaml   # staging-only overrides (1 replica, 1 GPU)
β”œβ”€β”€ values-production.yaml  # production-only overrides (3 replicas, GPU type specified)
β”œβ”€β”€ templates/
β”‚   β”œβ”€β”€ deployment.yaml   # template filled in from {{ .Values.* }}
β”‚   β”œβ”€β”€ service.yaml
β”‚   β”œβ”€β”€ hpa.yaml
β”‚   β”œβ”€β”€ _helpers.tpl      # name and label helper functions reused across templates
β”‚   └── hooks/
β”‚       └── pre-upgrade-migration.yaml
└── charts/               # dependency subcharts (e.g. postgresql) β€” filled by helm dependency update
templates/deployment.yaml β€” the Deployment from the gpu-inference-deploy section, templatedYAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "llm-inference.fullname" . }}
  labels:
    {{- include "llm-inference.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ include "llm-inference.fullname" . }}
  template:
    metadata:
      labels:
        app: {{ include "llm-inference.fullname" . }}
    spec:
      nodeSelector:
        gpu-type: {{ .Values.gpu.nodeType }}
      containers:
      - name: vllm
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        args: ["--model", "{{ .Values.model.path }}"]
        resources:
          requests:
            nvidia.com/gpu: {{ .Values.gpu.count }}
          limits:
            nvidia.com/gpu: {{ .Values.gpu.count }}
values-production.yaml β€” per-environment overridesYAML
replicaCount: 3
image:
  repository: vllm/vllm-openai
  tag: "0.6.2"
gpu:
  nodeType: a100    # staging overrides this to a10g in values-staging.yaml
  count: 1
model:
  path: /models/llama-3-70b
templates/hooks/pre-upgrade-migration.yaml β€” a Job run automatically before an upgradeYAML
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "llm-inference.fullname" . }}-migrate
  annotations:
    "helm.sh/hook": pre-upgrade,pre-install
    "helm.sh/hook-delete-policy": hook-succeeded   # clean up successful Jobs; keep failed ones for investigation
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: migrate
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        command: ["python", "scripts/sync_model_registry.py"]
BASH
# Install & deploy
helm create myapp                       # scaffold a basic chart
helm install myapp ./myapp -f values-production.yaml -n production --atomic --wait
#   --atomic: roll back to the previous state automatically on failure
#   --wait:   the command waits, without returning, until the Pods are Ready

# Check the rendered output before actually applying (applies nothing to the cluster)
helm template myapp ./myapp -f values-production.yaml

# Upgrade & rollback
helm upgrade myapp ./myapp -f values-production.yaml --set image.tag=1.3.0 --atomic
helm history myapp                       # summary of changes per revision
helm rollback myapp 2                    # return to a specific revision at once

# Download subchart dependencies (postgresql and so on)
helm dependency update ./myapp

# Package & publish the chart to an OCI registry (the current standard Helm distribution method)
helm package ./myapp
helm push myapp-1.0.0.tgz oci://ghcr.io/myorg/charts
Command/flagRole
helm templatePreviews only the rendered YAML without applying anything to the cluster (dry-run)
--atomicRolls back to the previous revision automatically when an upgrade fails β€” all but mandatory for production deploys
--waitReturns only after the Pods are actually Ready β€” lets a CI pipeline judge deploy success accurately
helm.sh/hook (pre-upgrade and so on)Includes in the chart tasks that must run exactly once before or after a deploy, such as DB migrations
charts/ (subcharts)Brings in dependent components such as postgresql and redis as charts made by other teams and composes them (umbrella chart)

Tip

Do not put secrets such as a DB password in values.yaml as plain text β€” those values stay in helm history as-is for every revision. Inject secrets from outside the values with External Secrets Operator or similar, and put only the name (reference) of the Secret resource in the chart.

Contour & Gateway API β€” replacing NGINX Ingress

Here you look at Contour & Gateway API β€” replacing NGINX Ingress alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Contour is an L7 Ingress Controller that uses Envoy Proxy as its data plane. Unlike NGINX Ingress, it updates the Envoy configuration through the xDS API with no downtime and no worker process reload on configuration changes, so even with a large number of routes there are no connections dropped by reloads. Routing policy can be declared in two ways β€” HTTPProxy, a Contour-specific CRD (strong at delegation between teams and fine-grained policies), and the Gateway API, the Kubernetes standard that is not tied to a controller vendor (the three-layer GatewayClass/Gateway/HTTPRoute model). For a new build, look at the Gateway API first; if you need per-team namespace delegation or fine-grained IP/rate limit policies, look at HTTPProxy first.
contour-install.shBASH
# Option A β€” HTTPProxy-centered (static manifest, creates the IngressClass "contour")
kubectl apply -f https://projectcontour.io/quickstart/contour.yaml
kubectl get pods -n projectcontour -w
kubectl get svc envoy -n projectcontour   # check the external IP of the LoadBalancer

# Option B β€” Gateway API-centered (Contour Gateway Provisioner)
# 1) Install the Gateway API CRDs (standard channel)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/standard-install.yaml

# 2) Install Contour + the Gateway Provisioner β€” the GatewayClass "contour" is created automatically
kubectl apply -f https://projectcontour.io/quickstart/contour-gateway-provisioner.yaml
kubectl get gatewayclass contour

# After that, when you create a Gateway resource, the Provisioner automatically deploys a Contour+Envoy workload dedicated to that Gateway.
gateway.yamlYAML
# The GatewayClass is created automatically when contour-gateway-provisioner.yaml is applied
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: contour-gateway
  namespace: projectcontour
spec:
  gatewayClassName: contour
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    hostname: "*.aidevops.us"
    allowedRoutes:
      namespaces:
        from: All
  - name: https
    protocol: HTTPS
    port: 443
    hostname: "*.aidevops.us"
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: wildcard-aidevops-us-tls   # issued by cert-manager through DNS-01
    allowedRoutes:
      namespaces:
        from: All
httproute-canary.yamlYAML
# Gateway API HTTPRoute β€” header-based routing + weighted canary deployment
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: production
spec:
  parentRefs:
  - name: contour-gateway
    namespace: projectcontour
  hostnames:
  - "api.aidevops.us"
  rules:
  # With the x-canary: true header, route 100% to the canary service
  - matches:
    - headers:
      - name: x-canary
        value: "true"
    backendRefs:
    - name: api-service-canary
      port: 8080
  # The rest of the traffic is split 90:10 by weight
  - backendRefs:
    - name: api-service
      port: 8080
      weight: 90
    - name: api-service-canary
      port: 8080
      weight: 10
httpproxy-policy.yamlYAML
# Contour HTTPProxy β€” root: defines the domain/TLS/global rate limit + delegates paths per team (includes)
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: aidevops-root
  namespace: production
spec:
  virtualhost:
    fqdn: api.aidevops.us
    tls:
      secretName: wildcard-aidevops-us-tls
    rateLimitPolicy:
      local:
        requests: 100
        unit: second
        burst: 50
  includes:
  - name: api-v1-routes          # managed by the team in the production namespace
    namespace: production
    conditions:
    - prefix: /api/v1
  - name: api-v2-routes          # managed independently by the payments team in its own namespace
    namespace: team-payments
    conditions:
    - prefix: /api/v2
  routes:
  - conditions:
    - prefix: /
    services:
    - name: frontend-service
      port: 3000
---
# The delegated child HTTPProxy in the production namespace β€” the team owns the detailed policy directly
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: api-v1-routes
  namespace: production
spec:
  routes:
  - conditions:
    - prefix: /api/v1
    services:
    - name: api-service
      port: 8080
      weight: 90
    - name: api-service-canary
      port: 8080
      weight: 10
    retryPolicy:
      count: 3
      perTryTimeout: 2s
    timeoutPolicy:
      response: 10s
      idle: 60s
    ipAllowFilterPolicy:         # allow only the internal network + the listed ranges
    - source: 10.0.0.0/8
    - source: 203.0.113.0/24
wildcard-dns01-clusterissuer.yamlYAML
# A wildcard domain (*.aidevops.us) cannot be issued through HTTP-01 β€” a DNS-01 challenge is required
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns01
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@aidevops.us
    privateKeySecretRef:
      name: letsencrypt-dns01
    solvers:
    - dns01:
        route53:
          region: us-east-1
          hostedZoneID: ZXXXXXXXXXXXXX   # Route53 Hosted Zone ID
      selector:
        dnsZones:
        - "aidevops.us"
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-aidevops-us
  namespace: projectcontour
spec:
  secretName: wildcard-aidevops-us-tls
  issuerRef:
    name: letsencrypt-dns01
    kind: ClusterIssuer
  dnsNames:
  - "aidevops.us"
  - "*.aidevops.us"
ItemNGINX IngressContour HTTPProxyGateway API HTTPRoute
Applying configurationRequires a reload β€” risk of dropped connections with many IngressesApplied dynamically with no downtime through Envoy xDSSame (Contour applies it to Envoy through xDS)
Delegation between teamsAnnotation-based; delegation across namespaces is hardDelegation per path/domain stated with `spec.includes`Cross-namespace access allowed with ReferenceGrant
Canary/weighted routingVendor-specific annotation (canary-weight)The `services[].weight` fieldThe `backendRefs[].weight` field
Header-based routingNeeds a combination of annotations`conditions[].header``matches[].headers`
Standard?Non-standard annotations per controllerContour-specific CRDOfficial Kubernetes SIG standard (k8s.io)
Rate limit / IP policyAnnotations that differ per controller`rateLimitPolicy`, `ipAllow/DenyFilterPolicy`Needs a GEP extension or a separate policy controller

Proxy Protocol β€” preserving the client IP behind an LB

Here you look at Proxy Protocol β€” preserving the client IP behind an LB alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

With an L4 LoadBalancer in front of Contour (Envoy), a common problem is that the client IP seen in the Envoy access log or by the application is not the real user IP but the IP of the LB or a node. This is because the LB does SNAT (the ip target mode of AWS NLB, Classic ELB, many managed LBs), or kube-proxy does SNAT when forwarding between nodes under externalTrafficPolicy: Cluster. With HTTPS, Envoy terminates TLS, so the L4 LB cannot look into the encrypted bytes and cannot insert an X-Forwarded-For header either.

Proxy Protocol is a convention proposed by HAProxy in which the L4 LB, as soon as it opens a TCP connection to the backend, first sends a short header at the very start of the connection carrying the original client IP/port, and then passes the original HTTP/TLS bytes through unchanged. v1 is a single line of human-readable text and v2 is a binary format; v2 can carry TLV extensions (the AWS VPC Endpoint ID and so on). The envoy.filters.listener.proxy_protocol listener filter of Envoy recognizes v1/v2 automatically, reads the header, and then replaces the downstream remote address of the following requests with the original IP from the header β€” so the X-Forwarded-For and X-Envoy-External-Address that Envoy sends upstream, the access log, and ipAllowFilterPolicy decisions all work on the real client IP.

⚠️
The key rule β€” the sending side and the receiving side must be turned on at the same time
The Proxy Protocol spec defines that the receiver must not "guess" whether a header is present, and that a listener requiring the header rejects connections without one. Turning Proxy Protocol on in Contour attaches this filter to every traffic listener of Envoy (HTTP 8080, HTTPS 8443), so all in-cluster calls that reach Envoy directly without going through the LB are cut off. Conversely, turning on only the LB and not Envoy makes the header get parsed as HTTP/TLS data, causing 400 Bad Request or TLS handshake failures. The subsections that follow in this guide work through these two problems by LB, by Contour version, and by alternative approach.
Rendering diagram…
proxy-protocol-header-example.txtTEXT
# v1 (text) β€” one line is sent first right after connecting, followed by the original HTTP/TLS data
PROXY TCP4 203.0.113.7 10.0.12.34 51234 443\r\n
#     protocol source IP     LB/dest IP source port dest port

# v2 (binary) β€” starts with a 12-byte signature
\x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A  + version/command + addresses + TLVs (optional)

# Envoy listener filter configuration (the form Contour generates when useProxyProtocol=true)
listener_filters:
- name: envoy.filters.listener.proxy_protocol
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol
    # allow_requests_without_proxy_protocol: false  ← the default. Contour offers no option to change this value
- name: envoy.filters.listener.tls_inspector    # on the HTTPS listener, runs next to check the SNI
Way of preserving the client IPSuitable environmentProsCaution
externalTrafficPolicy: LocalWhen the LB forwards packets as-is with no SNAT (passthrough) β€” GKE/AKS L4 LB, MetalLB, AWS NLB instance targetsOne line of Service configuration, no effect on internal callsNodes with no Envoy Pod drop out of the LB health check β†’ traffic can be uneven across nodes
Proxy ProtocolWhen the LB does SNAT/proxying β€” AWS NLB ip targets, Classic ELB, NCP (Naver Cloud), DigitalOcean, HAProxy/F5, and so onDelivers the original IP even though Envoy terminates TLS; independent of externalTrafficPolicyLB and Envoy must be configured together, internal calls with no header are blocked, hairpin traffic problem
L7 LB + X-Forwarded-ForWhen an L7 LB such as ALB or Application Gateway terminates TLS firstNo Proxy Protocol needed, easy WAF integrationThe Contour num-trusted-hops setting is required; TLS is terminated twice

Tip

  • Proxy Protocol is an L4 technology β€” it is bytes at the very start of the TCP stream, not an HTTP header, so application code needs no change and only has to read the X-Forwarded-For that Envoy produces.
  • If the LB is of the passthrough kind, externalTrafficPolicy: Local is far simpler than Proxy Protocol. Start by checking whether the LB you use does SNAT.
  • The moment you turn Proxy Protocol on, "every path that does not go through the LB" is affected β€” before applying it, first list the workloads that call the Envoy Service or the external domain from inside the cluster.

Proxy Protocol settings per LB

Here you look at Proxy Protocol settings per LB alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

It is the LB that adds the Proxy Protocol header. In Kubernetes, you put a cloud-specific annotation on the type: LoadBalancer Service that exposes Envoy so that the Cloud Controller Manager (or the AWS Load Balancer Controller) turns Proxy Protocol on for the LB listener/target group. With Proxy Protocol the original IP travels in the header, so externalTrafficPolicy can stay at Cluster, which is actually better for balancing traffic across nodes.

Health check caution: if the LB health check does not send the PROXY header when connecting to the traffic port, or conversely sends the header to a port that does not understand it (the Envoy 8002 /ready, the kube-proxy healthCheckNodePort), the targets go unhealthy. Check in the cloud documentation "whether health checks also include the Proxy Protocol header", and if unsure, start with a TCP health check on the traffic port, confirm the behavior, and then move to an HTTP health check.

The Envoy listener filter recognizes both v1 and v2 automatically, so the Contour-side configuration is the same whichever version the LB sends.
envoy-service-aws-nlb.yamlYAML
# When installed from the static manifest (contour.yaml) β€” add the annotation to the envoy Service
# With the Gateway Provisioner, this Service is managed by the Provisioner, so do not edit it directly;
# set it through ContourDeployment.spec.envoy.networkPublishing.serviceAnnotations (next section)
apiVersion: v1
kind: Service
metadata:
  name: envoy
  namespace: projectcontour
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"   # PP v2 on every target group
spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster     # the original IP travels in the PROXY header, so Cluster is enough
  selector:
    app: envoy
  ports:
  - name: http
    port: 80
    targetPort: 8080
    protocol: TCP
  - name: https
    port: 443
    targetPort: 8443
    protocol: TCP
aws-verify-target-group.shBASH
# Check that Proxy Protocol v2 is really on for the NLB target groups
aws elbv2 describe-target-groups \
  --load-balancer-arn "$NLB_ARN" \
  --query "TargetGroups[].TargetGroupArn" --output text |
tr '\t' '\n' | while read -r TG; do
  echo "== $TG"
  aws elbv2 describe-target-group-attributes --target-group-arn "$TG" \
    --query "Attributes[?Key=='proxy_protocol_v2.enabled']"
done
haproxy.cfg (on-premises L4 front end)TEXT
frontend fe_https
    mode tcp
    bind :443
    default_backend be_envoy_https

backend be_envoy_https
    mode tcp
    balance roundrobin
    # send-proxy-v2: send the PROXY v2 header when connecting to the backend (Envoy NodePort)
    # check-send-proxy: add the header to health check connections too, so the Envoy listener does not reject them
    server node1 10.0.0.11:30443 send-proxy-v2 check check-send-proxy
    server node2 10.0.0.12:30443 send-proxy-v2 check check-send-proxy
nginx-stream.conf (on-premises L4 front end)NGINX
stream {
    upstream envoy_https {
        server 10.0.0.11:30443;
        server 10.0.0.12:30443;
    }
    server {
        listen 443;
        proxy_pass envoy_https;
        proxy_protocol on;     # send the PROXY v1 header to the upstream (Envoy)
    }
}
LBHow to enable Proxy ProtocolVersionNotes
AWS NLB (AWS Load Balancer Controller)service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"v2Client IP preservation is off by default in ip target mode β†’ Proxy Protocol is effectively required. Check with the target group attribute proxy_protocol_v2.enabled
AWS Classic ELB (in-tree, legacy)The same annotation as abovev1NLB is recommended for new builds
GKE L4 (passthrough NLB)Not applicable β€” use externalTrafficPolicy: Local-Packets are forwarded as-is, so the original IP is already preserved
Azure LB (AKS)Not applicable β€” use externalTrafficPolicy: Local-Only when going through Private Link Service: service.beta.kubernetes.io/azure-pls-proxy-protocol: "true"
NCP NKS (Network Proxy LB)service.beta.kubernetes.io/ncloud-load-balancer-proxy-protocol: "true"v2Check the NCP documentation for the annotations supported per LB type and NKS version
DigitalOceanservice.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"v1The LB returns an IP, so the hairpin problem has to be handled (see the internal calls section)
On-premises HAProxy / F5 / NGINX streamHAProxy send-proxy-v2, NGINX proxy_protocol on;, F5 iRulev1/v2MetalLB (L2/BGP) does not add Proxy Protocol β†’ use the Local policy

Tip

  • Depending on the cloud, changing an LB annotation can cause the LB to be recreated (the external IP/DNS changes). For a Service in production, creating a new Service alongside and switching DNS over is the safe way.
  • The LB and Envoy settings cannot be changed "at the same instant", so there is a brief outage at the moment of the switch. If you need zero downtime, build a new LB + new Envoy set with Proxy Protocol on and shift over with DNS weights.

Envoy Proxy Protocol settings per Contour version (1.28 - 1.33)

Here you look at Envoy Proxy Protocol settings per Contour version (1.28 - 1.33) alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

There are three ways to make Envoy accept the PROXY header in Contour, depending on how it was installed. The result is the same in every case β€” Contour adds envoy.filters.listener.proxy_protocol at the front of the listener_filters of the ingress_http/ingress_https listeners of Envoy. It is not applied to the stats listener (8002) or the admin interface.
Install methodWhere to configureField/flag
Static manifest (quickstart contour.yaml)The contour serve arguments of the contour Deployment--use-proxy-protocol
Using the ContourConfiguration CRD (contour serve --contour-config-name)ContourConfigurationspec.envoy.listener.useProxyProtocol: true
Gateway Provisioner (Gateway API)ContourDeployment (the parametersRef of the GatewayClass)spec.runtimeSettings.envoy.listener.useProxyProtocol: true
Caution: the Contour configuration file (contour.yaml in the contour ConfigMap) has no key corresponding to Proxy Protocol. It is ignored if you put one in the ConfigMap, so use the flag or the CRD.

The version table below follows the official Contour compatibility matrix. Between 1.28 and 1.33, the way Proxy Protocol is configured did not change; what differs is the bundled Envoy version and the supported Kubernetes and Gateway API versions. In particular, in 1.33 the bundled Envoy jumped from 1.35 β†’ 1.38 midway through the patch releases (at 1.33.6), so even for a patch upgrade within 1.33.x, check the behavior changes in the Envoy release notes and verify in staging first.
A-static-manifest-flag.shBASH
# [Option A] Static manifest (quickstart contour.yaml) β€” add the flag to contour serve
kubectl -n projectcontour patch deployment contour --type=json -p='[
  {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--use-proxy-protocol"}
]'

kubectl -n projectcontour rollout status deployment/contour
kubectl -n projectcontour get deploy contour \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n'
# ["serve", "--incluster", ..., "--use-proxy-protocol"]

# Note: re-applying the quickstart manifest removes the flag.
#   Manage the patch alongside it as a Kustomize patch or in your GitOps repository.
B-contourconfiguration.yamlYAML
# [Option B] ContourConfiguration CRD β€” used when running with contour serve --contour-config-name=contour
apiVersion: projectcontour.io/v1alpha1
kind: ContourConfiguration
metadata:
  name: contour
  namespace: projectcontour
spec:
  envoy:
    listener:
      useProxyProtocol: true       # adds the proxy_protocol listener filter to every Envoy traffic listener
    network:
      numTrustedHops: 0            # 0 if only L4 (PP) is in front β€” if there is one more L7 proxy hop, set that many
  # ... keep the existing settings (gateway, httpproxy, tls, and so on) as they are
C-contourdeployment-provisioner.yamlYAML
# [Option C] Gateway Provisioner β€” manage the Contour+Envoy settings separately per GatewayClass
apiVersion: projectcontour.io/v1alpha1
kind: ContourDeployment
metadata:
  name: contour-external-pp
  namespace: projectcontour
spec:
  runtimeSettings:                 # = the same structure as the ContourConfiguration spec
    envoy:
      listener:
        useProxyProtocol: true
  envoy:
    networkPublishing:
      type: LoadBalancerService
      externalTrafficPolicy: Cluster
      serviceAnnotations:          # passed as-is to the envoy Service the Provisioner creates
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: contour-external-pp
spec:
  controllerName: projectcontour.io/gateway-controller
  parametersRef:
    group: projectcontour.io
    kind: ContourDeployment
    name: contour-external-pp
    namespace: projectcontour
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: external-gateway
  namespace: projectcontour
spec:
  gatewayClassName: contour-external-pp
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All
  - name: https
    protocol: HTTPS
    port: 443
    hostname: "*.aidevops.us"
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: wildcard-aidevops-us-tls
    allowedRoutes:
      namespaces:
        from: All
D-helm-values.yamlYAML
# [For reference] When installed from a Helm chart β€” key names differ per chart, so always check:
#   helm show values <repo>/contour --version <chart-version> | grep -n -i -E "extraArgs|proxy|annotations"
contour:
  extraArgs:
  - --use-proxy-protocol
envoy:
  service:
    externalTrafficPolicy: Cluster
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: external
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
check-contour-envoy-version.shBASH
# Check the Contour / Envoy image versions of the current cluster β†’ find the matching row in the table above
kubectl -n projectcontour get deploy,ds \
  -o custom-columns='NAME:.metadata.name,IMAGES:.spec.template.spec.containers[*].image'
# NAME      IMAGES
# contour   ghcr.io/projectcontour/contour:v1.33.x
# envoy     ghcr.io/projectcontour/contour:v1.33.x,docker.io/envoyproxy/envoy:v1.35.x

# Check that the setting really reached the Envoy listeners (Envoy admin: 9001 inside the Pod)
ENVOY_POD=$(kubectl -n projectcontour get pod -l app=envoy -o jsonpath='{.items[0].metadata.name}')
kubectl -n projectcontour port-forward "$ENVOY_POD" 9001:9001 >/dev/null &
sleep 2
curl -s localhost:9001/config_dump | grep -c "envoy.filters.listener.proxy_protocol"
# 2 or more (ingress_http + ingress_https) means applied; 0 means not applied
# Note: an Envoy created by the Provisioner may have different labels: kubectl get pod -n projectcontour --show-labels
ContourBundled EnvoyKubernetesGateway APIProxy Protocol configurationAllowing connections without PROXY (allow_requests_without_proxy_protocol)
1.28.x1.29.x1.27 - 1.29v1.0.0Flag / ContourConfiguration / ContourDeploymentSupported by Envoy, not exposed by Contour β†’ cannot be set
1.29.x1.30.x1.27 - 1.29v1.0.0SameCannot be set
1.30.x1.31.x1.28 - 1.30v1.1.0SameCannot be set
1.31.x1.34.x1.30 - 1.32v1.2.1SameCannot be set
1.32.x1.34.x1.31 - 1.33v1.2.1SameCannot be set
1.33.0 - 1.33.51.35.x1.32 - 1.34v1.3.0SameCannot be set
1.33.6 and later1.38.x1.32 - 1.34v1.3.0SameCannot be set β€” see the next section for alternatives

Tip

  • The Proxy Protocol setting applies to the whole of Contour (every Envoy traffic listener) at once. It cannot be turned on and off per listener, so splitting "external (PP on)" from "internal (PP off)" means running two Contour+Envoy sets.
  • With the Gateway Provisioner, each GatewayClass (ContourDeployment) gets its own Contour+Envoy set, so a layout with separate external/internal Gateways is the cleanest.
  • Check the Compatibility Matrix on projectcontour.io for the exact Envoy patch version per release β€” even within the same Contour minor version, the bundled Envoy moves up with security updates from patch to patch.

Allowing internal calls with no PROXY header & alternatives

Here you look at Allowing internal calls with no PROXY header & alternatives alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

The problem that blows up most often after turning Proxy Protocol on is calls that reach Envoy without going through the LB. There are three typical paths.

β‘  In-cluster Pod β†’ Envoy Service (ClusterIP) called directly β€” for example, requests to envoy.projectcontour.svc.cluster.local in order to go through the internal API gateway policies. There is no PROXY header, so Envoy drops the connection.
β‘‘ In-cluster Pod β†’ external domain (LB IP) call (hairpin) β€” when the LB records an IP in the Service status.loadBalancer.ingress, kube-proxy intercepts packets bound for that IP on the node and sends them straight to Envoy without going through the LB. As a result they arrive with no PROXY header and fail. This usually does not happen with LBs that record only a hostname, like AWS NLB, and does happen in environments that record an IP, such as DigitalOcean, Hetzner, and some on-premises LBs.
β‘’ Corporate network/VPN β†’ internal LB β†’ Envoy β€” this also fails if Proxy Protocol is not turned on for the internal LB.

The Envoy allow_requests_without_proxy_protocol option
The proxy_protocol listener filter of Envoy has an allow_requests_without_proxy_protocol: true option that makes "optional acceptance" possible: read the header if present and just pass the connection through if not. Every Envoy bundled with Contour 1.28 - 1.33 supports this field. However, Contour does not expose this field β€” there is no corresponding option anywhere in the Contour configuration file, the contour serve flags, or ContourConfiguration/ContourDeployment, and because Contour keeps regenerating the listeners through xDS, there is no arbitrary patch path like the EnvoyFilter of Istio either. So up to Contour 1.33, "read PROXY if present and allow it if absent" cannot be achieved by configuration, and you have to choose one of the alternatives below (check the Contour release notes for whether an option has been added in newer versions).

⚠️
Points to watch even where the allow option is available
The Envoy documentation warns that this option breaks compliance with the Proxy Protocol spec and should be turned on "only when all traffic entering the listener comes from a trusted source". For connections with no header, the TCP source IP (the node/SNAT IP) is recorded as the client IP, so IP-based allow policies can get mixed up; and conversely, a connection that bypasses the LB and comes in directly from outside can impersonate any IP by forging the PROXY header. Also, very short requests of 12 bytes or fewer that match the v2 signature (6 bytes or fewer for v1) time out because Envoy cannot tell whether they are a header. Restrict direct access to the Envoy listeners (NodePort/Pod IP) to the LB and trusted ranges with NetworkPolicy and security groups.
Rendering diagram…
option2-internal-gateway.yamlYAML
# [β‘‘ recommended] Internal-only Gateway β€” Proxy Protocol OFF + ClusterIP Service
apiVersion: projectcontour.io/v1alpha1
kind: ContourDeployment
metadata:
  name: contour-internal
  namespace: projectcontour
spec:
  runtimeSettings:
    envoy:
      listener:
        useProxyProtocol: false          # the internal one accepts connections with no header
  envoy:
    networkPublishing:
      type: ClusterIPService             # reachable only from inside the cluster
      # To expose it to the corporate network/VPN as well, use LoadBalancerService + an internal LB annotation:
      # type: LoadBalancerService
      # serviceAnnotations:
      #   service.beta.kubernetes.io/aws-load-balancer-scheme: internal
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: contour-internal
spec:
  controllerName: projectcontour.io/gateway-controller
  parametersRef:
    group: projectcontour.io
    kind: ContourDeployment
    name: contour-internal
    namespace: projectcontour
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: internal-gateway
  namespace: projectcontour
spec:
  gatewayClassName: contour-internal
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All
---
# Attach one HTTPRoute to both the external (PP on) and internal (PP off) Gateways β†’ a single set of routing rules
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: production
spec:
  parentRefs:
  - name: external-gateway          # the PP on Gateway from the previous section
    namespace: projectcontour
  - name: internal-gateway
    namespace: projectcontour
  hostnames:
  - "api.aidevops.us"
  rules:
  - backendRefs:
    - name: api-service
      port: 8080

# Internal Pods call the internal envoy Service created by the Provisioner:
#   kubectl -n projectcontour get svc   # look for a name of the form envoy-internal-gateway
#   curl -H "Host: api.aidevops.us" http://envoy-internal-gateway.projectcontour.svc.cluster.local/
option2-static-second-instance.yamlYAML
# [β‘‘ static manifest version] Running a second Contour+Envoy set without the Provisioner
# - Deploy a copy of the quickstart manifest as the second set in a separate namespace (e.g. projectcontour-internal)
# - Distinguish the IngressClass each handles with contour serve arguments, and add the PP flag to the external set only
#     external: contour serve ... --ingress-class-name=contour-external --use-proxy-protocol
#     internal: contour serve ... --ingress-class-name=contour-internal
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: api-internal
  namespace: production
spec:
  ingressClassName: contour-internal     # only the internal set handles this HTTPProxy
  virtualhost:
    fqdn: api.internal.aidevops.us
  routes:
  - conditions:
    - prefix: /
    services:
    - name: api-service
      port: 8080
option3-client-sends-proxy-header.shBASH
# [β‘’] Having the client send the PROXY header itself
# curl β€” for tests and health check scripts (sends a v1 header)
curl --haproxy-protocol -H "Host: api.aidevops.us" \
  http://envoy.projectcontour.svc.cluster.local/healthz

# curl 8.2+ β€” specify the client IP to put in the header
curl --haproxy-protocol --haproxy-clientip 10.1.2.3 -H "Host: api.aidevops.us" \
  http://envoy.projectcontour.svc.cluster.local/

# For application traffic, HTTP libraries mostly do not support the PROXY header, so
# put a TCP proxy sidecar in the same Pod and have the app call localhost (HAProxy example):
#   frontend local_in
#       mode tcp
#       bind 127.0.0.1:18080
#       default_backend envoy
#   backend envoy
#       mode tcp
#       server contour envoy.projectcontour.svc.cluster.local:80 send-proxy-v2
# The app sends requests to http://127.0.0.1:18080 + sets the Host header
option4-hairpin.yamlYAML
# [β‘£] Blocking the hairpin β€” making internal Pods go through the LB even when calling the external domain

# (a) Kubernetes 1.32+ (LoadBalancerIPMode GA): when the Cloud Controller records ipMode: Proxy,
#     kube-proxy does not intercept the LB IP on the node and sends it to the real LB β†’ the PROXY header is added
#     Check:
#     kubectl -n projectcontour get svc envoy -o jsonpath='{.status.loadBalancer.ingress}'
#     [{"ip":"203.0.113.10","ipMode":"Proxy"}]   ← Proxy is OK; VIP or absent means the hairpin occurs
#     ipMode is set by the Cloud Controller Manager, not the user β€” check whether your CCM supports it

# (b) Per-LB annotations that record a hostname in the status instead of an IP
apiVersion: v1
kind: Service
metadata:
  name: envoy
  namespace: projectcontour
  annotations:
    # DigitalOcean
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
    service.beta.kubernetes.io/do-loadbalancer-hostname: "lb.aidevops.us"
    # Hetzner Cloud
    # load-balancer.hetzner.cloud/uses-proxyprotocol: "true"
    # load-balancer.hetzner.cloud/hostname: "lb.aidevops.us"
spec:
  type: LoadBalancer
  selector:
    app: envoy
  ports:
  - name: https
    port: 443
    targetPort: 8443
option6-envoy-gateway.yamlYAML
# [β‘₯] When moving to Envoy Gateway β€” enable Proxy Protocol + optionally accept connections with no header
# 1) Enable Proxy Protocol with a ClientTrafficPolicy
#    (recent Envoy Gateway offers an optional setting in the proxyProtocol block β€” check the API docs for your version)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: enable-proxy-protocol
  namespace: envoy-gateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  enableProxyProtocol: true
---
# 2) On a version with no optional field, patch the Envoy field directly with an EnvoyPatchPolicy
#    (requires extensionApis.enableEnvoyPatchPolicy: true in the EnvoyGateway configuration)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: pp-allow-without-header
  namespace: envoy-gateway-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
  - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
    name: envoy-gateway-system/eg/http        # <namespace>/<gateway>/<listener>
    operation:
      op: add
      path: "/listener_filters/0/typed_config/allow_requests_without_proxy_protocol"
      value: true
# After applying, check the listener_filters order (that index 0 is proxy_protocol) with egctl config envoy-proxy listener
AlternativePath it solvesProsCons
β‘  Internal calls go straight to the app Service (svc.cluster.local)Pod β†’ EnvoySimplest, no extra resources, one fewer hopBypasses the rate limits, IP policies, TLS, and routing rules of Envoy
β‘‘ A separate internal-only Gateway (Envoy) β€” recommendedPod β†’ Envoy, corporate network β†’ internal LBReuses the routing rules by attaching one HTTPRoute to both the external and internal GatewaysAn extra Contour+Envoy set to run (resources, monitoring)
β‘’ The client sends the PROXY header itselfPod β†’ EnvoyReuses the existing EnvoyMost ordinary HTTP client libraries do not support it β€” an intermediate TCP proxy (HAProxy/Envoy sidecar) is needed
β‘£ Block the hairpin (ipMode: Proxy / hostname annotation)Pod β†’ external domainInternal traffic also goes through the LB and gets the PROXY headerNeeds Cloud Controller support; more latency and cost from going through the LB
β‘€ externalTrafficPolicy: Local instead of Proxy ProtocolEvery path (removes PP itself)The internal call problem disappears at the rootOnly possible when the LB is passthrough (no SNAT)
β‘₯ Move to Envoy Gateway + optional PPEvery pathallow_requests_without_proxy_protocol can be turned on directlyA controller swap (HTTPProxy cannot be migrated; HTTPRoute can be reused)
⑦ A forked/patched Contour buildEvery pathKeeps ContourPatch the listener filter generation code and build it yourself β€” reapplied on every upgrade, heavy operational burden (not recommended)

Tip

  • If you keep Contour, splitting into "external Gateway (PP on) + internal Gateway (PP off)" is the safest and most predictable layout. Putting both Gateways in the parentRefs of the HTTPRoute also means there is no need to maintain duplicate routing rules.
  • When building an internal-only Envoy, check that rate limit and authentication policies apply the same as on the external one. On the internal Envoy, the ipAllowFilterPolicy of HTTPProxy is evaluated against the Pod IP range.
  • Forging the PROXY header makes it possible to spoof the original IP. Lock down the Envoy Pods/NodePort with NetworkPolicy and security groups so they are reachable only from the LB and trusted ranges.

Proxy Protocol verification & troubleshooting

Here you look at Proxy Protocol verification & troubleshooting alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Proxy Protocol problems are mostly one of two things: "a mismatch between the LB and Envoy settings" or "a path that does not go through the LB". The symptoms alone look like a network failure, but the shape of the symptom lets you infer almost exactly which side is turned on. Narrow down the cause with the table below, then follow the order in the code block: Envoy configuration β†’ requests with and without the header β†’ statistics β†’ access log.
pp-verify.shBASH
NS=projectcontour
ENVOY_POD=$(kubectl -n $NS get pod -l app=envoy -o jsonpath='{.items[0].metadata.name}')

# 1) Whether the proxy_protocol filter is attached to the Envoy listeners
kubectl -n $NS port-forward "$ENVOY_POD" 9001:9001 >/dev/null &
PF_PID=$!; sleep 2
curl -s localhost:9001/config_dump \
  | grep -B2 -A4 "envoy.filters.listener.proxy_protocol"

# 2) Proxy Protocol statistics β€” whether the header parse failure/rejection counters are rising
curl -s localhost:9001/stats | grep -i proxy_proto
kill $PF_PID

# 3) Compare requests with and without the header (debug Pod inside the cluster)
kubectl run pp-test -n default --rm -it --restart=Never --image=curlimages/curl -- sh -c '
  echo "--- without the PROXY header (failure is expected when PP is on)";
  curl -sv -m 5 -H "Host: api.aidevops.us" http://envoy.projectcontour.svc.cluster.local/ 2>&1 | tail -3;
  echo "--- with a PROXY v1 header (success is expected when PP is on)";
  curl -sv -m 5 --haproxy-protocol -H "Host: api.aidevops.us" http://envoy.projectcontour.svc.cluster.local/ 2>&1 | tail -3'

# 4) After a request from outside, whether the real public IP shows up in the Envoy access log
curl -s https://api.aidevops.us/ -o /dev/null
kubectl -n $NS logs "$ENVOY_POD" -c envoy --tail=5
# The x-forwarded-for / downstream address in the default log format should be your public IP

# 5) Hairpin check β€” if the status has an IP and ipMode is not Proxy, internal calls bypass the LB
kubectl -n $NS get svc -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.loadBalancer.ingress}{"\n"}{end}'
envoy-networkpolicy.yamlYAML
# Guard against forged PROXY headers β€” allow the Envoy traffic ports only from the VPC range where the LB lives
# (assumes an ip-target NLB connects from VPC subnet IPs, and internal calls are split off to the internal Gateway)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: envoy-allow-lb-only
  namespace: projectcontour
spec:
  podSelector:
    matchLabels:
      app: envoy
  policyTypes: ["Ingress"]
  ingress:
  - from:
    - ipBlock:
        cidr: 10.0.0.0/16          # LB subnet / VPC CIDR
    ports:
    - port: 8080
    - port: 8443
  - from:                          # Prometheus metric scraping (8002)
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
    ports:
    - port: 8002
SymptomCauseCheck / fix
Every external request fails β€” curl "Empty reply from server", "Connection reset", HTTPS cut off mid-handshakePP on at Envoy only, off at the LBCheck the LB annotation / target group proxy_protocol_v2.enabled
400 Bad Request over HTTP, "wrong version number" or SSL errors over HTTPSPP on at the LB only, off at Envoy (the PROXY header is parsed as HTTP/TLS data)Check whether the proxy_protocol filter is in config_dump β†’ apply useProxyProtocol
External is fine; only internal Pod β†’ Envoy Service calls failInternal calls with no PROXY headerSeparate internal-only Gateway, or call the app Service directly
External is fine; only internal Pod β†’ external domain calls failkube-proxy hairpin (the LB IP is intercepted on the node)Check ipMode in the Service status, apply the hostname annotation
Every LB target is unhealthyMismatch between the PROXY header on health check connections and the target portSwitch to a TCP health check on the traffic port and check again
Works normally, but the client IP in the logs is still the node/LB IPPP not applied, or there is one more L7 proxy hop in frontCheck config_dump; if there is an L7, set numTrustedHops to the number of hops
Very short requests time out intermittently (when using optional acceptance)Requests of 12 bytes (v1: 6 bytes) or fewer are judged to be waiting for a headerSwitch to a split-listener layout instead of the allow option

Tip

  • Cutover checklist: β‘  check whether the LB does SNAT β†’ β‘‘ list the internal call paths with no header β†’ β‘’ decide on a separate internal Gateway or another alternative β†’ β‘£ apply to the LB and Envoy together in staging β†’ β‘€ confirm the original IP with config_dump and the access log β†’ β‘₯ in production, apply with zero downtime through a new LB + DNS switch.
  • After a Contour upgrade (especially a patch that changes the bundled Envoy minor version, like 1.33.5 β†’ 1.33.6), run pp-verify.sh again to confirm the listener filter and original IP logging are still in place.

Contour/Envoy monitoring β€” Prometheus & Grafana

Here you look at Contour/Envoy monitoring β€” Prometheus & Grafana alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Contour exposes its own control plane metrics at :8000/metrics, and Envoy, the data plane, exposes xDS sync state, upstream connections, and request outcomes at :8002/stats/prometheus. If you have already installed kube-prometheus-stack (see Setting up the monitoring stack), you only need to add two ServiceMonitors. Be sure to block external exposure of the Envoy admin interface (9001) so it is reachable only from inside the cluster.
servicemonitor-contour-envoy.yamlYAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: contour
  namespace: monitoring
  labels:
    release: kube-prometheus-stack   # must match the Helm release name to be scraped
spec:
  namespaceSelector:
    matchNames: ["projectcontour"]
  selector:
    matchLabels:
      app: contour
  endpoints:
  - port: metrics
    path: /metrics
    interval: 30s
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: envoy
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    matchNames: ["projectcontour"]
  selector:
    matchLabels:
      app: envoy
  endpoints:
  - port: metrics
    path: /stats/prometheus
    interval: 30s
prometheusrule-envoy.yamlYAML
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: contour-envoy-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
  - name: contour.rules
    rules:
    - alert: EnvoyHigh5xxRate
      expr: sum(rate(envoy_cluster_upstream_rq_xx{envoy_response_code_class="5"}[5m])) / sum(rate(envoy_cluster_upstream_rq_xx[5m])) > 0.05
      for: 5m
      labels: { severity: critical }
      annotations:
        summary: "Envoy 5xx response ratio above 5%"

    - alert: EnvoyUpstreamConnectFail
      expr: increase(envoy_cluster_upstream_cx_connect_fail[5m]) > 20
      for: 5m
      labels: { severity: warning }
      annotations:
        summary: "Surge in Envoy upstream connection failures β€” check the backend Pod health checks"

    - alert: ContourReconcileErrors
      expr: increase(contour_dagrebuild_total{result="error"}[15m]) > 0
      for: 5m
      labels: { severity: warning }
      annotations:
        summary: "Contour DAG rebuild errors β€” check for conflicting HTTPProxy/HTTPRoute configuration"

    - alert: WildcardCertExpiringSoon
      expr: (certmanager_certificate_expiration_timestamp_seconds{name="wildcard-aidevops-us"} - time()) / 86400 < 14
      for: 1h
      labels: { severity: warning }
      annotations:
        summary: "Wildcard certificate expires within 14 days β€” check whether DNS-01 renewal is failing"
grafana-dashboards.shBASH
# Official Envoy Grafana dashboards (grafana.com/dashboards) β€” usable just by entering the Import ID
# Envoy Global:   Dashboard ID 11021
# Envoy Clusters: Dashboard ID 11022

# No Contour dashboard is registered on grafana.com, so take the JSON directly from the repository
git clone --depth 1 https://github.com/projectcontour/contour.git /tmp/contour
# Upload the JSON under /tmp/contour/examples/grafana/ through the Grafana UI β†’ Dashboards β†’ Import
# (the path can change between versions, so check the examples/ directory of the tag you actually cloned)

Kubernetes Advanced: frequently asked questions

How do Helm and Kustomize differ?

Helm is a packaging tool that fills values into templates to produce manifests and manages what it installs as versioned releases. Kustomize has no templates; it overlays per-environment changes on base manifests. A common combination is Helm for installing software published by others and Kustomize for managing per-environment differences of your own applications.

What is the difference between Ingress and the Gateway API?

Ingress is the early, simple API for connecting HTTP requests to Services, and its finer features depend on annotations that differ per controller. The Gateway API is its successor: it separates the role that operates the entry point from the role that defines routes, and provides features such as traffic splitting and header-based routing as a standard. For a new setup, look at the Gateway API.

How should I set requests and limits?

requests is the amount the scheduler reserves when placing the Pod, and limits is the ceiling it cannot exceed. A container that exceeds its memory limit is killed (OOMKilled), so measure real usage and leave headroom. Exceeding the CPU limit does not kill the container; it is throttled. Begin by setting requests from observed usage.

Next steps

This section covers Next steps from a practical perspective. Instead of memorizing the concept, focus on the situations in which you would reach for it.

πŸš€
Roadmap after advanced K8s

β€’ GitOps: Git-based declarative deployment automation with ArgoCD / Flux
β€’ Service mesh: Istio / Linkerd β€” traffic control, mTLS, observability
β€’ CKA / CKAD certification: recognized certifications of practical K8s skills

Related guides: Kubernetes basics guide Β· CI/CD guide Β· Prometheus guide Β· Grafana guide
← Previous guideKubernetesNext guide β†’Prometheus