HPA/VPA autoscaling, RBAC security, StatefulSet, DaemonSet, and CronJob workloads, AI model operations strategies such as deploying GPU-based LLM inference servers, canary rollouts, and queue-based autoscaling, upgrades and failure recovery, advanced Helm covering chart structure, hooks, and subcharts, moving from Ingress to Contour/Gateway API, Proxy Protocol settings per LB and Contour version and handling internal calls, and monitoring β the advanced topics needed to run Kubernetes in production.
Advanced
Updated Oct 1, 2026
~53 min read
22 sections
77 code examples
Learning Paths that include this guide
This guide is one step in the paths below. Study it together with the steps before and after.
Autoscaling & zero-downtime deploymentRBAC-based permission managementOperating StatefulSets/DaemonSets/CronJobsLLM inference servers on GPU nodes & AI model operations strategyAdvanced Helm charts & GitOps integrationMoving from Ingress to Contour/Gateway API & policy settingsPreserving the client IP with Proxy Protocol & handling internal calls
Understanding Kubernetes Advanced as a real-world workflow
HPA/VPA autoscaling, RBAC security, StatefulSet, DaemonSet, and CronJob workloads, AI model operations strategies such as deploying GPU-based LLM inference servers, canary rollouts, and queue-based autoscaling, upgrades and failure recovery, advanced Helm covering chart structure, hooks, and subcharts, moving from Ingress to Contour/Gateway API, Proxy Protocol settings per LB and Contour version and handling internal calls, and monitoring β the advanced topics needed to run Kubernetes in production. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.
Key perspective
Infrastructure / operations
Rather than memorizing install commands, understand the order in which traffic, runtime, observability, and incident response connect.
Autoscaling & zero-downtime deploymentRBAC-based permission managementOperating StatefulSets/DaemonSets/CronJobsLLM inference servers on GPU nodes & AI model operations strategyAdvanced Helm charts & GitOps integrationMoving from Ingress to Contour/Gateway API & policy settingsPreserving the client IP with Proxy Protocol & handling internal calls
Architecture diagrams
To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning Kubernetes Advanced.
Learning flow
Rendering diagramβ¦
Architecture view
Rendering diagramβ¦
HPA & VPA autoscaling
When you first open Kubernetes Advanced, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.
A service with uneven traffic should use autoscalers rather than fixed replicas, adjusting the number of Pods (HPA) or the container resources (VPA) automatically to match the load. Turning both on for the same resource with CPU/memory targets makes them conflict, so it is safer to run them separately: the HPA on CPU/custom metrics and the VPA for request recommendations only (updateMode: Off).
# Install Metrics Server (a required HPA dependency)kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml# Check the statuskubectl get hpa myapp-hpa --watchkubectl describe hpa myapp-hpa # current metric values and scaling eventskubectl top pods # live CPU/memory per Pod
vpa.yamlYAML
apiVersion: autoscaling.k8s.io/v1kind: VerticalPodAutoscalermetadata: name: myapp-vpaspec: targetRef: apiVersion: apps/v1 kind: Deployment name: myapp updatePolicy: updateMode: "Off" # only computes recommendations, applied by hand β safe for production resourcePolicy: containerPolicies: - containerName: '*' minAllowed: { cpu: 50m, memory: 64Mi } maxAllowed: { cpu: 2, memory: 4Gi }
Autoscaler
What it adjusts
Prerequisite
Caution
HPA
Number of Pod replicas (horizontal scaling)
Metrics Server or Prometheus Adapter
With custom metrics, a scale-down delay (stabilizationWindow) needs to be set
VPA
Container requests/limits (vertical scaling)
The vertical-pod-autoscaler component
updateMode: Auto causes Pod restarts β risky for stateful workloads
Cluster Autoscaler
Number of worker nodes
Integration with the Auto Scaling Group of the cloud provider
Adding nodes is triggered only when Pods are unschedulable (Pending)
RBAC & security
Here you look at RBAC & security alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Role-Based Access Control (RBAC) separates API access rights per team and per service. ServiceAccounts are used to control the permissions of processes inside Pods, and Roles/ClusterRoles to separate user and team permissions.
# Create a kubeconfig for an external user (developer)# 1. Generate the private key & CSRopenssl genrsa -out devuser.key 4096openssl req -new -key devuser.key -out devuser.csr \ -subj "/CN=devuser/O=dev-team"# 2. Submit a K8s CertificateSigningRequestcat > devuser-csr.yaml << EOFapiVersion: certificates.k8s.io/v1kind: CertificateSigningRequestmetadata: name: devuserspec: request: $(base64 -w 0 devuser.csr) signerName: kubernetes.io/kube-apiserver-client expirationSeconds: 7776000 # 90 days usages: - client authEOFkubectl apply -f devuser-csr.yaml# 3. Approve the CSRkubectl certificate approve devuser# 4. Extract the certificate & build the kubeconfigkubectl get csr devuser -o jsonpath='{.status.certificate}' | base64 -d > devuser.crtkubectl config set-credentials devuser \ --client-key=devuser.key \ --client-certificate=devuser.crt \ --embed-certs=truekubectl config set-context devuser-context \ --cluster=kubernetes \ --user=devuser \ --namespace=production# 5. Grant permissions with a RoleBindingkubectl create rolebinding devuser-read \ --role=read-only \ --user=devuser \ --namespace=production
StatefulSet vs stateless (Deployment)
Here you look at StatefulSet vs stateless (Deployment) alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
The most important criterion when choosing a workload type in Kubernetes is "does the Pod hold state (data) itself?". A Deployment treats Pods as interchangeable replicas, while a StatefulSet gives each Pod a unique ID, a stable network name, and independent storage.
Rendering diagramβ¦
Aspect
Deployment (Stateless)
StatefulSet (Stateful)
Pod name
Random suffix (app-7d4f9c-xkz)
Fixed ordinal (app-0, app-1, app-2)
Pod replacement
Created and deleted in any order
Ordered (created 0β1β2, deleted 2β1β0)
Network ID
Shared Service ClusterIP, changing Pod IPs
Fixed DNS such as app-0.svc, app-1.svc through a Headless Service
Storage
Data is lost when the Pod is deleted (no PVC guarantee)
volumeClaimTemplates β a dedicated PVC kept for each Pod
Scale out
Immediate parallel expansion, no ordering
Sequential expansion in the order 0, 1, 2
Rolling update
Random replacement (maxSurge/maxUnavailable)
Sequential replacement in reverse order (2β1β0)
Main uses
API servers, web apps, AI inference servers, and so on
Here you look at Complete guide to StatefulSets alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Using the example of deploying PostgreSQL as a StatefulSet, this section goes step by step through the Headless Service, volumeClaimTemplates, stable network IDs, and operational patterns.
# Deploykubectl apply -f postgres-statefulset.yaml# Check the Pod status (ordered: postgres-1 is created after postgres-0 is Running)kubectl get pods -l app=postgres -w# Check the fixed DNS per Pod# postgres-0.postgres-svc.default.svc.cluster.local# postgres-1.postgres-svc.default.svc.cluster.localkubectl exec -it postgres-0 -- psql -U postgres -c "SELECT inet_server_addr();"# The PVC remains even after the Pod is deletedkubectl delete pod postgres-2kubectl get pvc | grep postgres # postgres-data-postgres-2 is still there# Scale down (deleted in reverse order 2β1β0)kubectl scale statefulset postgres --replicas=1# Rolling update (updateStrategy: RollingUpdate, reverse order 2β1β0)kubectl set image statefulset/postgres postgres=postgres:17-alpinekubectl rollout status statefulset/postgres# Restart one specific Pod only (PVC kept)kubectl delete pod postgres-1
pdb.yamlYAML
# PodDisruptionBudget β keep at least 2 during voluntary disruptions such as node drainsapiVersion: policy/v1kind: PodDisruptionBudgetmetadata: name: postgres-pdbspec: minAvailable: 2 selector: matchLabels: app: postgres
kafka-statefulset.yamlYAML
# Kafka StatefulSet example β a unique ID and independent storage per brokerapiVersion: apps/v1kind: StatefulSetmetadata: name: kafkaspec: serviceName: kafka-svc replicas: 3 selector: matchLabels: app: kafka template: metadata: labels: app: kafka spec: containers: - name: kafka image: bitnami/kafka:3.7 env: - name: KAFKA_BROKER_ID # the broker ID is derived from the Pod ordinal valueFrom: fieldRef: fieldPath: metadata.annotations['statefulset.kubernetes.io/pod-name'] - name: KAFKA_ADVERTISED_LISTENERS value: "PLAINTEXT://$(POD_NAME).kafka-svc:9092" - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name volumeMounts: - name: kafka-data mountPath: /bitnami/kafka volumeClaimTemplates: - metadata: name: kafka-data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 20Gi
StatefulSet deployment workflow
Here you look at StatefulSet deployment workflow alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
A 6-step procedure for deploying a StatefulSet to production safely. Proceed in the order Namespace β StorageClass β Secret β Headless Service β StatefulSet β PodDisruptionBudget.
01-namespace.shBASH
# 1. Set up the Namespace and ResourceQuotakubectl create namespace databasekubectl apply -f - <<EOFapiVersion: v1kind: ResourceQuotametadata: name: db-quota namespace: databasespec: hard: requests.cpu: "4" requests.memory: 8Gi limits.cpu: "8" limits.memory: 16Gi persistentvolumeclaims: "10"EOF
02-storageclass.yamlYAML
# 2. StorageClass β SSD-based dynamic provisioningapiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: fast-ssdprovisioner: kubernetes.io/gce-pd # GKE: gce-pd | EKS: ebs.csi.aws.com | NCP: nks-block-storageparameters: type: pd-ssd replication-type: regional-pdreclaimPolicy: Retain # keep the disk even after the Pod/PVC is deletedallowVolumeExpansion: true # allow online volume expansionvolumeBindingMode: WaitForFirstConsumer
# 4. Deploy in the order Headless Service β StatefulSet (the order is mandatory)kubectl apply -f postgres-headless-svc.yaml -n databasekubectl apply -f postgres-statefulset.yaml -n database# 5. Monitor the Pod startup order (postgres-1 starts after postgres-0 is Running)kubectl get pods -n database -l app=postgres -w# Check the Ready statekubectl rollout status statefulset/postgres -n database# 6. Apply the PodDisruptionBudgetkubectl apply -f postgres-pdb.yaml -n database# Check all resourceskubectl get all,pvc,pdb -n database -l app=postgres
05-scale-update.shBASH
# ββ Scale up/down βββββββββββββββββββββββββββββββββββββββββββββ# Scale up: 3 β 5 (created sequentially in the order 3, 4)kubectl scale statefulset postgres --replicas=5 -n databasekubectl get pods -n database -l app=postgres -w# Scale down: 5 β 3 (deleted in the reverse order 4, 3; PVCs are kept)kubectl scale statefulset postgres --replicas=3 -n database# PVCs are not deleted β if manual cleanup is neededkubectl delete pvc postgres-data-postgres-3 -n databasekubectl delete pvc postgres-data-postgres-4 -n database# ββ Rolling update ββββββββββββββββββββββββββββββββββββββββββββ# Update the image (reverse order: pod-2 β pod-1 β pod-0)kubectl set image statefulset/postgres \ postgres=postgres:17-alpine -n database# Update progresskubectl rollout status statefulset/postgres -n database# Pause the update (canary deployment pattern)kubectl patch statefulset postgres -n database \ -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'# partition=2 β only pod-2 gets the new version; pod-0 and pod-1 stay# After verifying, update everything with partition=0# ββ Rollback βββββββββββββββββββββββββββββββββββββββββββββββββkubectl rollout undo statefulset/postgres -n databasekubectl rollout history statefulset/postgres -n database
06-disaster-recovery.shBASH
# ββ Failure recovery scenarios βββββββββββββββββββββββββββββββ# Force-restart a Pod (PVC kept, data preserved)kubectl delete pod postgres-1 -n databasekubectl get pods -n database -w # confirm automatic recreation# Diagnose when a Pod is Pendingkubectl describe pod postgres-0 -n databasekubectl get events -n database --sort-by='.lastTimestamp'# Check the PVC volume statuskubectl get pvc -n databasekubectl describe pvc postgres-data-postgres-0 -n database# Force Pod rescheduling on node failure (when the taint cannot be removed)kubectl delete pod postgres-0 -n database --grace-period=0 --force# View the logs of a specific Pod (including the previous container logs)kubectl logs postgres-0 -n database --previouskubectl logs postgres-0 -n database -f
Real-world samples
Here you look at Real-world samples alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Three production patterns: Redis Sentinel, MySQL Primary-Replica, and Elasticsearch. Each sample is built on the Headless Service + StatefulSet + volumeClaimTemplates structure.
Here you look at Advanced StatefulSets β Parallel Β· volume expansion Β· snapshots alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Once you are comfortable with basic StatefulSet operations, you need the practical patterns of controlling startup order with podManagementPolicy, expanding PVCs with no downtime, and backing up and restoring data with VolumeSnapshots. In particular, the constraint that volumeClaimTemplates cannot be modified directly after the StatefulSet is created is a frequent source of mistakes.
# ββ Online PVC expansion (requires allowVolumeExpansion: true on the StorageClass) ββ# β οΈ volumeClaimTemplates cannot be patched/edited directly after the StatefulSet is created (immutable field)# β the PVCs already created have to be patched individually# 1. Expand the capacity of each PVC directly (10Gi β 50Gi)for i in 0 1 2; do kubectl patch pvc postgres-data-postgres-$i -n database \ -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'done# 2. Check the expansion progress (FileSystemResizePending β disappears when done)kubectl get pvc -n database -wkubectl describe pvc postgres-data-postgres-0 -n database | grep -A3 Conditions# 3. Bring the volumeClaimTemplates capacity in the StatefulSet manifest into line as well# (so new PVCs are created at the same size on the next recreation/new replica)# The spec.volumeClaimTemplates field itself cannot be modified, so when a change is# needed, recreate only the StatefulSet while preserving the Pods with --cascade=orphankubectl delete statefulset postgres -n database --cascade=orphankubectl apply -f postgres-statefulset.yaml -n database # reapply after changing the capacity to 50Gi
volumesnapshot-backup.yamlYAML
# ββ VolumeSnapshot-based backup & restore (requires a CSI driver + VolumeSnapshotClass) ββapiVersion: snapshot.storage.k8s.io/v1kind: VolumeSnapshotClassmetadata: name: csi-snapclassdriver: ebs.csi.aws.com # GKE: pd.csi.storage.gke.io | AKS: disk.csi.azure.comdeletionPolicy: Retain # keep the snapshot even if the original PVC is deleted---# 1. Create a snapshot β a backup before a deployment / large migrationapiVersion: snapshot.storage.k8s.io/v1kind: VolumeSnapshotmetadata: name: postgres-snap-before-upgrade namespace: databasespec: volumeSnapshotClassName: csi-snapclass source: persistentVolumeClaimName: postgres-data-postgres-0---# 2. Restore a new PVC from the snapshot β on failure, restore under a separate name, verify the data, then swapapiVersion: v1kind: PersistentVolumeClaimmetadata: name: postgres-data-restored namespace: databasespec: storageClassName: fast-ssd dataSource: name: postgres-snap-before-upgrade kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: ["ReadWriteOnce"] resources: requests: storage: 50Gi
snapshot-ops.shBASH
# Check the snapshot status (wait until readyToUse: true)kubectl get volumesnapshot -n database -wkubectl describe volumesnapshot postgres-snap-before-upgrade -n database# Mount the restored PVC in a verification Pod to check the data, then swap it for the existing PVC if all is wellkubectl run pg-verify --image=postgres:16-alpine --restart=Never \ --overrides='{"spec":{"containers":[{"name":"pg-verify","image":"postgres:16-alpine","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}],"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"postgres-data-restored"}}]}}' \ -n database
podManagementPolicy
Behavior
Suitable workloads
OrderedReady (default)
Pod-1 is created only after Pod-0 is Running & Ready. Deletion is one at a time too, in reverse order
Where startup order affects data consistency, as with PostgreSQL Primary-Replica
Parallel
Creates/deletes all Pods at the same time. No ordering guarantee, but far faster
Where each node joins the cluster independently, as with Elasticsearch, Cassandra, and Kafka
Complete guide to DaemonSets
Here you look at Complete guide to DaemonSets alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
A DaemonSet deploys exactly one Pod on every node in the cluster (or every node selected by label). When a new node is added a Pod is created automatically, and when a node is removed the Pod is cleaned up with it. It suits infrastructure components that "need one per node", such as log collectors, monitoring agents, and CNI/CSI plugins.
Rendering diagramβ¦
fluent-bit-daemonset.yamlYAML
apiVersion: apps/v1kind: DaemonSetmetadata: name: fluent-bit namespace: loggingspec: selector: matchLabels: { app: fluent-bit } updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 # restart only 1 node at a time β minimizes the window of lost logs template: metadata: labels: { app: fluent-bit } spec: tolerations: # deploy to control-plane nodes too, to collect the master logs as well - key: node-role.kubernetes.io/control-plane effect: NoSchedule serviceAccountName: fluent-bit containers: - name: fluent-bit image: fluent/fluent-bit:3.0 resources: requests: { cpu: 100m, memory: 128Mi } # multiplied by the node count, so keep it minimal limits: { cpu: 300m, memory: 256Mi } volumeMounts: - name: varlog mountPath: /var/log # mount the log directory of the node directly readOnly: true - name: dockercontainers mountPath: /var/lib/docker/containers readOnly: true volumes: - name: varlog hostPath: { path: /var/log } - name: dockercontainers hostPath: { path: /var/lib/docker/containers }
daemonset-ops.shBASH
# Check the rollout per node β READY must equal the number of nodeskubectl get daemonset fluent-bit -n loggingkubectl get pods -n logging -l app=fluent-bit -o wide# Restrict it to run only on certain nodes (e.g. deploy the device plugin only on GPU nodes)kubectl label node gpu-node-1 workload=gpu# add spec.template.spec.nodeSelector: { workload: gpu } to the manifest# Rolling updatekubectl set image daemonset/fluent-bit fluent-bit=fluent/fluent-bit:3.1 -n loggingkubectl rollout status daemonset/fluent-bit -n loggingkubectl rollout history daemonset/fluent-bit -n logging# Confirm automatic scheduling when a new node is addedkubectl get nodes -w
Trait
DaemonSet
Deployment
Setting replicas
Not possible β decided automatically by the number of target nodes
Specified explicitly
Scheduling
Exactly 1 per node; target nodes can be restricted with nodeSelector/affinity
The scheduler places Pods freely on nodes with spare resources
Deploying to control-plane nodes
Deployed only if tolerations allow the NoSchedule taint
Excluded by default
Update strategy
RollingUpdate (maxUnavailable) or OnDelete
RollingUpdate (maxSurge/maxUnavailable) or Recreate
General applications such as web servers and API servers
Complete guide to CronJobs & Jobs
Here you look at Complete guide to CronJobs & Jobs alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
A Job runs a one-off task whose completion must be guaranteed (batch jobs, migrations), and a CronJob creates that Job periodically on a cron schedule. Unless you clearly define the retry count on failure (backoffLimit), the maximum run time (activeDeadlineSeconds), and the concurrency policy (concurrencyPolicy), batch jobs end up in incidents where they retry forever or run in duplicate.
db-migration-job.yamlYAML
apiVersion: batch/v1kind: Jobmetadata: name: db-migration-v2 namespace: productionspec: backoffLimit: 3 # give up after 3 failures (prevents endless retries) activeDeadlineSeconds: 600 # force-terminate after 10 minutes ttlSecondsAfterFinished: 3600 # clean up automatically 1 hour after completion template: spec: restartPolicy: Never containers: - name: migrate image: myapp-migrator:2.1.0 command: ["./migrate.sh", "--target=v2"] resources: requests: { cpu: 200m, memory: 256Mi } limits: { cpu: 500m, memory: 512Mi } envFrom: - secretRef: { name: db-credentials }
nightly-backup-cronjob.yamlYAML
apiVersion: batch/v1kind: CronJobmetadata: name: postgres-nightly-backup namespace: databasespec: schedule: "0 3 * * *" # every day at 3 AM (in the time zone of the cluster kube-controller-manager) timeZone: "America/New_York" # K8s 1.27+ β the time zone can be set on the CronJob itself concurrencyPolicy: Forbid # skip this run if the previous backup is still running startingDeadlineSeconds: 300 # give up on this run if it cannot start within 5 minutes successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 5 # keep failure history a little longer for root cause analysis jobTemplate: spec: backoffLimit: 2 activeDeadlineSeconds: 1800 template: spec: restartPolicy: OnFailure containers: - name: backup image: postgres:16-alpine command: - sh - -c - | pg_dump -h postgres-svc -U postgres mydb | \ gzip > /backup/mydb-$(date +%Y%m%d).sql.gz volumeMounts: - name: backup-storage mountPath: /backup volumes: - name: backup-storage persistentVolumeClaim: { claimName: backup-pvc }
job-cronjob-ops.shBASH
# Run once manually, right now, from the CronJob (test without waiting for the schedule)kubectl create job --from=cronjob/postgres-nightly-backup manual-backup-test -n database# Check the run historykubectl get cronjob postgres-nightly-backup -n databasekubectl get jobs -n database -l job-name --sort-by=.metadata.creationTimestamp# View the Pod logs of a specific Jobkubectl logs -n database -l job-name=postgres-nightly-backup-28912345 --tail=100# Suspend/resume the CronJob (during a deployment or an incident investigation)kubectl patch cronjob postgres-nightly-backup -n database -p '{"spec":{"suspend":true}}'kubectl patch cronjob postgres-nightly-backup -n database -p '{"spec":{"suspend":false}}'# Clean up finished Jobs (manual cleanup when ttlSecondsAfterFinished is not set)kubectl delete job --field-selector status.successful=1 -n database
Field
Applies to
Role
backoffLimit
Job
Maximum number of retries on failure (default 6) β when exceeded, the Job ends as Failed
activeDeadlineSeconds
Job
Time limit for the whole run β force-terminated when exceeded, prevents endless runs
restartPolicy
Job Pod
Never (a new Pod is created on retry) or OnFailure (the same Pod is restarted) β Always is not allowed
ttlSecondsAfterFinished
Job
Deletes the Job/Pod automatically N seconds after completion β stops finished Jobs from piling up
concurrencyPolicy
CronJob
Allow (default, overlap permitted) / Forbid (skip if the previous run is still going) / Replace (cancel the previous run and start anew)
startingDeadlineSeconds
CronJob
How many seconds of delayed start to allow when the scheduled time is missed (controller down and so on)
Number of successful/failed Jobs to keep in history β only the last few are kept for debugging
Deploying a Java Agent StatefulSet
Here you look at Deploying a Java Agent StatefulSet alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
A hands-on guide to deploying a Spring Boot based Java AI Agent as a StatefulSet. Two PVCs are provisioned for each Pod: one storing the Agent state and vector index, and one keeping the audit logs separately.
Here you look at GPU workloads & deploying LLM inference servers alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
AI Agent/LLM serving workloads must request GPU resources precisely and be isolated so they are not scheduled on the same nodes as ordinary CPU Pods. Once the NVIDIA device plugin exposes GPUs as a node resource (nvidia.com/gpu), they can be specified in requests/limits only in whole numbers β unlike CPU/memory, fractional GPU requests are not possible.
BASH
# Install the NVIDIA device plugin β exposes GPU nodes as the nvidia.com/gpu resourcekubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/main/deployments/static/nvidia-device-plugin.yml# Check the GPU nodeskubectl get nodes -o json | jq '.items[].status.capacity."nvidia.com/gpu"'# Put a dedicated taint on GPU nodes β stops ordinary Pods from being scheduled there by mistakekubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule
llm-inference-deploy.yamlYAML
apiVersion: apps/v1kind: Deploymentmetadata: name: llm-inference namespace: ai-agentspec: replicas: 2 selector: matchLabels: { app: llm-inference } template: metadata: labels: { app: llm-inference } spec: nodeSelector: gpu-type: a10g # separate node pools per GPU type and label them tolerations: - key: nvidia.com/gpu operator: Equal value: present effect: NoSchedule containers: - name: vllm image: vllm/vllm-openai:latest args: ["--model", "/models/llama-3-8b", "--gpu-memory-utilization", "0.9"] resources: requests: cpu: "4" memory: 24Gi nvidia.com/gpu: "1" # GPUs can only be requested in whole numbers β no fractions limits: cpu: "8" memory: 32Gi nvidia.com/gpu: "1" volumeMounts: - name: model-cache mountPath: /models readinessProbe: # block traffic until model loading (tens of seconds to minutes) finishes httpGet: { path: /health, port: 8000 } initialDelaySeconds: 60 periodSeconds: 10 failureThreshold: 30 livenessProbe: httpGet: { path: /health, port: 8000 } initialDelaySeconds: 120 periodSeconds: 30 volumes: - name: model-cache persistentVolumeClaim: claimName: llm-model-pvc # model weights are mounted from a PVC, not the image
llm-inference-hpa.yamlYAML
# Autoscaling GPU Pods β custom metrics based on queue length/concurrent requests are recommended over CPUapiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata: name: llm-inference-hpa namespace: ai-agentspec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: llm-inference minReplicas: 1 # GPUs are expensive β keep the minimum low maxReplicas: 6 # cap at the number of GPUs obtainable in the cluster metrics: - type: Pods pods: metric: name: inference_queue_length # custom metric exposed through the Prometheus Adapter target: type: AverageValue averageValue: "5"
Component
Role
Notes
NVIDIA device plugin (DaemonSet)
Exposes GPU nodes as the schedulable resource nvidia.com/gpu
Deployed automatically to GPU nodes only
nodeSelector / taint-toleration
Schedules GPU Pods only on GPU nodes and keeps ordinary Pods off
Putting a NoSchedule taint on GPU nodes is the usual practice
Model weight storage
Mounted from a PVC/object storage instead of baked into the container image
Smaller images and shorter deploys; easy to swap model versions
Readiness probe
Blocks incoming traffic until model loading finishes
LLMs take tens of seconds to minutes to load β be generous with initialDelaySeconds
AI model deployment strategy & operations
Here you look at AI model deployment strategy & operations alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
An LLM inference server carries different operational risks from an ordinary web service. GPUs are expensive, so the minimum replica count has to stay low; changing one model means revalidating tens of GB of weights; and CPU/memory metrics alone cannot tell you how many requests are backed up in the queue right now. Progressive model rollout, queue-based autoscaling, and GPU node failure isolation have to be designed separately to suit these traits.
Rendering diagramβ¦
model-canary-route.yaml β splitting traffic between two model versions with the Gateway APIYAML
apiVersion: gateway.networking.k8s.io/v1kind: HTTPRoutemetadata: name: llm-inference-routespec: parentRefs: - name: main-gateway rules: - backendRefs: - name: llm-inference-stable port: 8000 weight: 95 # verified existing model - name: llm-inference-canary port: 8000 weight: 5 # new model β raise the weight gradually
keda-scaledobject.yaml β autoscaling on queue lengthYAML
# Scale directly on external queue metrics such as Redis/SQS, even without the Prometheus AdapterapiVersion: keda.sh/v1alpha1kind: ScaledObjectmetadata: name: llm-inference-scaler namespace: ai-agentspec: scaleTargetRef: name: llm-inference minReplicaCount: 1 # GPUs are expensive β going down to 0 when idle is worth considering maxReplicaCount: 6 cooldownPeriod: 300 # wait before scaling down β keep it generous, since shrinking and reacquiring GPU Pods is costly triggers: - type: redis metadata: address: redis.ai-agent.svc:6379 listName: inference-queue listLength: "5" # scale out when 5 or more pile up in the queue
gpu-pdb.yaml β preventing simultaneous eviction of GPU PodsYAML
apiVersion: policy/v1kind: PodDisruptionBudgetmetadata: name: llm-inference-pdb namespace: ai-agentspec: minAvailable: 1 # keep at least 1 able to serve even during node maintenance selector: matchLabels: { app: llm-inference }
Operational item
Ordinary web service
LLM inference server
Deployment risk
Swapping one image
When the model weights change, response quality itself can change β needs canary verification first
Scaling signal
CPU/memory utilization is often enough
Scaling on external metrics such as queue length and concurrent requests is more accurate
Recovery cost on failure
Seconds for a Pod restart
Minutes to reacquire a GPU node and reload the model β preventing simultaneous eviction with a PodDisruptionBudget is essential
Operations & upgrades
Here you look at Operations & upgrades alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
A kubectl cheat sheet, node maintenance, cluster version upgrades, etcd backup and restore, and failure diagnosis patterns. Day-to-day operations last many times longer than building the cluster, so having practiced in advance the procedure for safely emptying and returning a node, and means of rolling back on failure such as etcd backups, is what determines how fast you respond to a real incident.
kubectl-cheatsheet.shBASH
# ββ Essential kubectl commands βββββββββββββββββββββββββββββββ# Switch contextkubectl config get-contextskubectl config use-context production# Set the default namespacekubectl config set-context --current --namespace=production# Watch Pod status in real timekubectl get pods -n production -wkubectl get pods -n production -o wide # includes node placement# Logskubectl logs <pod> -f # livekubectl logs <pod> --previous # previous container logskubectl logs <pod> -c <container> --tail=100 # multi-container# Debuggingkubectl describe pod <pod> -n productionkubectl exec -it <pod> -- /bin/shkubectl debug <pod> -it --image=busybox # ephemeral debug container# Port forwarding (local testing)kubectl port-forward svc/postgres-svc 5432:5432 -n databasekubectl port-forward pod/redis-0 6379:6379 -n cache# Resource usagekubectl top nodeskubectl top pods -n production --sort-by=cpu# View eventskubectl get events -n production --sort-by='.lastTimestamp'# Force a restart (rolling restart of a Deployment)kubectl rollout restart deployment/<name> -n production# Detailed resource outputkubectl get all -n productionkubectl get all,pvc,ingress,cm,secret -n production
node-maintenance.shBASH
# ββ Node maintenance (cordon + drain) ββββββββββββββββββββββββ# 1. Block new Pods from being scheduled (existing Pods stay)kubectl cordon k8s-worker-01# 2. Move existing Pods to other nodes (respects PodDisruptionBudgets)kubectl drain k8s-worker-01 \ --ignore-daemonsets \ # exclude DaemonSet Pods from the drain --delete-emptydir-data \ # allow deleting Pods with emptyDir volumes --grace-period=30 \ # termination grace period --timeout=300s # overall timeout# 3. Carry out the maintenance: OS patches, kernel upgrade, disk replacement, and so on# 4. Resume scheduling once the node is backkubectl uncordon k8s-worker-01# 5. Check the node statuskubectl get node k8s-worker-01 -o widekubectl describe node k8s-worker-01 | grep -A5 "Conditions:"
cluster-upgrade.shBASH
# ββ Cluster version upgrade (1.35 β 1.36, example) βββββββββββββββββ# kubeadm can only move up one minor version at a time β no going straight from 1.34 to 1.36; you must pass through 1.35# Always in the order Control Plane β Worker# 0. Pre-checkskubeadm upgrade plan# 1. Upgrade the Control Plane# Update kubeadmapt-mark unhold kubeadmapt-get install -y kubeadm=1.36.0-00apt-mark hold kubeadm# Run the upgradekubeadm upgrade apply v1.36.0# Update kubelet and kubectlapt-mark unhold kubelet kubectlapt-get install -y kubelet=1.36.0-00 kubectl=1.36.0-00apt-mark hold kubelet kubectlsystemctl daemon-reloadsystemctl restart kubelet# 2. Upgrade the Worker nodes (repeat for each node)# On the Control Planekubectl cordon k8s-worker-01kubectl drain k8s-worker-01 --ignore-daemonsets --delete-emptydir-data# On the Worker nodeapt-mark unhold kubeadm kubelet kubectlapt-get install -y kubeadm=1.36.0-00 kubelet=1.36.0-00 kubectl=1.36.0-00apt-mark hold kubeadm kubelet kubectlkubeadm upgrade nodesystemctl daemon-reload && systemctl restart kubelet# On the Control Planekubectl uncordon k8s-worker-01# 3. Confirm the upgrade finishedkubectl get nodes
etcd-backup.shBASH
# ββ etcd backup & restore ββββββββββββββββββββββββββββββββββββ# etcd is the store for all cluster state β regular backups are essential# Backup (on the Control Plane)ETCD_BACKUP_DIR=/backup/etcd/$(date +%Y%m%d-%H%M%S)mkdir -p $ETCD_BACKUP_DIRETCDCTL_API=3 etcdctl snapshot save $ETCD_BACKUP_DIR/snapshot.db \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key# Verify the backupETCDCTL_API=3 etcdctl snapshot status $ETCD_BACKUP_DIR/snapshot.db \ --write-out=table# Scheduled backup with cron (every day at 2 AM)echo "0 2 * * * root ETCDCTL_API=3 etcdctl snapshot save /backup/etcd/$(date +%Y%m%d).db --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key" >> /etc/crontab# ββ Restore procedure ββββββββββββββββββββββββββββββββββββββββ# 1. Stop the kubelet on every Control Plane nodesystemctl stop kubelet# 2. Back up the etcd data directory, then delete itmv /var/lib/etcd /var/lib/etcd.bak# 3. Restore the snapshotETCDCTL_API=3 etcdctl snapshot restore /backup/etcd/20240101.db \ --data-dir=/var/lib/etcd \ --name=k8s-master-01 \ --initial-cluster=k8s-master-01=https://192.168.1.10:2380 \ --initial-cluster-token=etcd-cluster-1 \ --initial-advertise-peer-urls=https://192.168.1.10:2380# 4. Restart the kubeletsystemctl start kubeletkubectl get nodes
troubleshoot.shBASH
# ββ Failure diagnosis patterns βββββββββββββββββββββββββββββββ# When a Pod is Pendingkubectl describe pod <pod> -n <ns># β Check the Events section:# "Insufficient cpu" β ResourceQuota or not enough node resources# "No nodes available" β every node is tainted/cordoned# "PVC not found" β a StorageClass or PV problem# When a Pod is in CrashLoopBackOffkubectl logs <pod> --previous -n <ns>kubectl describe pod <pod> -n <ns># β Check the exit code:# Exit Code 1: application error (check the logs)# Exit Code 137: OOM Kill β raise limits.memory# When a node is NotReadykubectl describe node k8s-worker-01# β Check Conditions: MemoryPressure, DiskPressure, PIDPressuressh k8s-worker-01systemctl status kubeletjournalctl -u kubelet -f# When a service is unreachablekubectl get endpoints <service> -n <ns> # whether Pod IPs are registeredkubectl exec <pod> -- curl http://<service>.<ns>.svc.cluster.localkubectl exec <pod> -- nslookup <service> # check DNS# Image pull failureskubectl describe pod <pod> | grep -A3 "Failed"# β ErrImagePull / ImagePullBackOff# private registry: check the imagePullSecrets settingkubectl create secret docker-registry regcred \ --docker-server=registry.aidevops.us \ --docker-username=<user> \ --docker-password=<pass># Health check script for the whole clusterecho "=== Nodes ===" && kubectl get nodesecho "=== System Pods ===" && kubectl get pods -n kube-system | grep -v Runningecho "=== Failed Pods ===" && kubectl get pods -A | grep -E "Error|CrashLoop|Pending|OOMKilled"echo "=== PVC ===" && kubectl get pvc -A | grep -v Boundecho "=== Recent Events ===" && kubectl get events -A --sort-by='.lastTimestamp' | tail -20
Helm charts
Here you look at Helm charts alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Helm bundles several YAML manifests into one deployable unit (a chart) and lets you reuse it by changing only the values that differ per environment (image tag, replica count, number of GPUs, and so on) in a values file. What makes it fundamentally different from repeating kubectl apply is the revision history β every time helm upgrade runs, the full set of manifests and values at that moment is recorded as a snapshot, so a single helm rollback line returns everything to the state of an earlier revision.
Rendering diagramβ¦
Chart directory structureTEXT
llm-inference-chart/
βββ Chart.yaml # chart name, version, and dependency (subchart) declarations
βββ values.yaml # defaults β the starting point for every environment
βββ values-staging.yaml # staging-only overrides (1 replica, 1 GPU)
βββ values-production.yaml # production-only overrides (3 replicas, GPU type specified)
βββ templates/
β βββ deployment.yaml # template filled in from {{ .Values.* }}
β βββ service.yaml
β βββ hpa.yaml
β βββ _helpers.tpl # name and label helper functions reused across templates
β βββ hooks/
β βββ pre-upgrade-migration.yaml
βββ charts/ # dependency subcharts (e.g. postgresql) β filled by helm dependency update
templates/deployment.yaml β the Deployment from the gpu-inference-deploy section, templatedYAML
# Install & deployhelm create myapp # scaffold a basic charthelm install myapp ./myapp -f values-production.yaml -n production --atomic --wait# --atomic: roll back to the previous state automatically on failure# --wait: the command waits, without returning, until the Pods are Ready# Check the rendered output before actually applying (applies nothing to the cluster)helm template myapp ./myapp -f values-production.yaml# Upgrade & rollbackhelm upgrade myapp ./myapp -f values-production.yaml --set image.tag=1.3.0 --atomichelm history myapp # summary of changes per revisionhelm rollback myapp 2 # return to a specific revision at once# Download subchart dependencies (postgresql and so on)helm dependency update ./myapp# Package & publish the chart to an OCI registry (the current standard Helm distribution method)helm package ./myapphelm push myapp-1.0.0.tgz oci://ghcr.io/myorg/charts
Command/flag
Role
helm template
Previews only the rendered YAML without applying anything to the cluster (dry-run)
--atomic
Rolls back to the previous revision automatically when an upgrade fails β all but mandatory for production deploys
--wait
Returns only after the Pods are actually Ready β lets a CI pipeline judge deploy success accurately
helm.sh/hook (pre-upgrade and so on)
Includes in the chart tasks that must run exactly once before or after a deploy, such as DB migrations
charts/ (subcharts)
Brings in dependent components such as postgresql and redis as charts made by other teams and composes them (umbrella chart)
Contour & Gateway API β replacing NGINX Ingress
Here you look at Contour & Gateway API β replacing NGINX Ingress alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Contour is an L7 Ingress Controller that uses Envoy Proxy as its data plane. Unlike NGINX Ingress, it updates the Envoy configuration through the xDS API with no downtime and no worker process reload on configuration changes, so even with a large number of routes there are no connections dropped by reloads. Routing policy can be declared in two ways β HTTPProxy, a Contour-specific CRD (strong at delegation between teams and fine-grained policies), and the Gateway API, the Kubernetes standard that is not tied to a controller vendor (the three-layer GatewayClass/Gateway/HTTPRoute model). For a new build, look at the Gateway API first; if you need per-team namespace delegation or fine-grained IP/rate limit policies, look at HTTPProxy first.
contour-install.shBASH
# Option A β HTTPProxy-centered (static manifest, creates the IngressClass "contour")kubectl apply -f https://projectcontour.io/quickstart/contour.yamlkubectl get pods -n projectcontour -wkubectl get svc envoy -n projectcontour # check the external IP of the LoadBalancer# Option B β Gateway API-centered (Contour Gateway Provisioner)# 1) Install the Gateway API CRDs (standard channel)kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/standard-install.yaml# 2) Install Contour + the Gateway Provisioner β the GatewayClass "contour" is created automaticallykubectl apply -f https://projectcontour.io/quickstart/contour-gateway-provisioner.yamlkubectl get gatewayclass contour# After that, when you create a Gateway resource, the Provisioner automatically deploys a Contour+Envoy workload dedicated to that Gateway.
gateway.yamlYAML
# The GatewayClass is created automatically when contour-gateway-provisioner.yaml is appliedapiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: contour-gateway namespace: projectcontourspec: gatewayClassName: contour listeners: - name: http protocol: HTTP port: 80 hostname: "*.aidevops.us" allowedRoutes: namespaces: from: All - name: https protocol: HTTPS port: 443 hostname: "*.aidevops.us" tls: mode: Terminate certificateRefs: - kind: Secret name: wildcard-aidevops-us-tls # issued by cert-manager through DNS-01 allowedRoutes: namespaces: from: All
httproute-canary.yamlYAML
# Gateway API HTTPRoute β header-based routing + weighted canary deploymentapiVersion: gateway.networking.k8s.io/v1kind: HTTPRoutemetadata: name: api-route namespace: productionspec: parentRefs: - name: contour-gateway namespace: projectcontour hostnames: - "api.aidevops.us" rules: # With the x-canary: true header, route 100% to the canary service - matches: - headers: - name: x-canary value: "true" backendRefs: - name: api-service-canary port: 8080 # The rest of the traffic is split 90:10 by weight - backendRefs: - name: api-service port: 8080 weight: 90 - name: api-service-canary port: 8080 weight: 10
httpproxy-policy.yamlYAML
# Contour HTTPProxy β root: defines the domain/TLS/global rate limit + delegates paths per team (includes)apiVersion: projectcontour.io/v1kind: HTTPProxymetadata: name: aidevops-root namespace: productionspec: virtualhost: fqdn: api.aidevops.us tls: secretName: wildcard-aidevops-us-tls rateLimitPolicy: local: requests: 100 unit: second burst: 50 includes: - name: api-v1-routes # managed by the team in the production namespace namespace: production conditions: - prefix: /api/v1 - name: api-v2-routes # managed independently by the payments team in its own namespace namespace: team-payments conditions: - prefix: /api/v2 routes: - conditions: - prefix: / services: - name: frontend-service port: 3000---# The delegated child HTTPProxy in the production namespace β the team owns the detailed policy directlyapiVersion: projectcontour.io/v1kind: HTTPProxymetadata: name: api-v1-routes namespace: productionspec: routes: - conditions: - prefix: /api/v1 services: - name: api-service port: 8080 weight: 90 - name: api-service-canary port: 8080 weight: 10 retryPolicy: count: 3 perTryTimeout: 2s timeoutPolicy: response: 10s idle: 60s ipAllowFilterPolicy: # allow only the internal network + the listed ranges - source: 10.0.0.0/8 - source: 203.0.113.0/24
wildcard-dns01-clusterissuer.yamlYAML
# A wildcard domain (*.aidevops.us) cannot be issued through HTTP-01 β a DNS-01 challenge is requiredapiVersion: cert-manager.io/v1kind: ClusterIssuermetadata: name: letsencrypt-dns01spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: admin@aidevops.us privateKeySecretRef: name: letsencrypt-dns01 solvers: - dns01: route53: region: us-east-1 hostedZoneID: ZXXXXXXXXXXXXX # Route53 Hosted Zone ID selector: dnsZones: - "aidevops.us"---apiVersion: cert-manager.io/v1kind: Certificatemetadata: name: wildcard-aidevops-us namespace: projectcontourspec: secretName: wildcard-aidevops-us-tls issuerRef: name: letsencrypt-dns01 kind: ClusterIssuer dnsNames: - "aidevops.us" - "*.aidevops.us"
Item
NGINX Ingress
Contour HTTPProxy
Gateway API HTTPRoute
Applying configuration
Requires a reload β risk of dropped connections with many Ingresses
Applied dynamically with no downtime through Envoy xDS
Same (Contour applies it to Envoy through xDS)
Delegation between teams
Annotation-based; delegation across namespaces is hard
Delegation per path/domain stated with `spec.includes`
Cross-namespace access allowed with ReferenceGrant
Canary/weighted routing
Vendor-specific annotation (canary-weight)
The `services[].weight` field
The `backendRefs[].weight` field
Header-based routing
Needs a combination of annotations
`conditions[].header`
`matches[].headers`
Standard?
Non-standard annotations per controller
Contour-specific CRD
Official Kubernetes SIG standard (k8s.io)
Rate limit / IP policy
Annotations that differ per controller
`rateLimitPolicy`, `ipAllow/DenyFilterPolicy`
Needs a GEP extension or a separate policy controller
Proxy Protocol β preserving the client IP behind an LB
Here you look at Proxy Protocol β preserving the client IP behind an LB alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
With an L4 LoadBalancer in front of Contour (Envoy), a common problem is that the client IP seen in the Envoy access log or by the application is not the real user IP but the IP of the LB or a node. This is because the LB does SNAT (the ip target mode of AWS NLB, Classic ELB, many managed LBs), or kube-proxy does SNAT when forwarding between nodes under externalTrafficPolicy: Cluster. With HTTPS, Envoy terminates TLS, so the L4 LB cannot look into the encrypted bytes and cannot insert an X-Forwarded-For header either.
Proxy Protocol is a convention proposed by HAProxy in which the L4 LB, as soon as it opens a TCP connection to the backend, first sends a short header at the very start of the connection carrying the original client IP/port, and then passes the original HTTP/TLS bytes through unchanged. v1 is a single line of human-readable text and v2 is a binary format; v2 can carry TLV extensions (the AWS VPC Endpoint ID and so on). The envoy.filters.listener.proxy_protocol listener filter of Envoy recognizes v1/v2 automatically, reads the header, and then replaces the downstream remote address of the following requests with the original IP from the header β so the X-Forwarded-For and X-Envoy-External-Address that Envoy sends upstream, the access log, and ipAllowFilterPolicy decisions all work on the real client IP.
β οΈ
The key rule β the sending side and the receiving side must be turned on at the same time The Proxy Protocol spec defines that the receiver must not "guess" whether a header is present, and that a listener requiring the header rejects connections without one. Turning Proxy Protocol on in Contour attaches this filter to every traffic listener of Envoy (HTTP 8080, HTTPS 8443), so all in-cluster calls that reach Envoy directly without going through the LB are cut off. Conversely, turning on only the LB and not Envoy makes the header get parsed as HTTP/TLS data, causing 400 Bad Request or TLS handshake failures. The subsections that follow in this guide work through these two problems by LB, by Contour version, and by alternative approach.
Rendering diagramβ¦
proxy-protocol-header-example.txtTEXT
# v1 (text) β one line is sent first right after connecting, followed by the original HTTP/TLS data
PROXY TCP4 203.0.113.7 10.0.12.34 51234 443\r\n
# protocol source IP LB/dest IP source port dest port
# v2 (binary) β starts with a 12-byte signature
\x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A + version/command + addresses + TLVs (optional)
# Envoy listener filter configuration (the form Contour generates when useProxyProtocol=true)
listener_filters:
- name: envoy.filters.listener.proxy_protocol
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol
# allow_requests_without_proxy_protocol: false β the default. Contour offers no option to change this value
- name: envoy.filters.listener.tls_inspector # on the HTTPS listener, runs next to check the SNI
Way of preserving the client IP
Suitable environment
Pros
Caution
externalTrafficPolicy: Local
When the LB forwards packets as-is with no SNAT (passthrough) β GKE/AKS L4 LB, MetalLB, AWS NLB instance targets
One line of Service configuration, no effect on internal calls
Nodes with no Envoy Pod drop out of the LB health check β traffic can be uneven across nodes
Proxy Protocol
When the LB does SNAT/proxying β AWS NLB ip targets, Classic ELB, NCP (Naver Cloud), DigitalOcean, HAProxy/F5, and so on
Delivers the original IP even though Envoy terminates TLS; independent of externalTrafficPolicy
LB and Envoy must be configured together, internal calls with no header are blocked, hairpin traffic problem
L7 LB + X-Forwarded-For
When an L7 LB such as ALB or Application Gateway terminates TLS first
No Proxy Protocol needed, easy WAF integration
The Contour num-trusted-hops setting is required; TLS is terminated twice
Proxy Protocol settings per LB
Here you look at Proxy Protocol settings per LB alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
It is the LB that adds the Proxy Protocol header. In Kubernetes, you put a cloud-specific annotation on the type: LoadBalancer Service that exposes Envoy so that the Cloud Controller Manager (or the AWS Load Balancer Controller) turns Proxy Protocol on for the LB listener/target group. With Proxy Protocol the original IP travels in the header, so externalTrafficPolicy can stay at Cluster, which is actually better for balancing traffic across nodes.
Health check caution: if the LB health check does not send the PROXY header when connecting to the traffic port, or conversely sends the header to a port that does not understand it (the Envoy 8002 /ready, the kube-proxy healthCheckNodePort), the targets go unhealthy. Check in the cloud documentation "whether health checks also include the Proxy Protocol header", and if unsure, start with a TCP health check on the traffic port, confirm the behavior, and then move to an HTTP health check.
The Envoy listener filter recognizes both v1 and v2 automatically, so the Contour-side configuration is the same whichever version the LB sends.
envoy-service-aws-nlb.yamlYAML
# When installed from the static manifest (contour.yaml) β add the annotation to the envoy Service# With the Gateway Provisioner, this Service is managed by the Provisioner, so do not edit it directly;# set it through ContourDeployment.spec.envoy.networkPublishing.serviceAnnotations (next section)apiVersion: v1kind: Servicemetadata: name: envoy namespace: projectcontour annotations: service.beta.kubernetes.io/aws-load-balancer-type: external service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" # PP v2 on every target groupspec: type: LoadBalancer externalTrafficPolicy: Cluster # the original IP travels in the PROXY header, so Cluster is enough selector: app: envoy ports: - name: http port: 80 targetPort: 8080 protocol: TCP - name: https port: 443 targetPort: 8443 protocol: TCP
aws-verify-target-group.shBASH
# Check that Proxy Protocol v2 is really on for the NLB target groupsaws elbv2 describe-target-groups \ --load-balancer-arn "$NLB_ARN" \ --query "TargetGroups[].TargetGroupArn" --output text |tr '\t' '\n' | while read -r TG; do echo "== $TG" aws elbv2 describe-target-group-attributes --target-group-arn "$TG" \ --query "Attributes[?Key=='proxy_protocol_v2.enabled']"done
haproxy.cfg (on-premises L4 front end)TEXT
frontend fe_https
mode tcp
bind :443
default_backend be_envoy_https
backend be_envoy_https
mode tcp
balance roundrobin
# send-proxy-v2: send the PROXY v2 header when connecting to the backend (Envoy NodePort)
# check-send-proxy: add the header to health check connections too, so the Envoy listener does not reject them
server node1 10.0.0.11:30443 send-proxy-v2 check check-send-proxy
server node2 10.0.0.12:30443 send-proxy-v2 check check-send-proxy
nginx-stream.conf (on-premises L4 front end)NGINX
stream { upstream envoy_https { server 10.0.0.11:30443; server 10.0.0.12:30443; } server { listen 443; proxy_pass envoy_https; proxy_protocol on; # send the PROXY v1 header to the upstream (Envoy) }}
Client IP preservation is off by default in ip target mode β Proxy Protocol is effectively required. Check with the target group attribute proxy_protocol_v2.enabled
AWS Classic ELB (in-tree, legacy)
The same annotation as above
v1
NLB is recommended for new builds
GKE L4 (passthrough NLB)
Not applicable β use externalTrafficPolicy: Local
-
Packets are forwarded as-is, so the original IP is already preserved
Azure LB (AKS)
Not applicable β use externalTrafficPolicy: Local
-
Only when going through Private Link Service: service.beta.kubernetes.io/azure-pls-proxy-protocol: "true"
MetalLB (L2/BGP) does not add Proxy Protocol β use the Local policy
Envoy Proxy Protocol settings per Contour version (1.28 - 1.33)
Here you look at Envoy Proxy Protocol settings per Contour version (1.28 - 1.33) alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
There are three ways to make Envoy accept the PROXY header in Contour, depending on how it was installed. The result is the same in every case β Contour adds envoy.filters.listener.proxy_protocol at the front of the listener_filters of the ingress_http/ingress_https listeners of Envoy. It is not applied to the stats listener (8002) or the admin interface.
Install method
Where to configure
Field/flag
Static manifest (quickstart contour.yaml)
The contour serve arguments of the contour Deployment
--use-proxy-protocol
Using the ContourConfiguration CRD (contour serve --contour-config-name)
ContourConfiguration
spec.envoy.listener.useProxyProtocol: true
Gateway Provisioner (Gateway API)
ContourDeployment (the parametersRef of the GatewayClass)
Caution: the Contour configuration file (contour.yaml in the contour ConfigMap) has no key corresponding to Proxy Protocol. It is ignored if you put one in the ConfigMap, so use the flag or the CRD.
The version table below follows the official Contour compatibility matrix. Between 1.28 and 1.33, the way Proxy Protocol is configured did not change; what differs is the bundled Envoy version and the supported Kubernetes and Gateway API versions. In particular, in 1.33 the bundled Envoy jumped from 1.35 β 1.38 midway through the patch releases (at 1.33.6), so even for a patch upgrade within 1.33.x, check the behavior changes in the Envoy release notes and verify in staging first.
A-static-manifest-flag.shBASH
# [Option A] Static manifest (quickstart contour.yaml) β add the flag to contour servekubectl -n projectcontour patch deployment contour --type=json -p='[ {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--use-proxy-protocol"}]'kubectl -n projectcontour rollout status deployment/contourkubectl -n projectcontour get deploy contour \ -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n'# ["serve", "--incluster", ..., "--use-proxy-protocol"]# Note: re-applying the quickstart manifest removes the flag.# Manage the patch alongside it as a Kustomize patch or in your GitOps repository.
B-contourconfiguration.yamlYAML
# [Option B] ContourConfiguration CRD β used when running with contour serve --contour-config-name=contourapiVersion: projectcontour.io/v1alpha1kind: ContourConfigurationmetadata: name: contour namespace: projectcontourspec: envoy: listener: useProxyProtocol: true # adds the proxy_protocol listener filter to every Envoy traffic listener network: numTrustedHops: 0 # 0 if only L4 (PP) is in front β if there is one more L7 proxy hop, set that many # ... keep the existing settings (gateway, httpproxy, tls, and so on) as they are
C-contourdeployment-provisioner.yamlYAML
# [Option C] Gateway Provisioner β manage the Contour+Envoy settings separately per GatewayClassapiVersion: projectcontour.io/v1alpha1kind: ContourDeploymentmetadata: name: contour-external-pp namespace: projectcontourspec: runtimeSettings: # = the same structure as the ContourConfiguration spec envoy: listener: useProxyProtocol: true envoy: networkPublishing: type: LoadBalancerService externalTrafficPolicy: Cluster serviceAnnotations: # passed as-is to the envoy Service the Provisioner creates service.beta.kubernetes.io/aws-load-balancer-type: external service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: contour-external-ppspec: controllerName: projectcontour.io/gateway-controller parametersRef: group: projectcontour.io kind: ContourDeployment name: contour-external-pp namespace: projectcontour---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: external-gateway namespace: projectcontourspec: gatewayClassName: contour-external-pp listeners: - name: http protocol: HTTP port: 80 allowedRoutes: namespaces: from: All - name: https protocol: HTTPS port: 443 hostname: "*.aidevops.us" tls: mode: Terminate certificateRefs: - kind: Secret name: wildcard-aidevops-us-tls allowedRoutes: namespaces: from: All
D-helm-values.yamlYAML
# [For reference] When installed from a Helm chart β key names differ per chart, so always check:# helm show values <repo>/contour --version <chart-version> | grep -n -i -E "extraArgs|proxy|annotations"contour: extraArgs: - --use-proxy-protocolenvoy: service: externalTrafficPolicy: Cluster annotations: service.beta.kubernetes.io/aws-load-balancer-type: external service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
check-contour-envoy-version.shBASH
# Check the Contour / Envoy image versions of the current cluster β find the matching row in the table abovekubectl -n projectcontour get deploy,ds \ -o custom-columns='NAME:.metadata.name,IMAGES:.spec.template.spec.containers[*].image'# NAME IMAGES# contour ghcr.io/projectcontour/contour:v1.33.x# envoy ghcr.io/projectcontour/contour:v1.33.x,docker.io/envoyproxy/envoy:v1.35.x# Check that the setting really reached the Envoy listeners (Envoy admin: 9001 inside the Pod)ENVOY_POD=$(kubectl -n projectcontour get pod -l app=envoy -o jsonpath='{.items[0].metadata.name}')kubectl -n projectcontour port-forward "$ENVOY_POD" 9001:9001 >/dev/null &sleep 2curl -s localhost:9001/config_dump | grep -c "envoy.filters.listener.proxy_protocol"# 2 or more (ingress_http + ingress_https) means applied; 0 means not applied# Note: an Envoy created by the Provisioner may have different labels: kubectl get pod -n projectcontour --show-labels
Contour
Bundled Envoy
Kubernetes
Gateway API
Proxy Protocol configuration
Allowing connections without PROXY (allow_requests_without_proxy_protocol)
1.28.x
1.29.x
1.27 - 1.29
v1.0.0
Flag / ContourConfiguration / ContourDeployment
Supported by Envoy, not exposed by Contour β cannot be set
1.29.x
1.30.x
1.27 - 1.29
v1.0.0
Same
Cannot be set
1.30.x
1.31.x
1.28 - 1.30
v1.1.0
Same
Cannot be set
1.31.x
1.34.x
1.30 - 1.32
v1.2.1
Same
Cannot be set
1.32.x
1.34.x
1.31 - 1.33
v1.2.1
Same
Cannot be set
1.33.0 - 1.33.5
1.35.x
1.32 - 1.34
v1.3.0
Same
Cannot be set
1.33.6 and later
1.38.x
1.32 - 1.34
v1.3.0
Same
Cannot be set β see the next section for alternatives
Allowing internal calls with no PROXY header & alternatives
Here you look at Allowing internal calls with no PROXY header & alternatives alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
The problem that blows up most often after turning Proxy Protocol on is calls that reach Envoy without going through the LB. There are three typical paths.
β In-cluster Pod β Envoy Service (ClusterIP) called directly β for example, requests to envoy.projectcontour.svc.cluster.local in order to go through the internal API gateway policies. There is no PROXY header, so Envoy drops the connection. β‘ In-cluster Pod β external domain (LB IP) call (hairpin) β when the LB records an IP in the Service status.loadBalancer.ingress, kube-proxy intercepts packets bound for that IP on the node and sends them straight to Envoy without going through the LB. As a result they arrive with no PROXY header and fail. This usually does not happen with LBs that record only a hostname, like AWS NLB, and does happen in environments that record an IP, such as DigitalOcean, Hetzner, and some on-premises LBs. β’ Corporate network/VPN β internal LB β Envoy β this also fails if Proxy Protocol is not turned on for the internal LB.
The Envoy allow_requests_without_proxy_protocol option The proxy_protocol listener filter of Envoy has an allow_requests_without_proxy_protocol: true option that makes "optional acceptance" possible: read the header if present and just pass the connection through if not. Every Envoy bundled with Contour 1.28 - 1.33 supports this field. However, Contour does not expose this field β there is no corresponding option anywhere in the Contour configuration file, the contour serve flags, or ContourConfiguration/ContourDeployment, and because Contour keeps regenerating the listeners through xDS, there is no arbitrary patch path like the EnvoyFilter of Istio either. So up to Contour 1.33, "read PROXY if present and allow it if absent" cannot be achieved by configuration, and you have to choose one of the alternatives below (check the Contour release notes for whether an option has been added in newer versions).
β οΈ
Points to watch even where the allow option is available The Envoy documentation warns that this option breaks compliance with the Proxy Protocol spec and should be turned on "only when all traffic entering the listener comes from a trusted source". For connections with no header, the TCP source IP (the node/SNAT IP) is recorded as the client IP, so IP-based allow policies can get mixed up; and conversely, a connection that bypasses the LB and comes in directly from outside can impersonate any IP by forging the PROXY header. Also, very short requests of 12 bytes or fewer that match the v2 signature (6 bytes or fewer for v1) time out because Envoy cannot tell whether they are a header. Restrict direct access to the Envoy listeners (NodePort/Pod IP) to the LB and trusted ranges with NetworkPolicy and security groups.
Rendering diagramβ¦
option2-internal-gateway.yamlYAML
# [β‘ recommended] Internal-only Gateway β Proxy Protocol OFF + ClusterIP ServiceapiVersion: projectcontour.io/v1alpha1kind: ContourDeploymentmetadata: name: contour-internal namespace: projectcontourspec: runtimeSettings: envoy: listener: useProxyProtocol: false # the internal one accepts connections with no header envoy: networkPublishing: type: ClusterIPService # reachable only from inside the cluster # To expose it to the corporate network/VPN as well, use LoadBalancerService + an internal LB annotation: # type: LoadBalancerService # serviceAnnotations: # service.beta.kubernetes.io/aws-load-balancer-scheme: internal---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: contour-internalspec: controllerName: projectcontour.io/gateway-controller parametersRef: group: projectcontour.io kind: ContourDeployment name: contour-internal namespace: projectcontour---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: internal-gateway namespace: projectcontourspec: gatewayClassName: contour-internal listeners: - name: http protocol: HTTP port: 80 allowedRoutes: namespaces: from: All---# Attach one HTTPRoute to both the external (PP on) and internal (PP off) Gateways β a single set of routing rulesapiVersion: gateway.networking.k8s.io/v1kind: HTTPRoutemetadata: name: api-route namespace: productionspec: parentRefs: - name: external-gateway # the PP on Gateway from the previous section namespace: projectcontour - name: internal-gateway namespace: projectcontour hostnames: - "api.aidevops.us" rules: - backendRefs: - name: api-service port: 8080# Internal Pods call the internal envoy Service created by the Provisioner:# kubectl -n projectcontour get svc # look for a name of the form envoy-internal-gateway# curl -H "Host: api.aidevops.us" http://envoy-internal-gateway.projectcontour.svc.cluster.local/
option2-static-second-instance.yamlYAML
# [β‘ static manifest version] Running a second Contour+Envoy set without the Provisioner# - Deploy a copy of the quickstart manifest as the second set in a separate namespace (e.g. projectcontour-internal)# - Distinguish the IngressClass each handles with contour serve arguments, and add the PP flag to the external set only# external: contour serve ... --ingress-class-name=contour-external --use-proxy-protocol# internal: contour serve ... --ingress-class-name=contour-internalapiVersion: projectcontour.io/v1kind: HTTPProxymetadata: name: api-internal namespace: productionspec: ingressClassName: contour-internal # only the internal set handles this HTTPProxy virtualhost: fqdn: api.internal.aidevops.us routes: - conditions: - prefix: / services: - name: api-service port: 8080
option3-client-sends-proxy-header.shBASH
# [β’] Having the client send the PROXY header itself# curl β for tests and health check scripts (sends a v1 header)curl --haproxy-protocol -H "Host: api.aidevops.us" \ http://envoy.projectcontour.svc.cluster.local/healthz# curl 8.2+ β specify the client IP to put in the headercurl --haproxy-protocol --haproxy-clientip 10.1.2.3 -H "Host: api.aidevops.us" \ http://envoy.projectcontour.svc.cluster.local/# For application traffic, HTTP libraries mostly do not support the PROXY header, so# put a TCP proxy sidecar in the same Pod and have the app call localhost (HAProxy example):# frontend local_in# mode tcp# bind 127.0.0.1:18080# default_backend envoy# backend envoy# mode tcp# server contour envoy.projectcontour.svc.cluster.local:80 send-proxy-v2# The app sends requests to http://127.0.0.1:18080 + sets the Host header
option4-hairpin.yamlYAML
# [β£] Blocking the hairpin β making internal Pods go through the LB even when calling the external domain# (a) Kubernetes 1.32+ (LoadBalancerIPMode GA): when the Cloud Controller records ipMode: Proxy,# kube-proxy does not intercept the LB IP on the node and sends it to the real LB β the PROXY header is added# Check:# kubectl -n projectcontour get svc envoy -o jsonpath='{.status.loadBalancer.ingress}'# [{"ip":"203.0.113.10","ipMode":"Proxy"}] β Proxy is OK; VIP or absent means the hairpin occurs# ipMode is set by the Cloud Controller Manager, not the user β check whether your CCM supports it# (b) Per-LB annotations that record a hostname in the status instead of an IPapiVersion: v1kind: Servicemetadata: name: envoy namespace: projectcontour annotations: # DigitalOcean service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true" service.beta.kubernetes.io/do-loadbalancer-hostname: "lb.aidevops.us" # Hetzner Cloud # load-balancer.hetzner.cloud/uses-proxyprotocol: "true" # load-balancer.hetzner.cloud/hostname: "lb.aidevops.us"spec: type: LoadBalancer selector: app: envoy ports: - name: https port: 443 targetPort: 8443
option6-envoy-gateway.yamlYAML
# [β₯] When moving to Envoy Gateway β enable Proxy Protocol + optionally accept connections with no header# 1) Enable Proxy Protocol with a ClientTrafficPolicy# (recent Envoy Gateway offers an optional setting in the proxyProtocol block β check the API docs for your version)apiVersion: gateway.envoyproxy.io/v1alpha1kind: ClientTrafficPolicymetadata: name: enable-proxy-protocol namespace: envoy-gateway-systemspec: targetRefs: - group: gateway.networking.k8s.io kind: Gateway name: eg enableProxyProtocol: true---# 2) On a version with no optional field, patch the Envoy field directly with an EnvoyPatchPolicy# (requires extensionApis.enableEnvoyPatchPolicy: true in the EnvoyGateway configuration)apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyPatchPolicymetadata: name: pp-allow-without-header namespace: envoy-gateway-systemspec: targetRef: group: gateway.networking.k8s.io kind: Gateway name: eg type: JSONPatch jsonPatches: - type: "type.googleapis.com/envoy.config.listener.v3.Listener" name: envoy-gateway-system/eg/http # <namespace>/<gateway>/<listener> operation: op: add path: "/listener_filters/0/typed_config/allow_requests_without_proxy_protocol" value: true# After applying, check the listener_filters order (that index 0 is proxy_protocol) with egctl config envoy-proxy listener
Alternative
Path it solves
Pros
Cons
β Internal calls go straight to the app Service (svc.cluster.local)
Pod β Envoy
Simplest, no extra resources, one fewer hop
Bypasses the rate limits, IP policies, TLS, and routing rules of Envoy
β‘ A separate internal-only Gateway (Envoy) β recommended
Pod β Envoy, corporate network β internal LB
Reuses the routing rules by attaching one HTTPRoute to both the external and internal Gateways
An extra Contour+Envoy set to run (resources, monitoring)
β’ The client sends the PROXY header itself
Pod β Envoy
Reuses the existing Envoy
Most ordinary HTTP client libraries do not support it β an intermediate TCP proxy (HAProxy/Envoy sidecar) is needed
β£ Block the hairpin (ipMode: Proxy / hostname annotation)
Pod β external domain
Internal traffic also goes through the LB and gets the PROXY header
Needs Cloud Controller support; more latency and cost from going through the LB
β€ externalTrafficPolicy: Local instead of Proxy Protocol
Every path (removes PP itself)
The internal call problem disappears at the root
Only possible when the LB is passthrough (no SNAT)
β₯ Move to Envoy Gateway + optional PP
Every path
allow_requests_without_proxy_protocol can be turned on directly
A controller swap (HTTPProxy cannot be migrated; HTTPRoute can be reused)
β¦ A forked/patched Contour build
Every path
Keeps Contour
Patch the listener filter generation code and build it yourself β reapplied on every upgrade, heavy operational burden (not recommended)
Proxy Protocol verification & troubleshooting
Here you look at Proxy Protocol verification & troubleshooting alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Proxy Protocol problems are mostly one of two things: "a mismatch between the LB and Envoy settings" or "a path that does not go through the LB". The symptoms alone look like a network failure, but the shape of the symptom lets you infer almost exactly which side is turned on. Narrow down the cause with the table below, then follow the order in the code block: Envoy configuration β requests with and without the header β statistics β access log.
pp-verify.shBASH
NS=projectcontourENVOY_POD=$(kubectl -n $NS get pod -l app=envoy -o jsonpath='{.items[0].metadata.name}')# 1) Whether the proxy_protocol filter is attached to the Envoy listenerskubectl -n $NS port-forward "$ENVOY_POD" 9001:9001 >/dev/null &PF_PID=$!; sleep 2curl -s localhost:9001/config_dump \ | grep -B2 -A4 "envoy.filters.listener.proxy_protocol"# 2) Proxy Protocol statistics β whether the header parse failure/rejection counters are risingcurl -s localhost:9001/stats | grep -i proxy_protokill $PF_PID# 3) Compare requests with and without the header (debug Pod inside the cluster)kubectl run pp-test -n default --rm -it --restart=Never --image=curlimages/curl -- sh -c ' echo "--- without the PROXY header (failure is expected when PP is on)"; curl -sv -m 5 -H "Host: api.aidevops.us" http://envoy.projectcontour.svc.cluster.local/ 2>&1 | tail -3; echo "--- with a PROXY v1 header (success is expected when PP is on)"; curl -sv -m 5 --haproxy-protocol -H "Host: api.aidevops.us" http://envoy.projectcontour.svc.cluster.local/ 2>&1 | tail -3'# 4) After a request from outside, whether the real public IP shows up in the Envoy access logcurl -s https://api.aidevops.us/ -o /dev/nullkubectl -n $NS logs "$ENVOY_POD" -c envoy --tail=5# The x-forwarded-for / downstream address in the default log format should be your public IP# 5) Hairpin check β if the status has an IP and ipMode is not Proxy, internal calls bypass the LBkubectl -n $NS get svc -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.loadBalancer.ingress}{"\n"}{end}'
envoy-networkpolicy.yamlYAML
# Guard against forged PROXY headers β allow the Envoy traffic ports only from the VPC range where the LB lives# (assumes an ip-target NLB connects from VPC subnet IPs, and internal calls are split off to the internal Gateway)apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata: name: envoy-allow-lb-only namespace: projectcontourspec: podSelector: matchLabels: app: envoy policyTypes: ["Ingress"] ingress: - from: - ipBlock: cidr: 10.0.0.0/16 # LB subnet / VPC CIDR ports: - port: 8080 - port: 8443 - from: # Prometheus metric scraping (8002) - namespaceSelector: matchLabels: kubernetes.io/metadata.name: monitoring ports: - port: 8002
Symptom
Cause
Check / fix
Every external request fails β curl "Empty reply from server", "Connection reset", HTTPS cut off mid-handshake
PP on at Envoy only, off at the LB
Check the LB annotation / target group proxy_protocol_v2.enabled
400 Bad Request over HTTP, "wrong version number" or SSL errors over HTTPS
PP on at the LB only, off at Envoy (the PROXY header is parsed as HTTP/TLS data)
Check whether the proxy_protocol filter is in config_dump β apply useProxyProtocol
External is fine; only internal Pod β Envoy Service calls fail
Internal calls with no PROXY header
Separate internal-only Gateway, or call the app Service directly
External is fine; only internal Pod β external domain calls fail
kube-proxy hairpin (the LB IP is intercepted on the node)
Check ipMode in the Service status, apply the hostname annotation
Every LB target is unhealthy
Mismatch between the PROXY header on health check connections and the target port
Switch to a TCP health check on the traffic port and check again
Works normally, but the client IP in the logs is still the node/LB IP
PP not applied, or there is one more L7 proxy hop in front
Check config_dump; if there is an L7, set numTrustedHops to the number of hops
Very short requests time out intermittently (when using optional acceptance)
Requests of 12 bytes (v1: 6 bytes) or fewer are judged to be waiting for a header
Switch to a split-listener layout instead of the allow option
Contour/Envoy monitoring β Prometheus & Grafana
Here you look at Contour/Envoy monitoring β Prometheus & Grafana alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Contour exposes its own control plane metrics at :8000/metrics, and Envoy, the data plane, exposes xDS sync state, upstream connections, and request outcomes at :8002/stats/prometheus. If you have already installed kube-prometheus-stack (see Setting up the monitoring stack), you only need to add two ServiceMonitors. Be sure to block external exposure of the Envoy admin interface (9001) so it is reachable only from inside the cluster.
servicemonitor-contour-envoy.yamlYAML
apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata: name: contour namespace: monitoring labels: release: kube-prometheus-stack # must match the Helm release name to be scrapedspec: namespaceSelector: matchNames: ["projectcontour"] selector: matchLabels: app: contour endpoints: - port: metrics path: /metrics interval: 30s---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata: name: envoy namespace: monitoring labels: release: kube-prometheus-stackspec: namespaceSelector: matchNames: ["projectcontour"] selector: matchLabels: app: envoy endpoints: - port: metrics path: /stats/prometheus interval: 30s
# Official Envoy Grafana dashboards (grafana.com/dashboards) β usable just by entering the Import ID# Envoy Global: Dashboard ID 11021# Envoy Clusters: Dashboard ID 11022# No Contour dashboard is registered on grafana.com, so take the JSON directly from the repositorygit clone --depth 1 https://github.com/projectcontour/contour.git /tmp/contour# Upload the JSON under /tmp/contour/examples/grafana/ through the Grafana UI β Dashboards β Import# (the path can change between versions, so check the examples/ directory of the tag you actually cloned)
Kubernetes Advanced: frequently asked questions
How do Helm and Kustomize differ?
Helm is a packaging tool that fills values into templates to produce manifests and manages what it installs as versioned releases. Kustomize has no templates; it overlays per-environment changes on base manifests. A common combination is Helm for installing software published by others and Kustomize for managing per-environment differences of your own applications.
What is the difference between Ingress and the Gateway API?
Ingress is the early, simple API for connecting HTTP requests to Services, and its finer features depend on annotations that differ per controller. The Gateway API is its successor: it separates the role that operates the entry point from the role that defines routes, and provides features such as traffic splitting and header-based routing as a standard. For a new setup, look at the Gateway API.
How should I set requests and limits?
requests is the amount the scheduler reserves when placing the Pod, and limits is the ceiling it cannot exceed. A container that exceeds its memory limit is killed (OOMKilled), so measure real usage and leave headroom. Exceeding the CPU limit does not kill the container; it is throttled. Begin by setting requests from observed usage.
Next steps
This section covers Next steps from a practical perspective. Instead of memorizing the concept, focus on the situations in which you would reach for it.