The Kubernetes Admission Webhook Deadlock: Why failurePolicy: Fail Silently Bricks Cluster Upgrades

In cloud-native platform engineering, dynamic admission controllers—powered by ValidatingWebhookConfiguration and MutatingWebhookConfiguration—are the foundation of Kubernetes governance. Organizations deploy policy engines like OPA Gatekeeper, Kyverno, Datadog Admission Controller, or cert-manager to enforce vital zero-trust constraints: blocking privileged containers, mandating resource limits, injecting service mesh sidecars, and restricting image registries. To satisfy strict security and compliance audits, security teams configure a non-negotiable setting on their webhook declarations: failurePolicy: Fail. The rationale appears airtight: if the security webhook is offline or fails to validate a resource, the Kubernetes API server must reject the object rather than allowing unvetted pods to launch. In production, however, this single configuration directive creates an existential circular dependency: the Admission Webhook Deadlock. During node reboots, cluster version upgrades, or certificate rotations, a temporary outage of the webhook pod causes the API server to reject all incoming pod creation requests—including the webhook controller itself and CoreDNS—permanently bricking the cluster control plane.

Enforcing failurePolicy: Fail on your admission webhooks without excluding system namespaces is a self-inflicted denial of service. When your webhook pod restarts, the API server rejects all new pods—including the webhook itself. Your cluster cannot heal because the tool needed to approve the replacement pod is dead.

The Anatomy of Dynamic Admission: The Control Plane Interceptor

To understand how a cluster bricks itself, one must trace the request path of the Kubernetes API server (kube-apiserver):

1. The Admission Phase: When a developer, CI/CD pipeline, or internal controller submits a pod manifest (POST /api/v1/namespaces/default/pods), the request passes through authentication, authorization (RBAC), and schema validation.
2. The Webhook Dispatch: Before committing the resource to etcd, the API server evaluates active MutatingWebhookConfiguration and ValidatingWebhookConfiguration objects. For each matching rule, the API server halts request processing and sends an HTTPS POST request containing an AdmissionReview JSON payload to the webhook's service endpoint (e.g., https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admit).
3. The Binary Enforcement Policy: The webhook configuration's failurePolicy parameter dictates how the API server behaves if the webhook call fails (due to network timeouts, connection refused errors, 500 internal errors, or expired TLS certificates):
   • failurePolicy: Ignore: The API server logs the failure, bypasses validation, and writes the pod to etcd.
   • failurePolicy: Fail: The API server aborts the transaction immediately, returning an HTTP 500 admission error to the client.

The Chicken-and-Egg Catastrophe: How the Deadlock Snaps Shut

The fatal trap occurs during routine infrastructure lifecycle events—such as an automated EKS/GKE control plane upgrade, worker node AMI rollouts, or a cluster-wide reboot:

1. The Webhook Eviction: As worker nodes drain, the active pods running the admission controller (e.g., gatekeeper-controller-manager in the gatekeeper-system namespace) are terminated.
2. The Scheduling Attempt: The Kubernetes Deployment controller recognizes that desired replicas is 3 and current replicas is 0. It generates a new Pod manifest and submits it to the API server.
3. The Self-Validation Trap: The API server receives the request to create the replacement webhook pod. It inspects its active ValidatingWebhookConfiguration list. The rule matches on all Pod creation events cluster-wide.
4. The Unreachable Endpoint: The API server attempts to contact https://gatekeeper-webhook-service.gatekeeper-system.svc:443. But the service has zero active endpoints because all webhook pods are currently terminated. Or worse: the node reboot terminated CoreDNS, so the API server cannot even resolve internal cluster DNS.
5. The Lockout: The connection attempt fails with dial tcp: connect: connection refused. Because failurePolicy: Fail is active, the API server rejects the creation of the replacement webhook pod:

   Internal error occurred: failed calling webhook "validation.gatekeeper.sh": failed to call webhook: Post "https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admit": dial tcp 10.96.0.15:443: connect: connection refused

6. Total Cluster Paralysis: The replacement webhook pod cannot start because the webhook is not running to approve its own creation. Concurrently, no other pod in the entire cluster can start: CoreDNS cannot reboot, CNI network daemonsets cannot initialize, ingress controllers cannot restart, and application deployments are completely frozen. The cluster is deadlocked.

The Certificate Expiration Timebomb

An even more insidious trigger occurs via automated TLS certificate rotation. Dynamic admission webhooks require valid TLS certificates to communicate with kube-apiserver over HTTPS. Most organizations rely on cert-manager or self-signed webhook secret updaters to rotate these certificates every 90 days.

If cert-manager's controller restarts while a certificate is expiring or if an invalid CA bundle is injected into the ValidatingWebhookConfiguration:

• The API server rejects the webhook's TLS handshake due to certificate expiration.
• The failurePolicy: Fail trips, blocking all pod creation.
• cert-manager attempts to schedule a certificate-issuing pod to generate a new TLS secret.
• The API server rejects cert-manager's pod because the expired webhook is failing validation.

The cluster cannot heal itself because the tool required to fix the certificate is blocked by the very failure it is trying to solve.

Naive Security Hardening vs. Calricon High-Availability Admission Architecture

The Calricon Production Playbook: Bulletproofing Dynamic Admission

Preserving zero-trust policy enforcement while guaranteeing cluster survivability requires strict architectural isolation and defensive webhook configuration.

Step 1: Enforce Absolute Namespace Exclusions

Never permit an admission webhook to validate the core Kubernetes control plane or its own controller namespace. In your ValidatingWebhookConfiguration, configure explicit namespaceSelector rules using matchExpressions:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: "enterprise-security-gatekeeper"
webhooks:
  - name: "validation.gatekeeper.sh"
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
        scope: "Namespaced"
    clientConfig:
      service:
        name: "gatekeeper-webhook-service"
        namespace: "gatekeeper-system"
        path: "/v1/admit"
    failurePolicy: Fail
    timeoutSeconds: 2 # Cap timeout to 2 seconds to protect apiserver
    namespaceSelector:
      matchExpressions:
        - key: "kubernetes.io/metadata.name"
          operator: "NotIn"
          values: ["kube-system", "kube-public", "kube-node-lease", "gatekeeper-system", "cert-manager"]

By strictly excluding kube-system and gatekeeper-system, replacement webhook controllers and CoreDNS pods can be scheduled unconditionally by the API server, ensuring that a dead webhook controller can always boot and recover.

Step 2: Assign System-Critical Scheduling Priority

Ensure that admission controller pods are never evicted to make room for user workloads. Assign them the highest scheduling priority class:

spec:
  template:
    spec:
      priorityClassName: system-cluster-critical
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  control-plane: gatekeeper-controller-manager
              topologyKey: kubernetes.io/hostname

Distribute at least three replicas across distinct physical worker nodes with anti-affinity to ensure that even during multi-node drains, at least one webhook pod remains available.

Step 3: Define Break-Glass Emergency Operational Procedures

Every SRE and Platform team must maintain an emergency runbook for admission deadlocks. If a cluster enters an unrecoverable chicken-and-egg state:

1. Delete the Webhook Configuration Object: Because ValidatingWebhookConfiguration objects are cluster-scoped configuration metadata (not pods), you can delete them via kubectl even when pod creation is deadlocked:

   kubectl delete validatingwebhookconfiguration enterprise-security-gatekeeper --ignore-not-found=true
   kubectl delete mutatingwebhookconfiguration enterprise-security-gatekeeper --ignore-not-found=true

2. Allow System Recovery: With the webhook removed, the API server immediately permits CoreDNS and the webhook deployment to spawn and stabilize.
3. Re-apply Configurations: Once the webhook pods report healthy readiness probes, re-apply the webhook configuration manifest via GitOps pipelines.

Step 4: Instrument Prometheus Alerts for Admission Latencies and Drops

Monitor the health of your admission webhooks directly from kube-apiserver telemetry:

- alert: WebhookAdmissionRejectionSpike
  expr: rate(apiserver_admission_webhook_rejection_count[2m]) > 5
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Kubernetes API server is rejecting pod creation due to admission webhook failures."

- alert: WebhookAdmissionLatencyHigh
  expr: histogram_quantile(0.99, sum(rate(apiserver_admission_webhook_admission_duration_seconds_bucket[5m])) by (le, name)) > 1.0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Admission webhook {{ $labels.name }} P99 latency exceeds 1.0s; risking API server thread pool exhaustion."

Security Controls Must Never Be More Fragile Than the Systems They Protect

Zero-trust policy enforcement is an indispensable requirement of modern enterprise computing. But a security mechanism that destroys cluster availability during an operational disruption is not a control; it is an unforced architectural error.

Audit your Kubernetes admission webhook configurations today. Verify that failurePolicy: Fail is paired with strict namespace exclusions for kube-system and webhook controllers, clamp webhook timeouts to 2 seconds, elevate controller pods to system-cluster-critical, and test your break-glass operational runbooks. In enterprise infrastructure, true reliability is not proven by enforcing rigid rules in calm seas; it is demonstrated by designing safety controls that can survive their own reboot.