CRITICAL

How to Fix Kubernetes 502 Bad Gateway from Ingress-NGINX (K8s 1.30+)

Quick Fix Summary

TL;DR

Check backend service health and verify Ingress-NGINX can reach your Pods on the correct port.

A 502 Bad Gateway from Ingress-NGINX indicates it cannot establish a successful connection to your backend Pods. This is a critical networking or service health issue between the Ingress controller and your application.

Diagnosis & Causes

  • Backend Pods are not running or are crashing.
  • Service selector does not match Pod labels.
  • Service or Pod port configuration is incorrect.
  • NetworkPolicy is blocking Ingress controller traffic.
  • Readiness probe failures causing endpoint removal.
  • Recovery Steps

    1

    Step 1: Verify Backend Pod and Service Health

    First, confirm your application Pods are running and the Service is correctly routing to them. A missing or unhealthy backend is the most common cause.

    bash
    kubectl get pods -l app=my-app
    kubectl describe svc my-app-service
    kubectl get endpoints my-app-service
    2

    Step 2: Check Ingress-NGINX Controller Logs

    Examine the Ingress controller logs for specific connection errors (refused, timeout, reset) to the upstream backend.

    bash
    POD=$(kubectl get pods -n ingress-nginx -l app.kubernetes.io/component=controller -o jsonpath='{.items[0].metadata.name}')
    kubectl logs -n ingress-nginx $POD --tail=50 | grep -A5 -B5 "502"
    3

    Step 3: Validate Network Connectivity from the Ingress Pod

    Exec into an Ingress controller Pod and test connectivity to the backend Service's ClusterIP and Pod IP directly.

    bash
    INGRESS_POD=$(kubectl get pod -n ingress-nginx -l app.kubernetes.io/component=controller -o jsonpath='{.items[0].metadata.name}')
    kubectl exec -it -n ingress-nginx $INGRESS_POD -- sh
    # Inside the pod:
    apk add curl
    curl -v http://<SERVICE_CLUSTER_IP>:<PORT>
    4

    Step 4: Inspect and Fix Service/Ingress Configuration

    Ensure the Ingress resource's `service.name`, `service.port.number` (or `name`) match an existing, correctly configured Service.

    yaml
    kubectl describe ingress my-ingress
    # Critical YAML Check:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    spec:
      rules:
      - host: myapp.example.com
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app-service # Must exist
                port:
                  number: 8080 # Must match Service port
    5

    Step 5: Review and Adjust Readiness Probes

    If Pod readiness probes are failing, the Pod's IP will be removed from the Service Endpoints, causing 502s. Fix the probe or adjust timeouts.

    yaml
    kubectl describe pod my-app-pod | grep -A 10 Readiness
    # Example fixed probe in deployment:
    readinessProbe:
      httpGet:
        path: /health
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
      failureThreshold: 3 # Pod marked unready after 3 failures
    6

    Step 6: Check for Misconfigured Network Policies

    A NetworkPolicy in your application's namespace may be blocking ingress traffic from the `ingress-nginx` namespace.

    yaml
    kubectl get networkpolicy -n my-app-namespace
    # Allow Ingress-NGINX traffic:
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: allow-ingress
    spec:
      podSelector: {}
      policyTypes:
      - Ingress
      ingress:
      - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx

    Architect's Pro Tip

    "For intermittent 502s under load, increase `proxy-read-timeout` and `proxy-next-upstream-tries` in your Ingress-NGINX ConfigMap to handle slow backends more gracefully."

    Frequently Asked Questions

    My Pods are 'Running' but I still get a 502. What's wrong?

    'Running' doesn't mean 'ready'. Check the Pod's `READY` column (e.g., '1/2'). Also, verify the container's process is listening on the port specified in the Service. Use `kubectl logs <pod>` and `kubectl exec <pod> -- netstat -tulpn` inside the Pod.

    Can a Pod Disruption Budget (PDB) cause 502 errors?

    Indirectly, yes. During voluntary disruptions (node drain, rollout), a PDB can block eviction, leaving a single, potentially overloaded or unhealthy Pod to handle traffic, which may time out and cause 502s. Check PDBs and ensure you have sufficient replicas.

    Related Kubernetes Guides