Kubernetes Pod Status OOMKilled (Out Of Memory Killed)

The OOMKilled status indicates that a Kubernetes pod was terminated by the operating system's Out-Of-Memory (OOM) killer. This happens when a container within the pod attempts to use more memory than its configured memory limit or more than is available on the node, leading to its termination to protect the system.

Common Causes

Insufficient memory limits defined for the container in the pod's YAML.
Application memory leak or inefficient memory usage within the container.
Sudden spike in application traffic or workload exceeding expected memory consumption.
Node-level memory pressure affecting multiple pods on the same worker node.
Incorrectly configured application-specific memory settings (e.g., JVM heap size).

How to Fix

1 Increase Pod Memory Limits

Review the current memory limits and requests for the affected container in the pod's YAML definition. Incrementally increase the `resources.limits.memory` value. Monitor the pod's memory usage to find an optimal balance.

BASH

 $ resources:
  limits:
    memory: "768Mi" # Example: Increase from 512Mi to 768Mi
  requests:
    memory: "512Mi" # Ensure requests are also appropriate, typically lower than limits

2 Optimize Application Memory Usage

Analyze the application code for potential memory leaks or inefficient memory patterns. Use profiling tools to identify memory hotspots. For applications like Java, tune JVM heap size parameters (`-Xmx`, `-Xms`) within the application's startup command or Dockerfile.

BASH

 $ # Example for Java application in Dockerfile or entrypoint:
CMD ["java", "-Xmx512m", "-jar", "myapp.jar"]

# Or for Node.js:
CMD ["node", "--max-old-space-size=512", "app.js"]

3 Scale Pods Horizontally

If the OOMKilled event is due to high traffic or increased workload, consider scaling out your deployment by increasing the number of replicas. This distributes the load across more pods, reducing the memory pressure on individual instances.

BASH

$ kubectl scale deployment <deployment-name> --replicas=<new-replica-count>

4 Monitor and Analyze Pod Events and Metrics

Use `kubectl describe pod` to check for specific OOMKilled events and reasons. Leverage Kubernetes monitoring tools (e.g., Prometheus, Grafana) to track historical memory usage of pods and nodes to identify trends or spikes leading to the OOMKilled state.

BASH

 $ kubectl describe pod <pod-name>
kubectl get events --field-selector involvedObject.name=<pod-name>