Linux Kernel / OS Process Killed by Signal 9

SIGKILL is an unblockable, uncatchable signal that immediately terminates a process. It's often sent by the kernel (e.g., OOM killer) or by a user/process with sufficient permissions to forcefully stop a misbehaving or unresponsive application. The process cannot clean up or save its state.

Common Causes

Out-of-Memory (OOM) Killer: The Linux kernel automatically terminates processes to free up memory when the system runs out.
Manual termination: A user or script explicitly used `kill -9` or `pkill -9`.
Parent process termination: A parent process might send SIGKILL to its child processes upon its own termination, especially if it's exiting forcefully.
Container orchestration: Systems like Kubernetes or Docker might send SIGKILL after a grace period for a process that fails to shut down gracefully (SIGTERM).
Resource limits: A process exceeding its configured resource limits (e.g., cgroups, ulimits) might be terminated by the kernel or a resource manager.

How to Fix

1 Investigate OOM Killer Activity

Check kernel logs (dmesg, journalctl) for messages indicating the Out-of-Memory (OOM) killer was invoked, which is a very common reason for SIGKILL. Look for entries like 'Out of memory: Kill process' or 'oom-killer'.

BASH

 $ dmesg -T | grep -i 'oom-killer|killed process'
journalctl -k | grep -i 'oom-killer|killed process'

2 Monitor Resource Usage

Use tools like `top`, `htop`, `free -h`, `vmstat`, or `sar` to identify processes consuming excessive CPU, memory, or I/O resources before the SIGKILL occurs. Implement proactive monitoring with alerts.

BASH

 $ top
free -h
vmstat 1 5

3 Review Application Logs

Examine the application's own logs for any errors, warnings, or indications of resource exhaustion, deadlocks, or unhandled exceptions that might precede a SIGKILL, even if the SIGKILL itself isn't logged by the application.

BASH

 $ tail -f /var/log/your-app/app.log
grep -i 'error|exception|memory' /var/log/your-app/app.log

4 Increase System Resources or Optimize Application

If OOM is the cause, consider increasing RAM, optimizing application memory usage, or adjusting OOM killer parameters (e.g., `vm.overcommit_memory`, `oom_score_adj`). For CPU-bound issues, scale CPU resources or optimize code. For containerized applications, adjust resource requests/limits.

BASH

 $ kubectl describe pod <pod-name> # Check K8s resource limits
# Example: Increase memory for a Docker container
docker run -m 2g my-app

5 Verify Manual Kills or Orchestration Actions

Confirm if a user, script, or orchestration system (e.g., Kubernetes, Docker Swarm) explicitly sent a `kill -9` command or forcefully terminated the container/pod. Check audit logs, CI/CD pipelines, or orchestration events.

BASH

 $ history | grep 'kill -9' # For user history
kubectl get events --field-selector involvedObject.name=<pod-name> # K8s events