OOMKilled tells you what happened, not why

At 19:59 on a Wednesday in April, the Node process serving our landing page died:

19:59:59  <--- Last few GCs --->
19:59:59  FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
20:00:07  BackOff: Back-off restarting failed container
20:00:18  BackOff: Back-off restarting failed container

The supervisor restarted it. About three and a half minutes later it died again, and the pod settled into the crashloop rhythm every on-call engineer recognizes.

The status line most of these deaths wear is one word: OOMKilled. It tells you what happened. It does not tell you why, and the why matters, because at least three different diseases produce the same symptom, and each one has a different fix. Raise the limit on a real leak and you have only delayed the next crash. Hunt for a leak when the limit was simply set too low and you will burn a day proving that nothing is wrong with your code.

This is a field guide to telling those three apart, drawn from three incidents on our own infrastructure. We run Zeabur, a cloud platform where a team of fewer than ten engineers operates clusters across more than eight clouds, so we collect OOM stories at a decent rate. These are ours, failures included. A pattern to watch for as you read: most of the hour an OOM investigation costs is not thinking, it is reassembling evidence from four different places.

Who actually did the killing

OOM deaths come from two different killers, and they leave different exit codes. That difference is a free diagnosis, and it is the first fork in the investigation.

• The kernel. When a container's cgroup exceeds its memory limit, the kernel's OOM killer sends SIGKILL: exit code 137 (128 + 9), which nothing can catch and nothing cleans up after. Kubernetes reports the container state as OOMKilled, and the evidence lives in node-level events and cgroup accounting, and in the pod state itself:

Last State:  Terminated
  Reason:    OOMKilled
  Exit Code: 137

• The runtime. A garbage-collected runtime with its own heap ceiling, like V8 or the JVM, aborts from the inside when its heap is exhausted, before the kernel ever gets involved: SIGABRT, exit code 134 (128 + 6), and the word OOMKilled never appears in the pod state at all. The pod just says Error and crash-loops. The evidence is a crash log in the container's own output. Our April incident was this kind: V8 hit its old-space limit and aborted, and the container's cgroup limit was never the binding constraint.

So read the exit code before anything else: 137 sends you to cgroup accounting and node events, 134 sends you to the container's own logs, which are often the single most useful artifact in the whole case. And note the trap in the naming: the runtime kind of memory death, the kind that hit us in April, never earns the OOMKilled label you were grepping for.

Three diseases, one symptom

Once you know who did the killing, the question becomes why the process was that big. In our experience the answer is one of three shapes.

A leak grows monotonically, mostly indifferent to load. Something retains references that should have been dropped: a module-scope cache, a listener registry, a growing map keyed by request. Restarts fix it for a while, and the interval between restarts is roughly constant.

An undersized limit produces a process that is behaving exactly as designed, inside a box that was drawn too small. The curve tracks load and peaks with traffic, and the previous container's logs are clean right up to the kill.

A changed baseline is the sneaky one. Nothing is growing without bound; the working set simply moved above the ceiling, usually because a rollout changed the process's behavior. The service was fine for months, then every replica starts dying within minutes of boot.

The reason to classify before fixing is the last column: the three fixes point in different directions.

Three incidents, one of each

Theory is cheap, so here is one real case of each from our own clusters.

The limit that was always too small

A TCP proxy on one of our shared clusters kept getting OOMKilled. The investigation was one comparison long:

memory limit:                128Mi
average usage, normal load:  ~130Mi

Nothing was leaking and nothing had regressed. The limit had been set below the service's real working set, and every busy hour pushed it over the line. We raised the limit to 512Mi and the problem never came back.

This is the boring case, and that is the point. It is also the one we see most often, which is why the usage-versus-limit comparison belongs at the top of the decision tree, not the bottom. A surprising number of OOM investigations should end three minutes after they start.

The rollout with a 23-hour fuse

The April incident from the opening of this post looked nothing like the boring case, and it earned the full investigation.

The evening before, a commit had shipped a promotional banner to our landing page. The change looked harmless in review. Its side effect was not: it added server-side logic to the app shell in a way that disabled Next.js's automatic static optimization, so pages that used to be served as prebuilt HTML started running full server-side rendering on every request, including every request from every crawler scanning our template pages.

Nothing happened for 23 hours. Then evening traffic arrived, and the newly started process crossed its heap limit in 216 seconds. The GC log settled the diagnosis:

Mark-Compact 501.2 (522.0) -> 494.0 (538.8) MB, ...
  (average mu = 0.248, current mu = 0.146)
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

A full collection freed about 7 MB out of a 500 MB heap, and current mu = 0.146 means the process was spending more than 85 percent of its time in garbage collection while still losing ground. That is the signature of a large retained set, not an allocation burst.

Combined with the timing, the picture was clear: the rollout had multiplied SSR traffic, and that traffic was feeding a server-side cache that accumulated across requests by design and was never evicted. Two problems, neither sufficient alone. The latent cache had existed for a long time under low SSR volume; the rollout turned a months-long timer into a five-minute one.

The response came in three layers, which is worth recording because the layering is the lesson:

1. Manual restart to stop the bleeding, which ended the user-facing incident after 17 minutes.
2. An environment patch that raised the runtime heap ceiling and enabled automatic heap snapshots near the limit, which bought a stable hour and an evidence-collection path in case the diagnosis was wrong.
3. The permanent fix: restore static optimization and stop the SSR queries from writing to the accumulating cache.

Each layer made the next one safer to do calmly.

The spike that outran the alert

The third case never showed up as a single OOMKilled pod at all:

• A workload on a shared cluster spiked past what its node could absorb in seconds.
• The node went NotReady and drained.
• It recovered before the five-minute alert threshold had even fired, so the first symptoms we saw were downstream neighbors misbehaving, not the culprit itself.

The fix was not in that workload's configuration. It was policy. Default memory limits on the platform were generous enough that a single service could take down a node, and invisible enough that nobody had consciously chosen them. The immediate change was tightening the NotReady alert from five minutes to one; the real fix was policy, a smaller default with the large allocation as an explicit opt-in. When the same OOM symptom shows up at the fleet level instead of the pod level, the root cause is usually a default somewhere, not a bug.

The OOMKilled triage ladder

Across all three cases the response follows the same ladder, and the diagnosis tells you which rung to stop on.

1. Restart stops the bleeding and proves nothing. It is the right first move during user impact, and it is only ever the first move. If a restart is your whole fix, you have scheduled the next incident.
2. Widen the limit when the numbers say the box is too small, or as a deliberate, temporary purchase of time while a real fix is written. The April patch that raised the heap ceiling was insurance rather than a diagnosis; it also armed the evidence collection for the next crash. Widening the limit on a true leak is the classic mistake: the curve does not care about your ceiling, it just takes longer to reach it.
3. Fix the cause according to the disease: evict or bound the cache, roll back the behavior change, or accept the new baseline and size the limit to it on purpose. The April incident closed with the static optimization restored and the cache writes removed, and the heap has been flat since.

The ladder only works if the diagnosis came first, and the diagnosis is mostly a fixed sequence of read-only questions. One caveat on the third: after a crash the pod has restarted, so a bare kubectl top shows the new process's clean numbers. The curve you actually need, the one that separates a climb from a load-shaped peak, lives in your metrics backend; top is a rough check at best.

kubectl describe pod <pod>              # who killed it, and how many times
kubectl logs <pod> --previous           # what the process said before it died
kubectl top pod <pod>                   # usage vs. limit, right now (history lives in your metrics backend)
kubectl rollout history deploy/<name>   # what changed, and when

Which raises an obvious thought.

The tree is mechanical

Every step in that sequence is a read. None of it modifies anything, and the order barely changes from incident to incident. That makes it exactly the kind of work you can hand off, because the judgment is in reading the answers, not in collecting them. Here is what that hand-off looks like: the same walk, from describe to the commit that changed the memory profile, run by an agent in a seeded demo workspace.

That April evening, the time from first page to a confirmed root cause was about an hour of an engineer's full attention. The reads that filled that hour are the ones in the checklist above. The thinking that decided between leak and limit and baseline still belongs to people; the tab-switching that fed it does not have to.