Your pods aren't crashing - they're being interrupted mid-sentence

A team ships a routine deploy on a Tuesday afternoon - no schema change, no config drift, just a new image tag on a Deployment with three replicas. The rollout goes green in under a minute. Readiness probes pass, the new pods come up, the old ones terminate, Grafana shows the deploy marker and nothing else. Twenty minutes later someone notices a small but consistent spike in 5xx responses that lines up exactly with the rollout window - a burst of connection-reset errors, a handful of orders that never got a confirmation write, a queue consumer that re-delivered the same message twice. Nothing in the incident timeline says "this failed." The service was healthy before, healthy after, and briefly wrong in between.

This is one of the most common and most invisible reliability bugs in a containerized stack, and it has nothing to do with your application logic. It happens because the process didn't know how to stop.

What actually happens when a Pod is deleted

Kubernetes documents this sequence precisely, and it's worth reading it exactly as written rather than as folklore. When a Pod is deleted - by a rolling update, a kubectl delete, a scale-down, or the control plane evicting it for a node drain - the kubelet does not simply kill the container. It runs a defined shutdown sequence:

1. If a preStop lifecycle hook is configured, it runs first, and the container's management blocks until that hook completes (or the grace period runs out).

2. The kubelet sends SIGTERM to PID 1 in each container.

3. Kubernetes starts (or continues) the countdown set by terminationGracePeriodSeconds - 30 seconds by default if the Pod spec doesn't override it.

4. If the container is still running when that countdown reaches zero, the kubelet sends SIGKILL, which cannot be caught, blocked, or ignored by the process.

This is documented on the Pod Lifecycle page under Pod termination, and the default value of terminationGracePeriodSeconds is stated explicitly there and on the container lifecycle hooks reference. The important framing is that the grace period is not extra time your container gets to notice it's being shut down - it's a hard deadline after which the kernel forcibly ends the process regardless of what it was doing. A database write half-committed, a response half-written to a socket, a message not yet acknowledged back to the broker - none of that is Kubernetes' problem once SIGKILL lands, because SIGKILL offers no opportunity for cleanup at all.

So the first fact to internalize: thirty seconds is not generous, it's a deadline your application has to actively use. If your process doesn't do anything in response to SIGTERM - because the language runtime's default handler just terminates immediately, or because nothing in your code registered a handler and the signal is ignored outright - you get one of two identical-looking failures. Either the process dies the instant SIGTERM arrives, dropping whatever was in flight with no warning, or it does nothing at all and rides out the full 30 seconds before SIGKILL ends it anyway, which is worse: every in-flight request during that window is still being served by a process that a moment later gets hard-killed regardless. Neither path involves an error in your code. Both drop work on a totally healthy service.

The part that in-process signal handling alone doesn't fix

Handling SIGTERM correctly in your application is necessary, but it is not sufficient, and this is the part most write-ups skip. Kubernetes' own documentation on Pod termination and endpoints describes the actual mechanics of what happens to Service traffic during that same window.

When a Pod is marked for deletion, two things start concurrently, not sequentially: the kubelet begins the SIGTERM/grace-period sequence described above, and, separately, the control plane begins updating the EndpointSlice objects behind any Service that selects the Pod - marking the endpoint's ready condition false and terminating true. Removing that endpoint from active rotation, and propagating that removal to every kube-proxy and every controller-managed load balancer or ingress watching the EndpointSlice, is not instantaneous. It takes a real, non-zero amount of time, and during that window some upstream components may still hold open connections or route new ones to a Pod that the kubelet has already sent SIGTERM to.

That's the actual race: your process can be behaving perfectly - draining connections, finishing in-flight work, exiting cleanly - while a load balancer somewhere hasn't yet learned it should stop sending new traffic there. A SIGTERM handler cannot fix a problem that happens outside your process, in the data plane, before your handler even runs.

Kubernetes' own documentation addresses this directly rather than leaving it to folklore: the same termination-flow tutorial ships a worked example that adds a preStop hook doing nothing but waiting, specifically to give the endpoint-removal propagation time to catch up before the container actually starts shutting down:

lifecycle:
  preStop:
    sleep: 15
terminationGracePeriodSeconds: 30

Because the preStop hook runs before SIGTERM is delivered, this buys real time: the Pod keeps serving normally, already-marked-terminating in the endpoint objects, while proxies and load balancers converge on removing it - and only once the hook returns does SIGTERM actually arrive and your application-level shutdown begin. This is not a workaround bolted on top of "proper" signal handling; it's a documented, first-class part of how Kubernetes expects graceful termination to work, because the endpoint-propagation delay is a property of the system, not a bug your application can code around.

What a correct SIGTERM handler in your process needs to do

With the preStop delay covering the traffic-routing race, the application still owns everything from the moment SIGTERM actually arrives to the moment it exits - and that has to happen well inside whatever remains of terminationGracePeriodSeconds. Concretely, a correct handler does four things, in order:

1. Stop accepting new work immediately. Close the listening socket, or at minimum stop the server from accepting new connections, and fail readiness probes if you haven't already (this should really happen during the preStop window, not after). New requests arriving after this point should be refused at the load balancer or immediately rejected, not accepted and then abandoned mid-response.

2. Let in-flight requests finish, with a bound. Existing connections and requests that were already accepted should be allowed to complete normally - that's the entire point of graceful shutdown. But "let them finish" needs an upper bound shorter than the remaining grace period, because if a stuck request never finishes, you want your process to give up and exit cleanly rather than get SIGKILLed with connections and file handles left in an undefined state.

3. Close resources deliberately. Database connection pools, message broker consumers, outbound HTTP clients, file handles - all of these should be shut down explicitly rather than left for the OS to reap via SIGKILL. A consumer that's mid-processing a message and gets killed without acking or explicitly nacking it forces the broker to redeliver after a visibility timeout, which is how you get duplicate processing on top of dropped requests.

4. Exit. Once new work is refused, in-flight work is drained or bounded out, and resources are closed, the process should exit on its own rather than waiting to be killed. This matters because a process that exits at second 12 of a 30-second grace period frees up scheduling and shutdown time cleanly; a process that just sits there until SIGKILL at second 30 gains nothing and makes every rollout slower than it needs to be.

Runtime behavior here varies more than people expect, and it's worth checking rather than assuming. Some language runtimes register a default SIGTERM handler that terminates the process immediately with no drain step at all. Others, particularly when the container's entrypoint is a shell script rather than the application binary directly, never deliver the signal to the actual application process in the first place - the shell swallows it, and the app only dies when SIGKILL arrives and takes down the whole process tree. That's a distinct and very common failure mode: it looks identical to "we didn't handle SIGTERM" from the outside, but the fix is making sure your application binary is actually PID 1 (or that signals get forwarded to it), not just adding a handler that a shell wrapper never lets execute.

Why "the health checks looked fine" is exactly the wrong signal to trust

The scenario at the top of this piece is deliberately unremarkable: no alert fired, no probe failed, the rollout dashboard was green the entire time. That's expected, not surprising, once you see the actual mechanism. Liveness and readiness probes check whether a Pod is currently healthy to receive traffic - they say nothing about whether the handful of requests that were in flight at the exact moment SIGTERM landed were completed or dropped. A rolling deploy can look perfectly clean in every metric that measures steady-state health and still drop a small, consistent percentage of requests on every single termination, forever, because that's not a health problem - it's a shutdown-sequencing problem, and it only shows up if you're specifically looking at error rates correlated with Pod termination timestamps, not general health.

That's also why this class of bug survives so long in real systems: it's small per-incident (a handful of requests per rollout), it's consistent rather than anomalous (so it doesn't trip anomaly-based alerting), and it correlates with an event - deploys, scale-downs, node maintenance - that teams already treat as routine and safe. The fix is not a bigger grace period or a retrying client; it's making sure the two independent timelines - traffic routing convergence and application shutdown - are actually sequenced correctly, with a preStop wait covering the first and a real SIGTERM handler covering the second.

The minimum viable fix

None of this requires a service mesh or a complicated shutdown framework. The concrete checklist:

• Set an explicit terminationGracePeriodSeconds sized to your slowest realistic in-flight request, not the 30-second default by accident.

• Add a preStop hook that sleeps for a few seconds - long enough for endpoint removal to propagate through your specific networking setup - before the container's shutdown begins.

• Register an actual SIGTERM handler in your application: stop accepting new connections, drain or bound in-flight work, close resources, exit.

• Verify your container's entrypoint delivers SIGTERM to your application process, not to a shell that swallows it.

• Fail readiness deliberately as part of shutdown, so any component still checking readiness (rather than relying purely on endpoint state) also learns to stop routing traffic quickly.

Each of these is a few lines of YAML or a few lines of shutdown code. The reason they're worth writing deliberately, rather than inheriting whatever a base image or a runtime's defaults happen to do, is that every one of your deploys, scale-downs, and node reschedules is a shutdown event - and a service that silently drops a little work on every single one of them is failing constantly in a way that never shows up as an incident.

Sources

• Kubernetes documentation - Pod Lifecycle: Pod termination, terminationGracePeriodSeconds

• Kubernetes documentation - Container Lifecycle Hooks (preStop, blocking behavior, grace period interaction)

• Kubernetes documentation - Attach Handlers to Container Lifecycle Events (preStop example)

• Kubernetes documentation - Pod Termination Flow and Service Endpoints tutorial (endpoint removal race, preStop sleep example)