Graceful shutdown

One-pager

What: a worker being removed stops receiving new requests immediately, then finishes the generations it already accepted before the process exits. Why: a severed generation cannot be retried — the tokens already streamed cannot be un-sent — so without this, every rolling upgrade or scale-down produces a burst of client errors. Requires: nothing, for finishing in-flight work; Kubernetes with the default kubernetes discovery backend for the advance notice described below.

Important

Finishing in-flight work happens on every backend, bounded by --drain-timeout. What needs Kubernetes with the default kubernetes discovery backend is the advance notice — the router learning a worker is leaving before the process is signalled. That is not an implementation gap: it relies on the orchestrator knowing a Pod is being removed, which nothing outside Kubernetes can tell the router. discoveryBackend: etcd is rejected by the operator for in-cluster deployments.

What happens

Removing a worker — a rolling update, a scale-down, draining a node — separates two things that would otherwise happen at once:

1. It stops receiving. Kubernetes marks the Pod the moment its removal is requested, which is before the worker process is signalled. The router sees that mark and stops choosing the worker within milliseconds, so new requests go elsewhere while it is still running and long before it is told to stop.

2. It keeps serving. The worker finishes the generations it already accepted, bounded by --drain-timeout, and only then exits.

The early mark is what makes this different from simply stopping a process. The preStop delay that follows is not spent waiting for the router to notice — that already happened — but letting work in progress finish before the process is signalled at all.

Deploying through the operator needs no configuration: it injects the preStop delay and sizes the termination grace period to cover the whole sequence. For hand-written manifests and the per-stage timings, see Scaling a fleet.

When the Pod is not being deleted

A worker can also be stopped without its Pod going anywhere — a liveness probe failing and restarting the container, a node being shut down gracefully, someone killing the process. There is no deletion, so there is no early mark, and the router has no way to know until the worker says so.

On those paths the worker removes its own registration as its first act on SIGTERM, which stops new requests arriving, and drains after. In-flight generations still finish. What is missing is the head start: from the moment the decision is made to the moment the process is signalled, the router is still sending work, because nothing has told it otherwise.

Elsewhere

Deployments outside Kubernetes use an external etcd for discovery, where a worker record is simply present or absent and nothing observes that a process is leaving. Shutdown behaves as in the section above — deregister, then drain — with in-flight work finished either way. The early notice is the part that needs Kubernetes.

In both cases the worker is absent from /v1/workers while it drains rather than shown as draining, since removing the record is what stops new work arriving. A Pod being deleted does show as draining, but earlier — between its deletion being requested and the process being signalled, before the drain itself begins.