The reflex
Something’s wrong. A GitLab runner stops picking up jobs. An event processor starts dropping messages. A pod restarts in a loop. The node looks healthy — CPU fine, memory fine — but something is clearly off.
The reflex: restart the node, see if it clears.
Sometimes it does clear, and you move on. But you didn’t fix anything. You reset the state and crossed your fingers. If it happens again in two weeks, you’ll do the same thing. After enough iterations you have a “flaky node” that everyone reboots periodically and nobody understands.
There’s a better sequence. It takes twenty minutes instead of two, and you come out with either a real fix or actual knowledge of what happened.
Step one: quarantine, don’t kill
Before you touch anything, take the node out of rotation without destroying its current state.
kubectl cordon <node>
Cordon marks the node as unschedulable. No new pods land on it. Existing pods keep running. If you need the workloads somewhere else immediately:
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
Now you’ve removed the node from production traffic without rebooting. The node is still alive. Everything that happened on it is still there: logs, open files, kernel ring buffer, running processes, memory state.
This is the difference. A reboot wipes that. A cordon preserves it.
Step two: look at what’s actually there
SSH in. Don’t grep for anything specific yet — do a pass for anything unusual.
Kernel messages first. The kernel will often tell you exactly what went wrong before any application did.
dmesg -T --level=err,warn | tail -50
OOM kills show up here. Disk errors show up here. CPU soft lockups show up here. If you’ve got any of those, you have your answer before you’ve even looked at application logs.
Check for filesystem problems.
df -h # is anything full?
dmesg | grep -i "ext4\|xfs\|btrfs\|i/o error\|ata"
A filesystem at 100% is silent until it isn’t. A flaky drive starts dropping I/O errors into dmesg long before SMART reports anything. Application developers rarely think about this case — their app just starts writing logs that say “failed to write” without specifying that the disk is full or dying.
System resource pressure.
vmstat 1 5 # is there swap activity?
iostat -x 1 5 # is a disk saturated?
cat /proc/pressure/io # kernel PSI — pressure stall info
PSI is underused. It tells you whether processes were actually stalled waiting for I/O, not just whether throughput was high. A disk at 80% utilisation might be fine; a disk with 40% I/O PSI pressure is actively hurting performance.
What were the pods doing right before things went sideways?
kubectl describe node <node> # events section at the bottom
kubectl get events --field-selector involvedObject.kind=Pod -A | sort -k1
Look for OOMKilled exits, failed liveness probes, and throttling events. Kubernetes events expire after an hour by default — another reason not to reboot immediately; those events are still there if you look now.
A real example: the GitLab runner
A GitLab runner pod stops picking up jobs. It looks alive — the process is running, no crashes in the pod logs. Jobs sit in the queue.
Restart reflex: delete the pod, let it reschedule, it picks up jobs again.
But why did it stop?
journalctl -u gitlab-runner --since "1 hour ago"
# or, if it's a container:
kubectl logs <runner-pod> --previous
In one instance: the runner’s working directory was on a tmpfs that hit its size limit. The runner silently failed to create job workspaces and stopped accepting new jobs. The error was one line in the pod logs: mkdir /builds: no space left on device. The pod was healthy by every other metric.
Fix: bump the tmpfs size limit in the runner config. The restart would have cleared tmpfs temporarily, and the runner would have failed again the next time a large job filled it up.
The debug took five minutes. The permanent fix took two minutes. Without quarantining the node first, the evidence was gone.
Another one: the event consumer
An event processor starts falling behind. Messages queue up. The pod shows no errors. Memory looks fine.
This one was subtler: the processor was connected to a downstream dependency over a persistent TCP connection. The connection had gone into a half-open state — the processor thought it was alive, the remote end had already dropped it. New messages were being sent into a dead socket and silently discarded.
ss -tnp | grep <pid> # look at the socket state
CLOSE_WAIT on a connection that should be ESTABLISHED. The application wasn’t checking whether the connection was actually working before using it, just whether it existed.
Restart would have cleared the socket state, fixed the symptom, and left the bug in the code.
What to look for — a short checklist
When a node is misbehaving, in order:
dmesg -T --level=err,warn— kernel errors, OOM kills, disk errorsdf -h && df -i— full filesystems (space and inodes separately)kubectl describe node <node>— pressure conditions, recent eventskubectl logs <pod> --previous— what the pod logged before it died or got stuckss -tnp— socket states for network-adjacent issuesvmstat 1 5+iostat -x 1 5— resource pressurejournalctl -p err -b— system journal errors since last boot
Most problems show up in the first three.
After you’ve found something (or not found something)
If you found the cause: fix it, test it, uncordon the node.
kubectl uncordon <node>
Document what you found — a comment in the relevant config, a commit message, a note. “Fixed runner tmpfs limit” in the commit history is more useful than “flaky runner, restarted.”
If you genuinely found nothing: that’s information too. Cordon, reboot, uncordon, and note that the node rebooted clean with no identified cause. If it happens again, you have a pattern. Check whether anything changed in the workloads around that time. Check whether the reboot timing correlates with anything — cron jobs, backups, maintenance windows.
A reboot you can explain is a fix. A reboot you can’t explain is a time bomb.
Why this matters on a single-node cluster
In a multi-node setup you can afford to be lazier — cordon, drain, reboot, let the scheduler handle it, look at it later. On a single node there’s no “later.” The node coming back is all you’ve got.
But the habit is worth building regardless of node count. The engineers who understand their systems are the ones who looked before they rebooted.
The actual rule
Quarantine first. Debug second. Restart third (if you still need to).
A restart takes two minutes. The evidence it destroys might take two hours to reconstruct — or might be gone for good. The cordon costs you nothing.