Kubernetes for People Who Have to Run It

Kubernetes tutorials teach you to deploy a pod. Nobody teaches you what to do when a node silently drops out of Ready state at 3am and your on-call phone won't stop buzzing. This is about the second part.

Resource requests and limits aren't optional

If you skip resources.requests and resources.limits on your containers, the scheduler is guessing, and the kubelet has no idea what to evict first when a node runs low on memory. Set requests to what the container actually uses under normal load, not what you hope it uses. Set memory limits close to requests since OOM-killed pods restart cleanly, but CPU limits should generally be looser or absent unless you're fighting noisy neighbors, because CPU throttling under a hard limit causes latency spikes that are miserable to debug. Run kubectl top pods -n your-namespace regularly and compare against what you've declared. Mismatches pile up fast in clusters with more than a handful of services.

etcd is the whole cluster, treat it that way

Everything Kubernetes knows lives in etcd. Lose it and you lose your cluster state, not just some metadata. Run etcdctl snapshot save /backup/etcd-snapshot.db on a schedule, and actually test restoring it on a scratch cluster at least once. I've seen teams with nightly backups that turned out to be corrupt for three months because nobody verified them. If you're on a managed control plane (EKS, GKE, AKS), you don't manage etcd directly, but you should still know your provider's backup and restore story before you need it, not during an incident.

Liveness and readiness probes need to disagree sometimes

A common mistake is using the same endpoint for both liveness and readiness probes. Liveness should answer