<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Deployments on hippotion</title><link>https://blog.hippotion.com/tags/deployments/</link><description>Recent content in Deployments on hippotion</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Fri, 20 Jun 2025 00:00:00 +0000</lastBuildDate><atom:link href="https://blog.hippotion.com/tags/deployments/index.xml" rel="self" type="application/rss+xml"/><item><title>⚡ Your Deployment Causes 30 Seconds of Downtime. What Went Wrong?</title><link>https://blog.hippotion.com/posts/k8s-zero-downtime/</link><pubDate>Fri, 20 Jun 2025 00:00:00 +0000</pubDate><guid>https://blog.hippotion.com/posts/k8s-zero-downtime/</guid><description>Kubernetes rolling updates don&amp;rsquo;t give you zero-downtime for free. There are four separate things you have to get right, and most clusters get at least one wrong.</description><content:encoded><![CDATA[<h2 id="the-question">The question</h2>
<p><em>&ldquo;How do you achieve zero-downtime deployments in Kubernetes?&rdquo;</em></p>
<p>The expected answer: rolling updates. That&rsquo;s correct but incomplete. Rolling updates are the mechanism. They don&rsquo;t give you zero downtime automatically — they give you a framework in which zero downtime is achievable, if you configure everything correctly.</p>
<p>Most clusters cause brief downtime on every deployment. Usually 5–30 seconds. Usually blamed on &ldquo;the load balancer&rdquo; or &ldquo;DNS&rdquo;. Almost always caused by one of four missing pieces.</p>
<hr>
<h2 id="the-rolling-update-baseline">The rolling update baseline</h2>
<p>Kubernetes replaces pods in waves. You control the pace:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">strategy</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">RollingUpdate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rollingUpdate</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">maxSurge</span><span class="p">:</span><span class="w"> </span><span class="m">1</span><span class="w">        </span><span class="c"># how many extra pods can exist during update</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">maxUnavailable</span><span class="p">:</span><span class="w"> </span><span class="m">0</span><span class="w">  </span><span class="c"># how many pods can be unavailable during update</span><span class="w">
</span></span></span></code></pre></div><p><code>maxUnavailable: 0</code> means Kubernetes never terminates a pod until a replacement is ready. This prevents the obvious failure mode where you have zero running pods mid-deployment.</p>
<p><code>maxSurge: 1</code> means one extra pod beyond the desired count runs during the update. For a deployment with 3 replicas, you&rsquo;ll briefly have 4 pods running.</p>
<p>This alone doesn&rsquo;t prevent downtime.</p>
<hr>
<h2 id="piece-1-the-readiness-probe-the-most-common-missing-piece">Piece 1: The readiness probe (the most common missing piece)</h2>
<p>Kubernetes considers a pod &ldquo;ready&rdquo; when all its containers pass their readiness probes. If you don&rsquo;t define a readiness probe, Kubernetes considers the pod ready as soon as the container starts. Containers start before applications are ready to serve traffic.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># Without this, traffic arrives before your app is listening</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">readinessProbe</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">httpGet</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/healthz</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">8080</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">initialDelaySeconds</span><span class="p">:</span><span class="w"> </span><span class="m">5</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">periodSeconds</span><span class="p">:</span><span class="w"> </span><span class="m">5</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">failureThreshold</span><span class="p">:</span><span class="w"> </span><span class="m">3</span><span class="w">
</span></span></span></code></pre></div><p>What happens without it: Kubernetes starts the new pod, marks it ready immediately, adds it to the Service endpoints, routes traffic to it — while your app is still initialising (loading config, connecting to the database, warming caches). The first few requests to the new pod fail or time out.</p>
<p>The fix: define a readiness probe that actually checks application readiness. An HTTP endpoint that returns 200 only after the app has finished starting is the minimum. A deeper check that verifies the database connection is better.</p>
<p>Common mistake: using the same endpoint for liveness and readiness with the same thresholds. They serve different purposes:</p>
<ul>
<li><strong>Readiness</strong>: &ldquo;am I ready to accept traffic?&rdquo; — controls whether traffic is sent</li>
<li><strong>Liveness</strong>: &ldquo;am I still alive?&rdquo; — controls whether the pod is restarted</li>
</ul>
<p>A pod can fail its readiness probe (temporarily overloaded, warming up) without failing its liveness probe. If you make liveness too aggressive, Kubernetes restarts pods that would have recovered on their own.</p>
<hr>
<h2 id="piece-2-the-termination-grace-period-the-other-common-missing-piece">Piece 2: The termination grace period (the other common missing piece)</h2>
<p>When Kubernetes wants to terminate a pod, it sends <code>SIGTERM</code>. Your application has <code>terminationGracePeriodSeconds</code> (default: 30) to finish in-flight requests and shut down cleanly. After that, Kubernetes sends <code>SIGKILL</code>.</p>
<p>The problem: there&rsquo;s a race condition. Kubernetes removes the pod from the Service endpoints and sends <code>SIGTERM</code> roughly simultaneously. The endpoint update has to propagate through the control plane, kube-proxy, and the load balancer. During that propagation window — typically 1–10 seconds — traffic can still arrive at a pod that has already started shutting down.</p>
<p>The fix is a <code>preStop</code> hook that adds a short sleep before the termination sequence:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">lifecycle</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">preStop</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">exec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;sleep&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;5&#34;</span><span class="p">]</span><span class="w">
</span></span></span></code></pre></div><p>This gives the endpoint removal time to propagate before your app receives <code>SIGTERM</code>. The total shutdown sequence is then:</p>
<ol>
<li>Kubernetes removes pod from endpoints</li>
<li><code>preStop</code> hook runs (sleep 5s — enough for endpoint propagation)</li>
<li><code>SIGTERM</code> is sent</li>
<li>App drains in-flight requests and shuts down</li>
<li>If still running after <code>terminationGracePeriodSeconds</code>: <code>SIGKILL</code></li>
</ol>
<p>Set <code>terminationGracePeriodSeconds</code> to cover the sleep plus your app&rsquo;s actual shutdown time:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">terminationGracePeriodSeconds</span><span class="p">:</span><span class="w"> </span><span class="m">60</span><span class="w">  </span><span class="c"># 5s preStop + up to 55s for app shutdown</span><span class="w">
</span></span></span></code></pre></div><p>Without the sleep: requests fail during the propagation window. With it: the window is covered.</p>
<hr>
<h2 id="piece-3-poddisruptionbudgets-for-node-maintenance">Piece 3: PodDisruptionBudgets (for node maintenance)</h2>
<p>Rolling updates handle normal deployments. Node drains (<code>kubectl drain</code>, cloud provider maintenance windows, k3s upgrades) are a different code path that bypasses your rolling update strategy entirely.</p>
<p>When a node is drained, Kubernetes evicts all pods on it as fast as it can. Without constraints, it will evict all replicas of your deployment simultaneously if they all happen to land on the same node.</p>
<p>A <code>PodDisruptionBudget</code> sets a floor:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">policy/v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">PodDisruptionBudget</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">myapp-pdb</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">minAvailable</span><span class="p">:</span><span class="w"> </span><span class="m">1</span><span class="w">   </span><span class="c"># at least 1 replica must stay up during disruption</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">selector</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">matchLabels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span></code></pre></div><p>Now node drain will evict pods one at a time, waiting for replacement pods to come up before evicting the next one. If no replacement can be scheduled (e.g., you&rsquo;re draining the only node), the drain will block rather than cause downtime.</p>
<p><code>minAvailable: 1</code> is the minimum. For production with 3+ replicas, <code>minAvailable: 2</code> or <code>maxUnavailable: 1</code> is more appropriate.</p>
<hr>
<h2 id="piece-4-minreadyseconds-the-one-everyone-forgets">Piece 4: minReadySeconds (the one everyone forgets)</h2>
<p>Even with a correct readiness probe, there&rsquo;s a subtle risk: a pod that passes its readiness probe briefly and then fails due to a transient startup issue (flapping). Kubernetes would add it to the endpoint pool, route traffic to it, watch it fail the readiness probe, remove it — and during that window, some requests fail.</p>
<p><code>minReadySeconds</code> says: a pod must pass its readiness probe continuously for this many seconds before Kubernetes considers it &ldquo;available&rdquo; and allows the next pod in the rolling update to be terminated:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">minReadySeconds</span><span class="p">:</span><span class="w"> </span><span class="m">10</span><span class="w">
</span></span></span></code></pre></div><p>This slows deployments slightly but catches flapping probes before they cause production traffic to hit an unstable pod.</p>
<hr>
<h2 id="the-complete-deployment-snippet">The complete deployment snippet</h2>
<p>Putting it together:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">apps/v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Deployment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">replicas</span><span class="p">:</span><span class="w"> </span><span class="m">3</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">minReadySeconds</span><span class="p">:</span><span class="w"> </span><span class="m">10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">strategy</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">RollingUpdate</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">rollingUpdate</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">maxSurge</span><span class="p">:</span><span class="w"> </span><span class="m">1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">maxUnavailable</span><span class="p">:</span><span class="w"> </span><span class="m">0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">selector</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">matchLabels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">template</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">terminationGracePeriodSeconds</span><span class="p">:</span><span class="w"> </span><span class="m">60</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">containers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">myapp:latest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">lifecycle</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">preStop</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="nt">exec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;sleep&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;5&#34;</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">readinessProbe</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">httpGet</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/healthz</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">8080</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">initialDelaySeconds</span><span class="p">:</span><span class="w"> </span><span class="m">5</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">periodSeconds</span><span class="p">:</span><span class="w"> </span><span class="m">5</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">failureThreshold</span><span class="p">:</span><span class="w"> </span><span class="m">3</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">livenessProbe</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">httpGet</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/healthz</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="m">8080</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">initialDelaySeconds</span><span class="p">:</span><span class="w"> </span><span class="m">15</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">periodSeconds</span><span class="p">:</span><span class="w"> </span><span class="m">10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">failureThreshold</span><span class="p">:</span><span class="w"> </span><span class="m">5</span><span class="w">
</span></span></span></code></pre></div><p>And the PDB alongside it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">policy/v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">PodDisruptionBudget</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">myapp-pdb</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">minAvailable</span><span class="p">:</span><span class="w"> </span><span class="m">2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">selector</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">matchLabels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span></code></pre></div><hr>
<h2 id="what-interviewers-are-actually-testing">What interviewers are actually testing</h2>
<p>The follow-up is usually: <em>&ldquo;What if your new version has a bug that isn&rsquo;t caught immediately — how do you roll back?&rdquo;</em></p>
<p><code>kubectl rollout undo deployment/myapp</code> reverts to the previous ReplicaSet. Kubernetes stores the last few ReplicaSets by default (<code>revisionHistoryLimit</code>, default 10). The rollback uses the same rolling update mechanism, so it&rsquo;s also zero-downtime.</p>
<p>The harder follow-up: <em>&ldquo;What if the bug only shows up after 10 minutes of load?&rdquo;</em> That&rsquo;s where you need a canary deployment — send a small percentage of traffic to the new version, observe, then shift the rest. Argo Rollouts handles this natively. Without it, you&rsquo;re doing it manually with two Deployments and weighted Services.</p>
<hr>
<p><em>This is part of a series on Kubernetes interview questions. Previously: <a href="/posts/k8s-gitops-secrets/">secrets in a GitOps repo</a>. Next: <a href="/posts/k8s-network-isolation/">network isolation between services</a>.</em></p>
]]></content:encoded></item></channel></rss>