<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Secrets on hippotion</title><link>https://blog.hippotion.com/tags/secrets/</link><description>Recent content in Secrets on hippotion</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Fri, 25 Apr 2025 00:00:00 +0000</lastBuildDate><atom:link href="https://blog.hippotion.com/tags/secrets/index.xml" rel="self" type="application/rss+xml"/><item><title>🤫 How Do You Handle Secrets in a GitOps Repository?</title><link>https://blog.hippotion.com/posts/k8s-gitops-secrets/</link><pubDate>Fri, 25 Apr 2025 00:00:00 +0000</pubDate><guid>https://blog.hippotion.com/posts/k8s-gitops-secrets/</guid><description>GitOps says Git is the source of truth. Secrets say don&amp;rsquo;t put them in Git. These two things appear to be in direct conflict. They&amp;rsquo;re not.</description><content:encoded><![CDATA[<h2 id="the-question">The question</h2>
<p><em>&ldquo;You&rsquo;re using GitOps — everything goes through Git. How do you handle secrets?&rdquo;</em></p>
<p>The wrong answer: base64-encode them and commit them as Kubernetes <code>Secret</code> objects. Base64 is not encryption. Anyone with read access to the repo has your secrets. If the repo is public, everyone does.</p>
<p>The slightly better wrong answer: use a private repo and just not think about it. This works until a deploy key leaks, someone joins and then leaves the company, or you need to rotate one secret and have to find every place it&rsquo;s referenced.</p>
<p>There are three real answers. They make different tradeoffs.</p>
<hr>
<h2 id="the-constraint">The constraint</h2>
<p>The constraint is actually tighter than &ldquo;don&rsquo;t commit secrets&rdquo;. It&rsquo;s: <strong>your Git repo should be safe to make public at any point</strong>, and <strong>secrets must be rotatable without touching Git</strong>.</p>
<p>If rotating a password requires a new commit, someone has to be awake to merge and deploy it. That&rsquo;s not how you want to handle a 3am incident.</p>
<hr>
<h2 id="option-1-external-secrets-operator--vault">Option 1: External Secrets Operator + Vault</h2>
<p>This is the most robust pattern and the one worth knowing for interviews.</p>
<p>The idea: secrets live in a dedicated secret store (HashiCorp Vault, or a cloud equivalent). A Kubernetes operator called ESO watches <code>ExternalSecret</code> CRD objects in the cluster and syncs the referenced secret into a real Kubernetes <code>Secret</code>. The CRD is safe to commit — it says where the secret lives, not what it is.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># This lives in Git — safe to commit</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">external-secrets.io/v1beta1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">ExternalSecret</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">myapp-db-credentials</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">refreshInterval</span><span class="p">:</span><span class="w"> </span><span class="l">1h</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">secretStoreRef</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">vault</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">ClusterSecretStore</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">target</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">myapp-db-credentials  </span><span class="w"> </span><span class="c"># the k8s Secret it creates</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">data</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">secretKey</span><span class="p">:</span><span class="w"> </span><span class="l">DB_PASSWORD</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">remoteRef</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">key</span><span class="p">:</span><span class="w"> </span><span class="l">secret/myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">property</span><span class="p">:</span><span class="w"> </span><span class="l">db-password</span><span class="w">
</span></span></span></code></pre></div><p>Rotation: you update the secret in Vault. ESO syncs it to the cluster within <code>refreshInterval</code>. No Git commit, no deployment. The pod reads the updated <code>Secret</code> on the next restart (or immediately if you mount it as an env var and the app handles <code>SIGHUP</code>).</p>
<p>Audit trail: Vault logs every read and write. You know exactly which service account read which secret at what time.</p>
<p>The cost: you&rsquo;re running Vault. For a homelab or small team, that&rsquo;s an extra thing to operate. For production, it&rsquo;s worth it.</p>
<p>Self-hosted setup:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># ClusterSecretStore — connects ESO to your Vault instance</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">external-secrets.io/v1beta1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">ClusterSecretStore</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">vault</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">provider</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">vault</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">server</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;http://sys-vault.sys-vault.svc.cluster.local:8200&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;secret&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;v2&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">auth</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">kubernetes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;kubernetes&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">role</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;external-secrets&#34;</span><span class="w">
</span></span></span></code></pre></div><p>ESO authenticates to Vault using the pod&rsquo;s Kubernetes ServiceAccount token. Vault validates it against the cluster&rsquo;s token review endpoint. No static credentials anywhere.</p>
<hr>
<h2 id="option-2-sealed-secrets">Option 2: Sealed Secrets</h2>
<p>Sealed Secrets uses asymmetric encryption. The cluster holds a private key. You use the <code>kubeseal</code> CLI to encrypt a secret with the cluster&rsquo;s public key. The resulting <code>SealedSecret</code> object is safe to commit — only the cluster can decrypt it.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Encrypt a secret for committing to Git</span>
</span></span><span class="line"><span class="cl">kubectl create secret generic myapp-db <span class="se">\
</span></span></span><span class="line"><span class="cl">  --from-literal<span class="o">=</span><span class="nv">DB_PASSWORD</span><span class="o">=</span>hunter2 <span class="se">\
</span></span></span><span class="line"><span class="cl">  --dry-run<span class="o">=</span>client -o yaml <span class="se">\
</span></span></span><span class="line"><span class="cl">  <span class="p">|</span> kubeseal <span class="se">\
</span></span></span><span class="line"><span class="cl">  &gt; sealed-secrets/myapp-db.yaml
</span></span></code></pre></div><p>The resulting YAML looks like:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">bitnami.com/v1alpha1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">SealedSecret</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">myapp-db</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">encryptedData</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">DB_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l">AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq...</span><span class="w">
</span></span></span></code></pre></div><p>This gets committed. The Sealed Secrets controller in the cluster decrypts it and creates the real <code>Secret</code> automatically.</p>
<p>The tradeoff: rotation means re-sealing. You need the cluster&rsquo;s public key (which is public) and access to the plaintext secret. You commit a new <code>SealedSecret</code>. That&rsquo;s a Git commit, which means a review, a merge, and a deploy. For a 3am incident, that&rsquo;s a lot of friction.</p>
<p>Also: if the cluster&rsquo;s private key is lost, you can&rsquo;t decrypt any of your sealed secrets. Back up the private key.</p>
<p>Good fit for: small teams, homelab, situations where secrets change rarely and the GitOps review process is actually desirable.</p>
<hr>
<h2 id="option-3-sops">Option 3: SOPS</h2>
<p>SOPS (Secrets OPerationS) encrypts files at rest using age keys or cloud KMS. You commit encrypted files. CI decrypts them during deployment using a key it holds in memory (not stored in Git).</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Encrypt a file for Git</span>
</span></span><span class="line"><span class="cl">sops --encrypt --age age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8q <span class="se">\
</span></span></span><span class="line"><span class="cl">  secrets/myapp.yaml &gt; secrets/myapp.enc.yaml
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># In CI: decrypt to temp file, apply, delete</span>
</span></span><span class="line"><span class="cl">sops --decrypt secrets/myapp.enc.yaml <span class="p">|</span> kubectl apply -f -
</span></span></code></pre></div><p>The difference from Sealed Secrets: SOPS encrypts at the file level, not the k8s object level. You can use it outside of Kubernetes (application configs, Terraform variables). The key can live in the CI environment, a cloud KMS, or a personal age key.</p>
<p>The tradeoff: CI needs the decryption key, which puts you back in &ldquo;secret in CI&rdquo; territory — just for the encryption key rather than the actual secrets. If you use a cloud KMS, OIDC federation handles that (no stored key). If you use an age key, it lives in CI secrets.</p>
<p>Good fit for: teams already using Helm and Helm Secrets, polyglot environments where not everything is Kubernetes, small teams where Vault feels like overengineering.</p>
<hr>
<h2 id="comparison">Comparison</h2>
<table>
	<thead>
			<tr>
					<th></th>
					<th>ESO + Vault</th>
					<th>Sealed Secrets</th>
					<th>SOPS</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Rotation without Git commit</td>
					<td>Yes</td>
					<td>No</td>
					<td>Depends</td>
			</tr>
			<tr>
					<td>Audit trail</td>
					<td>Full (Vault)</td>
					<td>None</td>
					<td>Depends on KMS</td>
			</tr>
			<tr>
					<td>Complexity</td>
					<td>High</td>
					<td>Low</td>
					<td>Medium</td>
			</tr>
			<tr>
					<td>Works outside k8s</td>
					<td>With effort</td>
					<td>No</td>
					<td>Yes</td>
			</tr>
			<tr>
					<td>Recovery if key lost</td>
					<td>Vault backup</td>
					<td>Lose all secrets</td>
					<td>Key backup</td>
			</tr>
			<tr>
					<td>CI needs secret material</td>
					<td>No</td>
					<td>No</td>
					<td>Yes (decrypt key)</td>
			</tr>
	</tbody>
</table>
<hr>
<h2 id="what-interviewers-are-actually-testing">What interviewers are actually testing</h2>
<p>The interesting follow-up question is: <em>&ldquo;How do you rotate a secret without downtime?&rdquo;</em></p>
<p>The answer requires you to understand that pods mount <code>Secret</code> objects at startup. Updating the <code>Secret</code> in Kubernetes doesn&rsquo;t automatically restart the pod. Your options are:</p>
<ol>
<li>Mount the secret as a volume and have the app watch for file changes (good)</li>
<li>Restart the deployment after rotation (<code>kubectl rollout restart</code>, automatable)</li>
<li>Use a sidecar like Vault Agent Injector that handles refresh in-process (complex but zero-restart)</li>
</ol>
<p>The correct answer depends on the app. An API key that can be rotated gradually is different from a database password where the old one is invalidated immediately.</p>
<hr>
<p><em>This is part of a series on Kubernetes interview questions. Previously: <a href="/posts/k8s-cicd-no-credentials/">deploying without cluster credentials</a>. Next: <a href="/posts/k8s-zero-downtime/">zero-downtime deployments</a>.</em></p>
]]></content:encoded></item></channel></rss>