<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Cicd on hippotion</title><link>https://blog.hippotion.com/tags/cicd/</link><description>Recent content in Cicd on hippotion</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Fri, 09 May 2025 00:00:00 +0000</lastBuildDate><atom:link href="https://blog.hippotion.com/tags/cicd/index.xml" rel="self" type="application/rss+xml"/><item><title>🔑 Deploy to Kubernetes Without Storing Any Cluster Credentials in CI</title><link>https://blog.hippotion.com/posts/k8s-cicd-no-credentials/</link><pubDate>Fri, 09 May 2025 00:00:00 +0000</pubDate><guid>https://blog.hippotion.com/posts/k8s-cicd-no-credentials/</guid><description>A common interview question in 2026. If your answer is &amp;lsquo;kubeconfig in a CI secret&amp;rsquo;, you&amp;rsquo;re not wrong — but you&amp;rsquo;re also not getting the job.</description><content:encoded><![CDATA[<h2 id="the-question">The question</h2>
<p><em>&ldquo;How would you design a CI/CD pipeline that deploys to Kubernetes without storing any cluster credentials anywhere?&rdquo;</em></p>
<p>The expected wrong answer: export your kubeconfig, base64-encode it, paste it into a CI secret named <code>KUBE_CONFIG</code>, and call it a day. This works. Most clusters that got hacked had this setup.</p>
<p>There are two correct answers in 2026, and which one you reach for depends on what you&rsquo;re actually deploying.</p>
<hr>
<h2 id="answer-1-gitops-the-one-your-interviewer-probably-wants">Answer 1: GitOps (the one your interviewer probably wants)</h2>
<p>In a GitOps setup, your CI pipeline never touches the cluster. It can&rsquo;t leak credentials it doesn&rsquo;t have.</p>
<p>The flow:</p>
<pre tabindex="0"><code>Developer pushes code
  → CI builds and tests
  → CI updates the image tag in the Git repo (a commit, not a kubectl command)
  → Argo CD detects the change
  → Argo CD applies it to the cluster
</code></pre><p>The cluster reaches out to Git. CI never reaches into the cluster. The only thing with cluster credentials is Argo CD itself — running inside the cluster, with no credentials to leak externally.</p>
<p>For self-hosted setups on Hetzner or Vultr, this is particularly clean because there&rsquo;s no cloud IAM to configure. You point Argo CD at your GitLab repo, tell it which branch to watch, and you&rsquo;re done.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># The Argo CD Application CRD — the only thing you need</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">argoproj.io/v1alpha1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Application</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">argocd</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">source</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">repoURL</span><span class="p">:</span><span class="w"> </span><span class="l">https://gitlab.example.com/myorg/myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">targetRevision</span><span class="p">:</span><span class="w"> </span><span class="l">main</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">helm-charts/myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">destination</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">server</span><span class="p">:</span><span class="w"> </span><span class="l">https://kubernetes.default.svc</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">syncPolicy</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">automated</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">prune</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">selfHeal</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span></code></pre></div><p><code>selfHeal: true</code> means if someone manually <code>kubectl apply</code>s something, Argo CD reverts it. The Git repo is the only source of truth.</p>
<p>The CI image-tag update step looks like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># .gitlab-ci.yml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">deploy</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l">deploy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">script</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">      # Update the image tag in values.yaml and push
</span></span></span><span class="line"><span class="cl"><span class="sd">      sed -i &#34;s/tag: .*/tag: ${CI_COMMIT_SHORT_SHA}/&#34; values/myapp.yml
</span></span></span><span class="line"><span class="cl"><span class="sd">      git config user.email &#34;ci@example.com&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">      git config user.name &#34;CI&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">      git add values/myapp.yml
</span></span></span><span class="line"><span class="cl"><span class="sd">      git commit -m &#34;chore: bump myapp to ${CI_COMMIT_SHORT_SHA}&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">      git push</span><span class="w">
</span></span></span></code></pre></div><p>CI needs write access to the Git repo — but that&rsquo;s a deploy key, not a cluster credential. If it leaks, someone can push code. You&rsquo;d rotate the deploy key and audit the commits. If a cluster credential leaks, someone owns your cluster.</p>
<hr>
<h2 id="answer-2-oidc-federation-for-when-you-genuinely-need-push-based">Answer 2: OIDC federation (for when you genuinely need push-based)</h2>
<p>Some operations don&rsquo;t fit the GitOps model. Infrastructure provisioning (<code>terraform apply</code>), one-off database migrations, or initial cluster bootstrapping — these need direct cluster access. The correct pattern here is OIDC federation.</p>
<p>The idea: your CI platform (GitLab, GitHub Actions) already issues JWT tokens to every job. These JWTs are signed by the CI platform and contain claims like which repo, which branch, which pipeline triggered the job. You configure your Kubernetes API server to trust those JWTs, and the CI job authenticates directly using the token it already has.</p>
<p>No stored credentials. Every job gets a fresh token. The token expires when the job ends.</p>
<p>For a self-hosted GitLab, configure your k8s API server to trust GitLab as an OIDC issuer:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># /etc/rancher/k3s/config.yaml (or kube-apiserver flags)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kube-apiserver-arg</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="s2">&#34;oidc-issuer-url=https://gitlab.example.com&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="s2">&#34;oidc-client-id=your_client_id&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="s2">&#34;oidc-username-claim=sub&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="s2">&#34;oidc-groups-claim=groups_direct&#34;</span><span class="w">
</span></span></span></code></pre></div><p>Then create a <code>ClusterRoleBinding</code> that maps a specific GitLab identity to a Kubernetes role:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">rbac.authorization.k8s.io/v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">ClusterRoleBinding</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">gitlab-ci-deployer</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">subjects</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">User</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;project_path:myorg/myapp:ref_type:branch:ref:main&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">apiGroup</span><span class="p">:</span><span class="w"> </span><span class="l">rbac.authorization.k8s.io</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">roleRef</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">ClusterRole</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">deploy-role</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">apiGroup</span><span class="p">:</span><span class="w"> </span><span class="l">rbac.authorization.k8s.io</span><span class="w">
</span></span></span></code></pre></div><p>The subject name is the <code>sub</code> claim from the GitLab JWT — it encodes the repo path and branch. Only jobs running on <code>main</code> in <code>myorg/myapp</code> get this binding. A job on a feature branch gets nothing.</p>
<p>In the CI job:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">deploy</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l">deploy</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">id_tokens</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">K8S_TOKEN</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">aud</span><span class="p">:</span><span class="w"> </span><span class="l">your_client_id</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">script</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">      kubectl config set-credentials gitlab-ci \
</span></span></span><span class="line"><span class="cl"><span class="sd">        --token=&#34;${K8S_TOKEN}&#34;
</span></span></span><span class="line"><span class="cl"><span class="sd">      kubectl config set-context deploy \
</span></span></span><span class="line"><span class="cl"><span class="sd">        --cluster=mycluster \
</span></span></span><span class="line"><span class="cl"><span class="sd">        --user=gitlab-ci
</span></span></span><span class="line"><span class="cl"><span class="sd">      kubectl config use-context deploy
</span></span></span><span class="line"><span class="cl"><span class="sd">      kubectl rollout restart deployment/myapp -n myapp</span><span class="w">
</span></span></span></code></pre></div><p>The token in <code>K8S_TOKEN</code> is injected by GitLab. It expires with the job. The API server validates the signature against GitLab&rsquo;s JWKS endpoint on every request.</p>
<hr>
<h2 id="which-one-to-use">Which one to use</h2>
<table>
	<thead>
			<tr>
					<th></th>
					<th>GitOps</th>
					<th>OIDC federation</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>CI needs cluster access</td>
					<td>No</td>
					<td>Yes (short-lived token)</td>
			</tr>
			<tr>
					<td>Audit trail</td>
					<td>Git history</td>
					<td>kube-apiserver audit log</td>
			</tr>
			<tr>
					<td>Revocability</td>
					<td>Revert the commit</td>
					<td>Token expires with the job</td>
			</tr>
			<tr>
					<td>Self-hosted setup effort</td>
					<td>Low</td>
					<td>Moderate (OIDC config)</td>
			</tr>
			<tr>
					<td>Works for infra provisioning</td>
					<td>Not really</td>
					<td>Yes</td>
			</tr>
	</tbody>
</table>
<p>For application deployments: GitOps. The cluster reconciles continuously, drift is impossible, and CI is completely decoupled from cluster state.</p>
<p>For infrastructure provisioning or one-off operations: OIDC federation. Short-lived credentials, branch-scoped permissions, nothing to rotate.</p>
<p>What you should never do: store a kubeconfig or a long-lived ServiceAccount token in CI secrets. Not because it&rsquo;s hard to make work — it&rsquo;s easy — but because the blast radius of a leak is unbounded, there&rsquo;s no audit trail, and there&rsquo;s no expiry. Everything that goes wrong with static secrets goes wrong eventually.</p>
<hr>
<p><em>This is part of a series on Kubernetes interview questions. Next: <a href="/posts/k8s-gitops-secrets/">how to handle secrets in a GitOps repository</a>.</em></p>
]]></content:encoded></item></channel></rss>