<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Helm on hippotion</title><link>https://blog.hippotion.com/tags/helm/</link><description>Recent content in Helm on hippotion</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Fri, 10 Oct 2025 00:00:00 +0000</lastBuildDate><atom:link href="https://blog.hippotion.com/tags/helm/index.xml" rel="self" type="application/rss+xml"/><item><title>📦 Five Ways to Manage Kubernetes Manifests (and Why They're Not All Equal)</title><link>https://blog.hippotion.com/posts/gitops-manifest-approaches/</link><pubDate>Fri, 10 Oct 2025 00:00:00 +0000</pubDate><guid>https://blog.hippotion.com/posts/gitops-manifest-approaches/</guid><description>Raw YAML, Kustomize, Helm, Jsonnet — there&amp;rsquo;s more than one way to describe what you want running in a cluster. Here&amp;rsquo;s what each actually looks like in practice and where each one breaks.</description><content:encoded><![CDATA[<h2 id="the-problem-everyone-hits">The problem everyone hits</h2>
<p>You&rsquo;ve got a Kubernetes cluster. Now you need to describe what should run in it. You write some YAML, apply it, it works.</p>
<p>Then you need a second environment. Or a second service. Or someone else joins the project and asks &ldquo;how do I add an app to this?&rdquo; and you don&rsquo;t have a good answer.</p>
<p>This is the manifest management problem, and there are five common solutions — ranging from &ldquo;this works until it doesn&rsquo;t&rdquo; to &ldquo;this is what production platforms actually look like.&rdquo;</p>
<hr>
<h2 id="approach-1-raw-manifests">Approach 1: Raw manifests</h2>
<p>The starting point for almost everyone. Write a YAML file, <code>kubectl apply -f</code>, done.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">apps/v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Deployment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">replicas</span><span class="p">:</span><span class="w"> </span><span class="m">1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">selector</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">matchLabels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">template</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">labels</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">containers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">myapp:v1.2.3</span><span class="w">
</span></span></span></code></pre></div><p><strong>Where it works:</strong> one service, one environment, learning Kubernetes. The feedback loop is immediate — write YAML, see what happens.</p>
<p><strong>Where it breaks:</strong></p>
<ul>
<li><strong>No templating.</strong> Want to change the image tag across ten services? Ten files, ten edits, ten chances to get it wrong.</li>
<li><strong>Live state leaks in.</strong> If you export existing resources with <code>kubectl get -o yaml</code>, you get <code>resourceVersion</code>, <code>generation</code>, <code>creationTimestamp</code>, and <code>managedFields</code> in the output. Commit that to Git and you&rsquo;ve created a permanent source of conflicts — ArgoCD compares what&rsquo;s in Git against what&rsquo;s in the cluster, sees stale version counters, and the diff never clears.</li>
<li><strong>Copy-paste hell.</strong> A Deployment, a Service, an IngressRoute, a ServiceAccount, a NetworkPolicy — five files per app. Add a new app, copy five files, change the names, forget to update one. This is how environments drift apart silently.</li>
</ul>
<p>The fix for the live-state problem is: only commit desired state. Strip every field that Kubernetes manages internally back to its clean spec. It&rsquo;s tedious and easy to forget, which is exactly why people move on from raw manifests.</p>
<hr>
<h2 id="approach-2-kustomize">Approach 2: Kustomize</h2>
<p>Kustomize is built into <code>kubectl</code> (<code>kubectl apply -k</code>) and natively supported by ArgoCD. The idea: you have a <code>base/</code> with your raw manifests, and overlays that patch on top of them for different environments.</p>
<pre tabindex="0"><code>app/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── staging/
    │   ├── kustomization.yaml    # patches replicas to 1, image to :staging
    └── production/
        └── kustomization.yaml    # patches replicas to 3, image to :v1.2.3
</code></pre><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># overlays/production/kustomization.yaml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">resources</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="l">../../base</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">patches</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">patch</span><span class="p">:</span><span class="w"> </span><span class="p">|-</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">      - op: replace
</span></span></span><span class="line"><span class="cl"><span class="sd">        path: /spec/replicas
</span></span></span><span class="line"><span class="cl"><span class="sd">        value: 3</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">target</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Deployment</span><span class="w">
</span></span></span></code></pre></div><p><strong>Where it works:</strong> multi-environment setups where the difference between environments is mostly configuration values, not structure. Kustomize is good at this — you write the base once and patch only what differs.</p>
<p><strong>Where it breaks:</strong></p>
<ul>
<li><strong>No real parameterization.</strong> Kustomize patches are surgical edits, not templates. If your base structure needs to vary (different resource shapes per environment, conditional blocks), you&rsquo;re fighting the tool.</li>
<li><strong>Patching deep structures is ugly.</strong> JSON patches on nested YAML are verbose and hard to read. You end up writing more patch YAML than it would take to just copy the file.</li>
<li><strong>Still repetitive across apps.</strong> Each app still gets its own base directory. You&rsquo;re not abstracting the shared patterns across apps, only the differences between environments of the same app.</li>
</ul>
<p>Kustomize is a significant step up from raw manifests for multi-environment setups. For complex templating or platform-level abstractions, it runs out of power quickly.</p>
<hr>
<h2 id="approach-3-helm">Approach 3: Helm</h2>
<p>Helm adds real templating. Charts are parameterized bundles — templates with variables, conditionals, and loops — and values files supply the parameters.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># templates/deployment.yaml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">apps/v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Deployment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span>{{<span class="w"> </span><span class="l">.Values.name }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span>{{<span class="w"> </span><span class="l">.Release.Namespace }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">replicas</span><span class="p">:</span><span class="w"> </span>{{<span class="w"> </span><span class="l">.Values.replicas | default 1 }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">template</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">containers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span>{{<span class="w"> </span><span class="l">.Values.name }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span>{{<span class="w"> </span><span class="l">.Values.image.repository }}:{{ .Values.image.tag }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span>{{- <span class="l">if .Values.resources }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">resources</span><span class="p">:</span><span class="w"> </span>{{<span class="w"> </span><span class="l">.Values.resources | toYaml | nindent 12 }}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span>{{- <span class="l">end }}</span><span class="w">
</span></span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># values-production.yaml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">replicas</span><span class="p">:</span><span class="w"> </span><span class="m">3</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">image</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">repository</span><span class="p">:</span><span class="w"> </span><span class="l">myorg/myapp</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">tag</span><span class="p">:</span><span class="w"> </span><span class="l">v1.2.3</span><span class="w">
</span></span></span></code></pre></div><p>Helm renders the templates at deploy time. What lands in the cluster is clean rendered YAML — no internal state, no conflicts.</p>
<p><strong>Where it works:</strong> almost everywhere. The Helm Hub has charts for most common software already. For custom apps, writing a chart once and parameterizing per-environment is straightforwardly better than copying YAML.</p>
<p><strong>Where it breaks:</strong></p>
<ul>
<li><strong>Chart authoring is verbose.</strong> Writing a Helm chart from scratch involves a lot of Go templating boilerplate. For a simple app, it can feel like more scaffolding than application.</li>
<li><strong>Debugging rendered output is annoying.</strong> <code>helm template</code> is your friend, but errors in templates produce unhelpful messages. The indentation rules (<code>nindent</code>, <code>indent</code>, <code>toYaml</code>) have sharp edges.</li>
<li><strong>Values files still pile up.</strong> If every app has its own values file and there&rsquo;s no shared structure between them, you&rsquo;re back to copy-paste but now in YAML-that-configures-YAML.</li>
</ul>
<p>Helm is the right tool for most Kubernetes deployments. The ecosystem support alone (upstream charts for Postgres, Redis, Vault, every CNCF project) makes it the pragmatic default.</p>
<hr>
<h2 id="approach-4-jsonnet--cue">Approach 4: Jsonnet / CUE</h2>
<p>For teams that need programmatic config generation — actual code, not templates — Jsonnet and CUE are the serious alternatives.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-jsonnet" data-lang="jsonnet"><span class="line"><span class="cl"><span class="c1">// deployment.jsonnet
</span></span></span><span class="line"><span class="cl"><span class="k">local</span><span class="w"> </span><span class="nv">k</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">import</span><span class="w"> </span><span class="s">&#34;k.libsonnet&#34;</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">local</span><span class="w"> </span><span class="nf">deployment</span><span class="p">(</span><span class="nv">name</span><span class="p">,</span><span class="w"> </span><span class="nv">image</span><span class="p">,</span><span class="w"> </span><span class="nv">replicas</span><span class="o">=</span><span class="mf">1</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nv">k</span><span class="p">.</span><span class="nv">apps</span><span class="p">.</span><span class="nv">v1</span><span class="p">.</span><span class="nv">deployment</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="nv">name</span><span class="p">,</span><span class="w"> </span><span class="nv">replicas</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nv">k</span><span class="p">.</span><span class="nv">core</span><span class="p">.</span><span class="nv">v1</span><span class="p">.</span><span class="nv">container</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="nv">name</span><span class="p">,</span><span class="w"> </span><span class="nv">image</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="p">]);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nv">&#34;deployment.yaml&#34;</span><span class="p">:</span><span class="w"> </span><span class="nf">deployment</span><span class="p">(</span><span class="s">&#34;myapp&#34;</span><span class="p">,</span><span class="w"> </span><span class="s">&#34;myorg/myapp:v1.2.3&#34;</span><span class="p">,</span><span class="w"> </span><span class="nv">replicas</span><span class="o">=</span><span class="mf">3</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><strong>Where it works:</strong> large platforms where configuration is genuinely complex — many environments, many apps, deep interdependencies. Jsonnet lets you write real functions, share libraries, compose abstractions properly.</p>
<p><strong>Where it breaks:</strong></p>
<ul>
<li><strong>Steep learning curve.</strong> Jsonnet is a full language. CUE even more so — it has types, schemas, and a constraint system that takes time to internalise.</li>
<li><strong>Small community.</strong> Excellent tooling, but you&rsquo;re solving problems that have fewer Stack Overflow answers.</li>
<li><strong>Overkill for most setups.</strong> If you&rsquo;re not managing hundreds of services across multiple clusters, Helm is simpler and has everything you need.</li>
</ul>
<p>Jsonnet is used seriously at Google-scale infrastructure teams and in some CNCF projects. For a homelab or a small-to-medium platform, it&rsquo;s the right answer to a question you probably aren&rsquo;t asking yet.</p>
<hr>
<h2 id="approach-5-app-of-apps-with-generated-application-crds">Approach 5: App-of-apps with generated Application CRDs</h2>
<p>This is the ArgoCD-native meta-layer. Instead of managing manifests, you manage <code>Application</code> resources — and potentially use a chart or tool to generate those too.</p>
<p>A naive version: commit a folder of <code>Application</code> YAML files to Git, one per service. ArgoCD watches the folder and deploys each app.</p>
<p>A more sophisticated version: one &ldquo;root app&rdquo; that points to a chart, which generates all the other <code>Application</code> resources dynamically from a single config file.</p>
<p><strong>Where it works:</strong> at the platform level, not the individual app level. App-of-apps is how you manage what ArgoCD manages, not how you write the service manifests themselves. Combined with Helm, it gives you centralized control over the entire cluster&rsquo;s structure.</p>
<p><strong>Where it breaks:</strong></p>
<ul>
<li><strong>Manual <code>Application</code> CRDs are painful.</strong> If you&rsquo;re maintaining a folder of hand-written <code>Application</code> YAML files — one per service — you&rsquo;ve traded manifest copy-paste for Application copy-paste. Each app needs its own CRD with its repo URL, path, sync policy, project reference.</li>
<li><strong>Sync ordering matters.</strong> The root app must exist before children can sync. Get the wave ordering wrong and apps try to deploy before their namespaces exist.</li>
</ul>
<hr>
<h2 id="how-this-homelab-compares">How this homelab compares</h2>
<p>My setup sits at the far end of approach 5, using Helm throughout.</p>
<p>There&rsquo;s a single <code>applications.yml</code> file that describes every service in the cluster. A root Helm chart reads it and generates all the ArgoCD <code>Application</code> and <code>AppProject</code> CRDs automatically. Adding a service means adding an entry to that file — not touching five different places across five different files.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># applications.yml — this is the entire service catalog</span><span class="w">
</span></span></span><span class="line"><span class="cl">- <span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">web-vaultwarden</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">networkPolicies</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">profile</span><span class="p">:</span><span class="w"> </span><span class="l">web-app</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">applications</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">applicationCode</span><span class="p">:</span><span class="w"> </span><span class="l">web-vaultwarden</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">helm-charts/extra-objects</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">autoSync</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span></code></pre></div><p>That one entry generates: a Namespace, an ArgoCD AppProject, an ArgoCD Application, a set of Cilium NetworkPolicies (deny-all with ingress from Traefik and DNS/HTTPS egress), and a ServiceAccount. Nothing is written by hand.</p>
<p>The actual service manifests live in an <code>extra-objects</code> chart — a thin wrapper that renders raw YAML from values files. No templating in the service manifests themselves (they&rsquo;re simple enough not to need it), but the infrastructure scaffolding around each app is entirely generated.</p>
<p>The result: every service gets the same operational properties. Same GitOps workflow, same secret management, same network isolation, same TLS termination. The platform work was done once. Adding a new app is writing manifests for the app&rsquo;s specific behavior, not recreating the scaffolding.</p>
<hr>
<h2 id="the-honest-spectrum">The honest spectrum</h2>
<table>
	<thead>
			<tr>
					<th>Approach</th>
					<th>Templating</th>
					<th>Abstraction</th>
					<th>Ecosystem</th>
					<th>Complexity</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Raw manifests</td>
					<td>None</td>
					<td>None</td>
					<td>None</td>
					<td>Low</td>
			</tr>
			<tr>
					<td>Kustomize</td>
					<td>Patches only</td>
					<td>Overlays</td>
					<td>Medium</td>
					<td>Low-medium</td>
			</tr>
			<tr>
					<td>Helm</td>
					<td>Full</td>
					<td>Per-chart</td>
					<td>Large</td>
					<td>Medium</td>
			</tr>
			<tr>
					<td>Jsonnet/CUE</td>
					<td>Full + typed</td>
					<td>Libraries</td>
					<td>Small</td>
					<td>High</td>
			</tr>
			<tr>
					<td>App-of-apps</td>
					<td>Depends</td>
					<td>Platform-level</td>
					<td>ArgoCD-native</td>
					<td>High</td>
			</tr>
	</tbody>
</table>
<p>Most setups should start at Helm. Kustomize if you&rsquo;re multi-environment and comfortable with patching. App-of-apps when you&rsquo;re managing the platform layer, not individual services. Jsonnet/CUE when you know you&rsquo;ve outgrown Helm — which is a specific and relatively rare problem to have.</p>
<p>Raw manifests are fine for learning. They&rsquo;re the wrong answer for anything you intend to maintain.</p>
<hr>
<p><em>More on how the homelab is structured: <a href="/posts/homelab-gitops/">My Homelab Runs on GitOps</a>.</em></p>
]]></content:encoded></item></channel></rss>