“How do you achieve zero-downtime deployments in Kubernetes?”

The expected answer: rolling updates. That’s correct but incomplete. Rolling updates are the mechanism. They don’t give you zero downtime automatically — they give you a framework in which zero downtime is achievable, if you configure everything correctly.

Most clusters cause brief downtime on every deployment. Usually 5–30 seconds. Usually blamed on “the load balancer” or “DNS”. Almost always caused by one of four missing pieces.

The rolling update baseline

Kubernetes replaces pods in waves. You control the pace:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # how many extra pods can exist during update
      maxUnavailable: 0  # how many pods can be unavailable during update

maxUnavailable: 0 means Kubernetes never terminates a pod until a replacement is ready. This prevents the obvious failure mode where you have zero running pods mid-deployment.

maxSurge: 1 means one extra pod beyond the desired count runs during the update. For a deployment with 3 replicas, you’ll briefly have 4 pods running.

This alone doesn’t prevent downtime.

Piece 1: The readiness probe (the most common missing piece)

Kubernetes considers a pod “ready” when all its containers pass their readiness probes. If you don’t define a readiness probe, Kubernetes considers the pod ready as soon as the container starts. Containers start before applications are ready to serve traffic.

# Without this, traffic arrives before your app is listening
readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3

What happens without it: Kubernetes starts the new pod, marks it ready immediately, adds it to the Service endpoints, routes traffic to it — while your app is still initialising (loading config, connecting to the database, warming caches). The first few requests to the new pod fail or time out.

The fix: define a readiness probe that actually checks application readiness. An HTTP endpoint that returns 200 only after the app has finished starting is the minimum. A deeper check that verifies the database connection is better.

Common mistake: using the same endpoint for liveness and readiness with the same thresholds. They serve different purposes:

• Readiness: “am I ready to accept traffic?” — controls whether traffic is sent
• Liveness: “am I still alive?” — controls whether the pod is restarted

A pod can fail its readiness probe (temporarily overloaded, warming up) without failing its liveness probe. If you make liveness too aggressive, Kubernetes restarts pods that would have recovered on their own.

Piece 2: The termination grace period (the other common missing piece)

When Kubernetes wants to terminate a pod, it sends SIGTERM. Your application has terminationGracePeriodSeconds (default: 30) to finish in-flight requests and shut down cleanly. After that, Kubernetes sends SIGKILL.

The problem: there’s a race condition. Kubernetes removes the pod from the Service endpoints and sends SIGTERM roughly simultaneously. The endpoint update has to propagate through the control plane, kube-proxy, and the load balancer. During that propagation window — typically 1–10 seconds — traffic can still arrive at a pod that has already started shutting down.

The fix is a preStop hook that adds a short sleep before the termination sequence:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]

This gives the endpoint removal time to propagate before your app receives SIGTERM. The total shutdown sequence is then:

1. Kubernetes removes pod from endpoints
2. preStop hook runs (sleep 5s — enough for endpoint propagation)
3. SIGTERM is sent
4. App drains in-flight requests and shuts down
5. If still running after terminationGracePeriodSeconds: SIGKILL

Set terminationGracePeriodSeconds to cover the sleep plus your app’s actual shutdown time:

spec:
  terminationGracePeriodSeconds: 60  # 5s preStop + up to 55s for app shutdown

Without the sleep: requests fail during the propagation window. With it: the window is covered.

Piece 3: PodDisruptionBudgets (for node maintenance)

Rolling updates handle normal deployments. Node drains (kubectl drain, cloud provider maintenance windows, k3s upgrades) are a different code path that bypasses your rolling update strategy entirely.

When a node is drained, Kubernetes evicts all pods on it as fast as it can. Without constraints, it will evict all replicas of your deployment simultaneously if they all happen to land on the same node.

A PodDisruptionBudget sets a floor:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
  namespace: myapp
spec:
  minAvailable: 1   # at least 1 replica must stay up during disruption
  selector:
    matchLabels:
      app: myapp

Now node drain will evict pods one at a time, waiting for replacement pods to come up before evicting the next one. If no replacement can be scheduled (e.g., you’re draining the only node), the drain will block rather than cause downtime.

minAvailable: 1 is the minimum. For production with 3+ replicas, minAvailable: 2 or maxUnavailable: 1 is more appropriate.

Piece 4: minReadySeconds (the one everyone forgets)

Even with a correct readiness probe, there’s a subtle risk: a pod that passes its readiness probe briefly and then fails due to a transient startup issue (flapping). Kubernetes would add it to the endpoint pool, route traffic to it, watch it fail the readiness probe, remove it — and during that window, some requests fail.

minReadySeconds says: a pod must pass its readiness probe continuously for this many seconds before Kubernetes considers it “available” and allows the next pod in the rolling update to be terminated:

spec:
  minReadySeconds: 10

This slows deployments slightly but catches flapping probes before they cause production traffic to hit an unstable pod.

The complete deployment snippet

Putting it together:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 3
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: myapp
          image: myapp:latest
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "5"]
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10
            failureThreshold: 5

And the PDB alongside it:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
  namespace: myapp
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: myapp

What interviewers are actually testing

The follow-up is usually: “What if your new version has a bug that isn’t caught immediately — how do you roll back?”

kubectl rollout undo deployment/myapp reverts to the previous ReplicaSet. Kubernetes stores the last few ReplicaSets by default (revisionHistoryLimit, default 10). The rollback uses the same rolling update mechanism, so it’s also zero-downtime.

The harder follow-up: “What if the bug only shows up after 10 minutes of load?” That’s where you need a canary deployment — send a small percentage of traffic to the new version, observe, then shift the rest. Argo Rollouts handles this natively. Without it, you’re doing it manually with two Deployments and weighted Services.

This is part of a series on Kubernetes interview questions. Previously: secrets in a GitOps repo. Next: network isolation between services.

“How do you prevent configuration drift in a Kubernetes cluster?”

Configuration drift: the cluster’s actual state diverges from what’s declared in your source of truth. Someone runs kubectl edit deployment myapp to bump a memory limit during an incident. Someone adds a debug sidecar directly. Someone applies a YAML file from their laptop that was never committed to Git. The fix works. It goes undocumented. Six months later, a new deployment overwrites it. The incident recurs.

There are two distinct problems here that require different solutions:

1. Detection and remediation: how do you notice drift and revert it?
2. Prevention: how do you stop non-compliant resources from being created in the first place?

Detection and remediation: Argo CD selfHeal

If you’re using GitOps with Argo CD, detection and remediation are handled for you:

syncPolicy:
  automated:
    prune: true
    selfHeal: true

selfHeal: true means Argo CD continuously compares the cluster state to the Git repo and reverts any divergence. Someone runs kubectl edit deployment myapp and changes the replica count? Argo CD detects the diff on its next reconciliation cycle (default: every 3 minutes) and reverts it.

prune: true means resources that exist in the cluster but not in Git are deleted. Someone kubectl apply’d a debug pod directly? Gone on the next sync.

This is the audit trail story too. Every legitimate change is a Git commit with an author, a timestamp, and a commit message. Everything that isn’t in Git doesn’t survive past the next reconciliation. If you want to know what changed and when, git log is the answer.

The gap selfHeal doesn’t close

selfHeal reverts drift after the fact. There’s a window — up to 3 minutes — where a drifted resource is serving traffic. For most changes, that’s fine. For a bad resource (wrong RBAC, missing network policy, container running as root), 3 minutes is enough to be a problem.

The other gap: selfHeal doesn’t tell you who made the change or generate an alert. It just silently fixes it. You need audit logging (kube-apiserver --audit-log-path) or an alerting rule on Argo CD’s health events to know that drift happened.

Prevention: Kyverno

Kyverno is a policy engine that runs as a Kubernetes admission webhook. Every resource creation or modification goes through it before being persisted. If the resource violates a policy, Kyverno can reject it outright (enforce mode) or allow it with a warning (audit mode).

The policies are Kubernetes resources themselves — they live in Git, they’re applied via GitOps, they’re versioned. No separate policy language to learn.

A policy that requires readiness probes on all Deployments:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-readiness-probe
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-readiness-probe
      match:
        any:
          - resources:
              kinds:
                - Deployment
      validate:
        message: "Deployments must define a readiness probe."
        pattern:
          spec:
            template:
              spec:
                containers:
                  - (name): "*"
                    readinessProbe:
                      (httpGet | tcpSocket | exec): "*"

With this policy active: kubectl apply -f deployment-without-probe.yaml is rejected at the API server. The error message is the one you defined in message. The deployment never reaches etcd.

A policy that blocks containers running as root:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-root-containers
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-runAsNonRoot
      match:
        any:
          - resources:
              kinds: [Deployment, StatefulSet, DaemonSet]
      validate:
        message: "Containers must not run as root."
        pattern:
          spec:
            template:
              spec:
                containers:
                  - (name): "*"
                    securityContext:
                      runAsNonRoot: true

A policy that enforces resource limits (common in multi-tenant clusters):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-limits
      match:
        any:
          - resources:
              kinds: [Deployment]
      validate:
        message: "CPU and memory limits are required."
        pattern:
          spec:
            template:
              spec:
                containers:
                  - resources:
                      limits:
                        memory: "?*"
                        cpu: "?*"

Kyverno can also mutate and generate

Policies aren’t only for validation. Kyverno can mutate incoming resources (add default labels, inject sidecars, set default resource requests) and generate new resources in response to events (create a NetworkPolicy whenever a new namespace is created).

Auto-add a standard label to every Deployment:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-labels
spec:
  rules:
    - name: add-team-label
      match:
        any:
          - resources:
              kinds: [Deployment]
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              managed-by: kyverno

Auto-create a default NetworkPolicy when a namespace is created:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-networkpolicy
spec:
  rules:
    - name: default-deny
      match:
        any:
          - resources:
              kinds: [Namespace]
      generate:
        kind: NetworkPolicy
        name: default-deny-all
        namespace: "{{request.object.metadata.name}}"
        data:
          spec:
            podSelector: {}
            policyTypes:
              - Ingress
              - Egress

The complete drift prevention picture

Developer runs: kubectl apply -f bad-deployment.yaml
  → API server receives request
  → Kyverno admission webhook intercepts
  → Policy check: no readiness probe → Rejected
  → API server returns 403 with Kyverno's message
  → Resource never reaches etcd

Developer runs: kubectl edit deployment myapp (valid change, just not via Git)
  → Edit succeeds (no policy violation)
  → Argo CD reconciliation fires (within 3 minutes)
  → Diff detected: cluster state ≠ Git state
  → selfHeal: revert to Git state
  → If audit logging enabled: event recorded with username and timestamp

Git is the audit trail for what should be there. kube-apiserver audit logs are the trail for what was attempted. Kyverno is the enforcer at admission time. Argo CD is the continuous reconciler. Four layers, each with a different job.

What interviewers are actually testing

The follow-up is usually: “What’s the difference between Kyverno and OPA Gatekeeper?”

Both are admission webhook policy engines. The practical differences:

• Kyverno: policies are k8s-native YAML, no separate language to learn. Generate and mutate policies built in. Easier to get started with.
• OPA Gatekeeper: policies are written in Rego, a purpose-built policy language that’s more expressive but has a steeper learning curve. Better if you’re already using OPA elsewhere (Terraform, microservice authorization).

For a Kubernetes-only environment, Kyverno is the pragmatic choice. For a platform team that uses OPA across the stack, Gatekeeper gives you policy consistency.

The deeper follow-up: “How do you test policies before enforcing them?” Use Audit mode first (validationFailureAction: Audit). Violations are logged as PolicyReport objects but requests aren’t rejected. Review the reports, fix the existing violations, then switch to Enforce. Never flip directly to Enforce in production — you’ll break things that were already running.

This is part of a series on Kubernetes interview questions. Previously: network isolation between services.

“How do you enforce network isolation between services in a Kubernetes cluster?”

The default Kubernetes network model is flat. Every pod can reach every other pod, in any namespace, on any port. There are no firewalls, no ACLs, no segmentation. A compromised frontend pod can connect directly to your PostgreSQL port, your Redis port, your internal admin API, and every other service in the cluster.

This is intentional — Kubernetes doesn’t assume you want isolation, because not everyone does. But if you do want it, you need to add it.

NetworkPolicy: the primitive

A NetworkPolicy is a Kubernetes resource that selects a set of pods and defines what traffic is allowed to reach them (ingress) and what traffic they’re allowed to send (egress). Traffic that isn’t explicitly allowed is dropped.

The catch: NetworkPolicy resources have no effect unless your CNI plugin supports them. The default k3s CNI (Flannel) does not. Calico, Cilium, and Canal do. If you’re running Flannel and you apply a NetworkPolicy, it will be silently ignored — no error, no warning.

The default-deny pattern

The correct starting point is a default-deny policy that blocks everything, applied to the namespace. You then add explicit allow policies for the traffic you actually need.

# Block all ingress and egress in this namespace by default
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: myapp
spec:
  podSelector: {}        # matches all pods in the namespace
  policyTypes:
    - Ingress
    - Egress

With this in place, your pods can’t receive traffic and can’t send traffic. You then add back what you need.

Allowing specific traffic

Allow the web frontend to receive traffic from the ingress controller:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-from-traefik
  namespace: myapp
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: sys-traefik

Allow the backend to talk to PostgreSQL:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-to-postgres
  namespace: myapp
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - port: 5432
          protocol: TCP

After these two policies: the frontend receives traffic from Traefik, and the backend can reach Postgres. The frontend cannot reach Postgres. The backend cannot receive traffic from the ingress controller. Neither can call anything else.

The DNS gotcha

Once you add a default-deny egress policy, DNS stops working. Your pods can no longer resolve service names because they can’t reach kube-dns in the kube-system namespace.

You need to explicitly allow it:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-dns
  namespace: myapp
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP

Missing this is the most common reason “everything broke after I added NetworkPolicies”. Add it to every namespace that has a default-deny policy.

Cilium: the same model with more power

Cilium implements the standard NetworkPolicy API and adds its own CiliumNetworkPolicy CRD with L7 capabilities.

Standard NetworkPolicy works at L3/L4 — IP addresses and ports. Cilium’s CRD adds:

L7 HTTP filtering: allow specific HTTP methods and paths, not just port 8080.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-api-reads
  namespace: myapp
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/v1/.*"

DNS-based egress: allow egress to github.com by hostname rather than IP address. This matters for external services with dynamic IPs.

egress:
  - toFQDNs:
      - matchName: "github.com"
    toPorts:
      - ports:
          - port: "443"
            protocol: TCP

Identity-based policies: Cilium assigns a cryptographic identity to each pod based on its labels. Policies are enforced by identity, not IP address. Pod restarts (which change IPs) don’t break policy enforcement.

What a real namespace policy set looks like

For a typical web app with frontend, backend, and database:

Namespace: myapp
├── default-deny-all (ingress + egress, all pods)
├── allow-egress-dns (egress, all pods, port 53)
├── allow-ingress-frontend (ingress frontend, from sys-traefik namespace)
├── allow-egress-frontend-to-backend (egress frontend, to backend:8080)
├── allow-ingress-backend (ingress backend, from frontend)
├── allow-egress-backend-to-postgres (egress backend, to postgres:5432)
└── allow-ingress-postgres (ingress postgres, from backend)

Eight policies. The database has exactly one inbound path: from the backend. The frontend has no path to the database at all. A compromised frontend pod cannot scan the internal network — egress to arbitrary destinations is blocked.

What interviewers are actually testing

The follow-up is usually: “How do you manage this at scale? Writing NetworkPolicies for every namespace by hand doesn’t scale.”

The answer: you don’t write them by hand. You template them. In a GitOps setup, your namespace configuration declares what network access the service needs in a structured form, and a Helm chart or operator generates the actual NetworkPolicy resources from those declarations.

For example, an applications.yml entry might look like:

networkPolicies:
  denyAll: true
  allowIngressFromIngress: true
  allowEgressToNamespaces: ["sys-postgres"]

And a Helm chart translates that into four concrete NetworkPolicy objects. The developer declares intent; the platform enforces it. No one writes raw YAML for each namespace.

The second follow-up: “What about east-west traffic between services in the same namespace?” Add allowIntraNamespace: true as a flag that generates a policy allowing all pod-to-pod traffic within the namespace, while still blocking cross-namespace traffic.

This is part of a series on Kubernetes interview questions. Previously: zero-downtime deployments. Next: preventing configuration drift.

“How would you design a CI/CD pipeline that deploys to Kubernetes without storing any cluster credentials anywhere?”

The expected wrong answer: export your kubeconfig, base64-encode it, paste it into a CI secret named KUBE_CONFIG, and call it a day. This works. Most clusters that got hacked had this setup.

There are two correct answers in 2026, and which one you reach for depends on what you’re actually deploying.

Answer 1: GitOps (the one your interviewer probably wants)

In a GitOps setup, your CI pipeline never touches the cluster. It can’t leak credentials it doesn’t have.

The flow:

Developer pushes code
  → CI builds and tests
  → CI updates the image tag in the Git repo (a commit, not a kubectl command)
  → Argo CD detects the change
  → Argo CD applies it to the cluster

The cluster reaches out to Git. CI never reaches into the cluster. The only thing with cluster credentials is Argo CD itself — running inside the cluster, with no credentials to leak externally.

For self-hosted setups on Hetzner or Vultr, this is particularly clean because there’s no cloud IAM to configure. You point Argo CD at your GitLab repo, tell it which branch to watch, and you’re done.

# The Argo CD Application CRD — the only thing you need
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  source:
    repoURL: https://gitlab.example.com/myorg/myapp
    targetRevision: main
    path: helm-charts/myapp
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

selfHeal: true means if someone manually kubectl applys something, Argo CD reverts it. The Git repo is the only source of truth.

The CI image-tag update step looks like this:

# .gitlab-ci.yml
deploy:
  stage: deploy
  script:
    - |
      # Update the image tag in values.yaml and push
      sed -i "s/tag: .*/tag: ${CI_COMMIT_SHORT_SHA}/" values/myapp.yml
      git config user.email "ci@example.com"
      git config user.name "CI"
      git add values/myapp.yml
      git commit -m "chore: bump myapp to ${CI_COMMIT_SHORT_SHA}"
      git push

CI needs write access to the Git repo — but that’s a deploy key, not a cluster credential. If it leaks, someone can push code. You’d rotate the deploy key and audit the commits. If a cluster credential leaks, someone owns your cluster.

Answer 2: OIDC federation (for when you genuinely need push-based)

Some operations don’t fit the GitOps model. Infrastructure provisioning (terraform apply), one-off database migrations, or initial cluster bootstrapping — these need direct cluster access. The correct pattern here is OIDC federation.

The idea: your CI platform (GitLab, GitHub Actions) already issues JWT tokens to every job. These JWTs are signed by the CI platform and contain claims like which repo, which branch, which pipeline triggered the job. You configure your Kubernetes API server to trust those JWTs, and the CI job authenticates directly using the token it already has.

No stored credentials. Every job gets a fresh token. The token expires when the job ends.

For a self-hosted GitLab, configure your k8s API server to trust GitLab as an OIDC issuer:

# /etc/rancher/k3s/config.yaml (or kube-apiserver flags)
kube-apiserver-arg:
  - "oidc-issuer-url=https://gitlab.example.com"
  - "oidc-client-id=your_client_id"
  - "oidc-username-claim=sub"
  - "oidc-groups-claim=groups_direct"

Then create a ClusterRoleBinding that maps a specific GitLab identity to a Kubernetes role:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gitlab-ci-deployer
subjects:
  - kind: User
    name: "project_path:myorg/myapp:ref_type:branch:ref:main"
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: deploy-role
  apiGroup: rbac.authorization.k8s.io

The subject name is the sub claim from the GitLab JWT — it encodes the repo path and branch. Only jobs running on main in myorg/myapp get this binding. A job on a feature branch gets nothing.

In the CI job:

deploy:
  stage: deploy
  id_tokens:
    K8S_TOKEN:
      aud: your_client_id
  script:
    - |
      kubectl config set-credentials gitlab-ci \
        --token="${K8S_TOKEN}"
      kubectl config set-context deploy \
        --cluster=mycluster \
        --user=gitlab-ci
      kubectl config use-context deploy
      kubectl rollout restart deployment/myapp -n myapp

The token in K8S_TOKEN is injected by GitLab. It expires with the job. The API server validates the signature against GitLab’s JWKS endpoint on every request.

Which one to use

For application deployments: GitOps. The cluster reconciles continuously, drift is impossible, and CI is completely decoupled from cluster state.

For infrastructure provisioning or one-off operations: OIDC federation. Short-lived credentials, branch-scoped permissions, nothing to rotate.

What you should never do: store a kubeconfig or a long-lived ServiceAccount token in CI secrets. Not because it’s hard to make work — it’s easy — but because the blast radius of a leak is unbounded, there’s no audit trail, and there’s no expiry. Everything that goes wrong with static secrets goes wrong eventually.

This is part of a series on Kubernetes interview questions. Next: how to handle secrets in a GitOps repository.

“You’re using GitOps — everything goes through Git. How do you handle secrets?”

The wrong answer: base64-encode them and commit them as Kubernetes Secret objects. Base64 is not encryption. Anyone with read access to the repo has your secrets. If the repo is public, everyone does.

The slightly better wrong answer: use a private repo and just not think about it. This works until a deploy key leaks, someone joins and then leaves the company, or you need to rotate one secret and have to find every place it’s referenced.

There are three real answers. They make different tradeoffs.

The constraint

The constraint is actually tighter than “don’t commit secrets”. It’s: your Git repo should be safe to make public at any point, and secrets must be rotatable without touching Git.

If rotating a password requires a new commit, someone has to be awake to merge and deploy it. That’s not how you want to handle a 3am incident.

Option 1: External Secrets Operator + Vault

This is the most robust pattern and the one worth knowing for interviews.

The idea: secrets live in a dedicated secret store (HashiCorp Vault, or a cloud equivalent). A Kubernetes operator called ESO watches ExternalSecret CRD objects in the cluster and syncs the referenced secret into a real Kubernetes Secret. The CRD is safe to commit — it says where the secret lives, not what it is.

# This lives in Git — safe to commit
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-db-credentials
  namespace: myapp
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: myapp-db-credentials   # the k8s Secret it creates
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: secret/myapp
        property: db-password

Rotation: you update the secret in Vault. ESO syncs it to the cluster within refreshInterval. No Git commit, no deployment. The pod reads the updated Secret on the next restart (or immediately if you mount it as an env var and the app handles SIGHUP).

Audit trail: Vault logs every read and write. You know exactly which service account read which secret at what time.

The cost: you’re running Vault. For a homelab or small team, that’s an extra thing to operate. For production, it’s worth it.

Self-hosted setup:

# ClusterSecretStore — connects ESO to your Vault instance
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: "http://sys-vault.sys-vault.svc.cluster.local:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"

ESO authenticates to Vault using the pod’s Kubernetes ServiceAccount token. Vault validates it against the cluster’s token review endpoint. No static credentials anywhere.

Option 2: Sealed Secrets

Sealed Secrets uses asymmetric encryption. The cluster holds a private key. You use the kubeseal CLI to encrypt a secret with the cluster’s public key. The resulting SealedSecret object is safe to commit — only the cluster can decrypt it.

# Encrypt a secret for committing to Git
kubectl create secret generic myapp-db \
  --from-literal=DB_PASSWORD=hunter2 \
  --dry-run=client -o yaml \
  | kubeseal \
  > sealed-secrets/myapp-db.yaml

The resulting YAML looks like:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: myapp-db
  namespace: myapp
spec:
  encryptedData:
    DB_PASSWORD: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq...

This gets committed. The Sealed Secrets controller in the cluster decrypts it and creates the real Secret automatically.

The tradeoff: rotation means re-sealing. You need the cluster’s public key (which is public) and access to the plaintext secret. You commit a new SealedSecret. That’s a Git commit, which means a review, a merge, and a deploy. For a 3am incident, that’s a lot of friction.

Also: if the cluster’s private key is lost, you can’t decrypt any of your sealed secrets. Back up the private key.

Good fit for: small teams, homelab, situations where secrets change rarely and the GitOps review process is actually desirable.

Option 3: SOPS

SOPS (Secrets OPerationS) encrypts files at rest using age keys or cloud KMS. You commit encrypted files. CI decrypts them during deployment using a key it holds in memory (not stored in Git).

# Encrypt a file for Git
sops --encrypt --age age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8q \
  secrets/myapp.yaml > secrets/myapp.enc.yaml

# In CI: decrypt to temp file, apply, delete
sops --decrypt secrets/myapp.enc.yaml | kubectl apply -f -

The difference from Sealed Secrets: SOPS encrypts at the file level, not the k8s object level. You can use it outside of Kubernetes (application configs, Terraform variables). The key can live in the CI environment, a cloud KMS, or a personal age key.

The tradeoff: CI needs the decryption key, which puts you back in “secret in CI” territory — just for the encryption key rather than the actual secrets. If you use a cloud KMS, OIDC federation handles that (no stored key). If you use an age key, it lives in CI secrets.

Good fit for: teams already using Helm and Helm Secrets, polyglot environments where not everything is Kubernetes, small teams where Vault feels like overengineering.

Comparison

What interviewers are actually testing

The interesting follow-up question is: “How do you rotate a secret without downtime?”

The answer requires you to understand that pods mount Secret objects at startup. Updating the Secret in Kubernetes doesn’t automatically restart the pod. Your options are:

1. Mount the secret as a volume and have the app watch for file changes (good)
2. Restart the deployment after rotation (kubectl rollout restart, automatable)
3. Use a sidecar like Vault Agent Injector that handles refresh in-process (complex but zero-restart)

The correct answer depends on the app. An API key that can be rotated gradually is different from a database password where the old one is invalidated immediately.

This is part of a series on Kubernetes interview questions. Previously: deploying without cluster credentials. Next: zero-downtime deployments.