My house runs on quiet little robots. A tracker watches my kombucha ferment. A job narrates kids’ books in Hungarian. A media stack pulls and files things. Home Assistant minds the sensors. A dozen services, all doing their jobs, all completely mute. When a batch finished or an import failed, I found out the same way every time: by going to look.

Then the silence got expensive. Claude Code stopped dead in the middle of a task because I’d burned through my plan’s usage window — no warning, no countdown, just a wall. The information existed; a dashboard in my own cluster was already polling it. It just had no way to reach my pocket.

So I built one thing: a push bus. One place anything in the cluster can POST to, that actually buzzes my phone. And the first job I gave it was to warn me before my AI assistant goes dark.

The boring part (said honestly)

The bus is ntfy — a self-hosted pub/sub notifier. Picking it took about five minutes, because self-hosting ntfy for a homelab is a thoroughly solved problem. There are at least three off-the-shelf bridges from Prometheus Alertmanager to ntfy. I’m not going to pretend the bus is the clever bit.

What I did do deliberately:

• 📦 Deployed it GitOps-native — one entry in my app-of-apps, reconciled by Argo CD, no docker run anywhere.
• 🔒 Locked it to deny-all auth with bearer tokens. Security alerts ride this bus; a world-readable topic on a public URL was a non-starter. (Which also means it sits outside my usual OAuth gate — the phone app can’t do an interactive login flow, so ntfy does its own token auth.)
• 🏷️ Topics by severity: hl-crit, hl-warn, hl-info, hl-event. Subscribe and mute by how much I care.

Then the interesting parts showed up at the edges, where they always do.

Edge one: my own firewall 403’d me

First test, the usage producer POSTing to https://ntfy.hippotion.com:

HTTP 403 Forbidden
error code: 1010

That 1010 looks like ntfy rejecting my token. It isn’t. It’s Cloudflare. Error 1010 means “your browser signature is banned” — Cloudflare’s bot protection took one look at a Python script’s urllib User-Agent and slammed the door.

My own producer couldn’t reach my own bus, because the request left the cluster, went all the way out to my own edge, and got flagged as a bot on the way back in.

The fix is the architecture I should’ve had from the start: in-cluster producers POST to the internal service address and never touch the public internet at all.

# wrong: out to Cloudflare and back, gets bot-blocked
https://ntfy.hippotion.com/hl-warn

# right: stays inside the cluster
http://ntfy.web-ntfy.svc.cluster.local/hl-warn

The phone still uses the public URL happily — the real ntfy app carries a signature Cloudflare trusts. Only scripts trip 1010. Lesson: your own edge is not your friend when you’re a script. Keep cluster traffic in the cluster.

Edge two: the obvious data source was lying

To warn me about Claude usage, the naïve move is to parse Claude Code’s local logs — they sit right there in ~/.claude/projects/.../*.jsonl, token counts and all.

Don’t. Those counts are unreliable for accounting — known to undercount, wildly, in some cases by ~100x. Every tool that parses that JSONL inherits the bug.

The number that’s actually true lives in the claude.ai usage API — the same five_hour and seven_day windows your plan enforces against. And I already had a service polling exactly that. So the producer is just a tiny sidecar on that existing pod, reading its /api/usage over localhost (same pod — no network policy to negotiate, no second credential, nothing else hammering claude.ai):

• 📈 ≥80% of a window → hl-warn (high).
• 🚨 ≥95% → hl-crit (urgent).
• 🔁 One ping per window per reset cycle, escalating warn→crit, keyed on the reset timestamp so it never spams.

The first time it mattered, my phone buzzed at 80% with hours of runway left instead of a brick wall mid-task.

What I’d tell past me

Three things, none of them about ntfy:

1. Reuse the signal you already have. I didn’t build a usage poller — I bolted a sidecar onto the one already running. The smallest producer is one that reads localhost.
2. Your own edge can betray you. A firewall that protects you from bots will happily block your own automation. In-cluster talks in-cluster.
3. Check whether your data source is telling the truth before you build an alert on it. An alert you don’t trust is worse than no alert — you’ll learn to ignore it, and then it’ll be right once.

Next, the high-leverage move: point Prometheus Alertmanager at the same bus, and every infra alert I have — plus every one I’ll ever add — lands on the phone through one bridge. The kombucha ping can wait. The disk-full one can’t.

The house is still full of quiet robots. The difference is now they know my number.

When I started the homelab I looked at the standard ways to make self-hosted services accessible and found the usual compromises:

Open ports on the router. Point a DNS record at your public IP, forward 443 to your server. Simple. Also: your home IP is now publicly associated with your services, your ISP can see your traffic, and a misconfigured app means an open door. Hard pass.

VPN for everything. WireGuard on the router, every device gets a tunnel. Secure, but every phone and laptop needs to be configured, split tunnelling adds complexity, and “let me pull up the recipe” becomes a two-tap operation that my family won’t do. And I still want public access for some services.

Cloudflare Tunnel, but broken HTTPS locally. This is the one I started with. The cloudflared pod dials out to Cloudflare, no open ports, external access works great. But locally, *.hippotion.com resolves to Cloudflare’s anycast IP, traffic leaves the house, Cloudflare terminates TLS, traffic comes back in through the tunnel. Every local request makes a round trip to a Cloudflare edge node. Worse: browsers cache HSTS for hippotion.com, so http:// URLs on the local network silently upgrade to https://, which fails because there’s no local certificate. Intermittent, confusing, and hard to explain to anyone else on the network.

What I wanted: the tunnel for external access, direct-to-server for local access, real TLS in both cases, and one configuration per application.

The insight: Pi-hole already controls local DNS

My network already runs Pi-hole for ad blocking. Pi-hole uses dnsmasq under the hood and can resolve any hostname to any IP you want. One config line in Pi-hole’s values:

dnsmasq:
  customDnsEntries:
    - address=/hippotion.com/192.168.0.109

The address= directive is a wildcard. Every device on the LAN that uses Pi-hole for DNS — which is all of them, because the router hands out Pi-hole’s IP via DHCP — will now resolve anything.hippotion.com to 192.168.0.109, the server’s LAN address. External traffic still goes to Cloudflare’s IP because it uses the public authoritative DNS. The split is automatic; no per-device configuration.

Local browser → Pi-hole → server’s LAN IP directly.

That solves routing. Now TLS.

Why HTTP-01 won’t work here, and why DNS-01 will

The standard way to get a Let’s Encrypt certificate is the HTTP-01 challenge: Let’s Encrypt sends a request to http://yourdomain.com/.well-known/acme-challenge/<token> and your server responds. This requires Let’s Encrypt’s servers to reach your server over the public internet.

That doesn’t work here. There are no open ports. Let’s Encrypt can’t reach the server. HTTP-01 is out.

DNS-01 is different. Instead of proving you control a server, you prove you control the DNS zone by creating a temporary TXT record at _acme-challenge.yourdomain.com. Let’s Encrypt checks DNS, finds the record, issues the cert. No inbound connection required — just API access to your DNS provider.

hippotion.com is on Cloudflare. cert-manager has a Cloudflare DNS solver that calls the Cloudflare API to create and delete the TXT record automatically. The certificate request flow:

1. cert-manager creates an ACME Order with Let’s Encrypt
2. cert-manager calls the Cloudflare API: add _acme-challenge.hippotion.com TXT <token>
3. Let’s Encrypt queries DNS, finds the record, issues the cert
4. cert-manager deletes the TXT record, writes the cert to a Kubernetes Secret
5. cert-manager renews automatically ~30 days before expiry

The Cloudflare API token needs Zone:DNS:Edit permission for hippotion.com. It lives in Vault and syncs to the sys-cert-manager namespace via External Secrets Operator — same pattern as every other secret in the cluster, nothing in Git.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hippotion-wildcard
  namespace: sys-traefik
spec:
  secretName: hippotion-wildcard-tls
  issuerRef:
    name: letsencrypt-cloudflare
    kind: ClusterIssuer
  dnsNames:
    - "hippotion.com"
    - "*.hippotion.com"

One certificate. Every subdomain. cert-manager stores it as hippotion-wildcard-tls in the sys-traefik namespace, where Traefik can read it.

Two Traefik entrypoints

Traefik has three entrypoints configured:

The cloudflare entrypoint is the key piece. Cloudflare Tunnel terminates TLS at Cloudflare’s edge and forwards plain HTTP to the cluster. If that plain HTTP landed on web (port 80), it would get redirected to websecure (port 443), which would fail because cloudflared isn’t sending HTTPS. A separate entrypoint on a separate port handles tunnel traffic without redirection.

Traefik is configured to use the wildcard cert as its default:

tlsStore:
  default:
    defaultCertificate:
      secretName: hippotion-wildcard-tls

Any websecure request that doesn’t match a more specific TLS configuration gets the wildcard cert. No per-app certificate configuration.

One IngressRoute handles both paths

Every application gets a single IngressRoute with both entrypoints:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: myapp
  namespace: myapp
spec:
  entryPoints:
    - cloudflare   # plain HTTP from cloudflared
    - websecure    # local HTTPS with wildcard cert
  routes:
    - match: Host(`myapp.hippotion.com`)
      kind: Rule
      middlewares:
        - name: oauth-auth
          namespace: sys-oauth2-gitlab
      services:
        - name: myapp
          port: 8080

That’s it. The same hostname, the same routing rule, the same middleware — served correctly on both paths. No conditional logic, no separate ingress for local vs external.

The OAuth middleware (oauth-auth) works on both paths too. Local browsers get redirected to GitLab for authentication the same way external browsers do. The SSO cookie is set on hippotion.com, so it works across all subdomains regardless of which path the traffic came through.

What the two traffic paths look like end to end

External browser (anywhere):
  Browser
    → Cloudflare DNS (hippotion.com → Cloudflare anycast IP)
    → Cloudflare edge (TLS terminated, certificate managed by Cloudflare)
    → cloudflared pod in cluster (plain HTTP)
    → Traefik :7080 (cloudflare entrypoint)
    → app pod

Local browser (home WiFi):
  Browser
    → Pi-hole DNS (*.hippotion.com → 192.168.0.109)
    → Traefik :443 (websecure entrypoint)
    → Traefik serves hippotion-wildcard-tls (Let's Encrypt cert, trusted by browser)
    → app pod

Both paths hit the same Traefik IngressRoute rule. The app sees an HTTP request either way. TLS is handled at the edge — Cloudflare for external traffic, Traefik for local traffic.

The HSTS detail

Cloudflare likely has HSTS enabled for your domain. Browsers cache this: once they see an HSTS header for hippotion.com, they’ll refuse to load any http:// URL under that domain for the duration of the max-age. They silently upgrade to https:// and fail if there’s no cert.

This is why the original setup — tunnel only, no local cert — felt unreliable locally. The browser was doing the right thing (enforcing HTTPS) but the cert didn’t exist. The wildcard cert fixes this because HTTPS now actually works locally. The HSTS enforcement is fine once TLS is real.

What it’s like to operate

Adding a new service means writing one IngressRoute with two entrypoints and pushing to Git. No DNS records to create (cloudflared picks up hostnames from a config list), no certificates to request (the wildcard covers everything), no VPN profiles to distribute. The platform handles it.

Local access works when the internet is down. The Pi-hole DNS and the wildcard cert are entirely on-premises — as long as the server is up, the services are reachable, Cloudflare outage or not. I noticed this during a brief Cloudflare incident a few months ago: external access went down, everything inside the house kept working without interruption.

I’m not a networking expert. I just followed the constraint — no open ports, no VPN — and the DNS-01 + split DNS solution fell out naturally. It turned out to be simpler to configure than the alternatives, and cleaner to operate.