Scoby: I built a free Kimchi

In July I gave Kimchi my kombucha app. What stayed with me was not the code it wrote. It was the shape: a planner, a builder and a judge on different models, checkpoints between them, and a mode called Ferment that scopes before it codes. I wanted that shape on my own machine, and I wanted it to cost nothing. Not because I am cheap. Because if it runs on free tiers, I can leave it running on things that have time.

So I built one. It is called scoby, the culture that ferments kombucha, and it lives at github.com/janos-gyorgy/scoby. It is a set of extensions for pi, the coding agent SDK Kimchi itself is built on. Honest context, same as last time: Claude wrote almost all of the code. I chose what to build, which numbers to believe, and when to stop.

What “free” actually means

I started with a plan on paper: Groq for speed, Gemini for quality, NVIDIA NIM for variety, Cerebras as a bonus. Then I measured.

Nobody publishes these numbers. You find them in the response headers and in the text of the 429. So the first thing scoby has is a router: you define connections, you map each role to an ordered list of models, and when a model says no (429, 503, a dead key, a network error) the session moves to the next one before pi’s own retry fires, and moves back when the first one recovers. A 400 does not fail over, because a 400 is usually my mistake and hiding it would be worse.

Compaction, inside out

Every compaction I looked at, Kimchi’s included, solves the same problem: a long session in a big context window. It waits until the context is nearly full, then asks a model to summarize the old part. Kimchi’s July session compacted twelve times, and each summary folded in the previous one, so the summary itself grew from 11K to 30K characters.

Free tiers pose the opposite problem. The window is big. The request is small. Groq does not care that the model has 131K of context if I may only send 8K per minute. So scoby shapes the context before every single request, to a budget per connection, in a fixed order: replace old content with a fold summary if one exists, drop repeated file reads, replace old tool output with a one-line stub that says how to get it back, trim old assistant prose, and only then drop the oldest things. The first message, the last message and a recent zone are never touched. It measures itself: after every response it compares real input tokens to its own estimate and adjusts, per model, because three models on the same endpoint have three tokenizers, and one shared correction let 17 requests through over budget before I keyed it by model.

Does it hold? On the same task I gave Kimchi, at a 32K budget the largest request was 30.4K tokens; at 16K it was 15.7K. The layers engaged in proportion to the pressure: 32K needed five folds and three stubs, 16K needed ten folds, 21 stubs and 27 dropped units. Below about 32K the cost is time, not correctness: the 16K run spent its turns re-reading what it had just elided and ran out the clock.

One thing I expected and still found surprising: the model never once called the tool that restores elided output. Zero calls in five runs. The stub tells it how. The system prompt mentions it. It does not happen. A paper on this (the billion-context-pi work) says the same. Models do not use recovery tools unless you make them.

The run that waited three hours

The first complete free run of the Kimchi task took eight invocations. Seven times a provider went down mid-run; scoby stopped, a small supervisor waited twenty minutes, probed, and continued the same session at the same step. 328 requests, seven million input tokens, three hours of wall time, zero dollars. Eleven of eleven steps done, gates green.

Then I had the same external judge that graded Kimchi’s run in July grade this one. Kimchi got a B. Scoby got a C. It built the lead-time warning that Kimchi had missed. It missed the integration that Kimchi had: starting a batch did not consume starter, because the plan chose a manual ledger and never connected it to brewing.

And here is the part I keep thinking about. Scoby’s own judge, the one that grades each phase from the diff, gave that run A, D, A, with the D fixed to an A in one round. It was not wrong. It graded each phase against the plan, and the phases did what the plan said. The plan was what was missing something. A judge anchored to the plan inherits the plan’s blind spots, and no amount of model quality fixes that, because the judge is not asked the right question.

In July, the thing that caught exactly this miss was me, at Kimchi’s scoping step, in one sentence.

The fair objection is that I scoped the judge too narrowly. Grade the plan against the task before any code, grade the result against the task after, and a model might catch this too. I think that is right, and it is the next thing to try. But today the cheapest check that I know works is a person reading a plan, and it costs one sentence.

Real work

So the next thing I built was the checkpoint. You open pi in a repo, you type what you want, scoby asks one question (plan and build, or just answer), and then the plan sits above your editor with three buttons: approve, change (one sentence, and it replans), cancel. It builds on its own branch. Your phone gets an ntfy push when the plan is ready, when a phase is graded, and when it needs you. If every model is down, it waits inside pi and carries on when one answers.

Two real builds so far, both on NIM’s free tier with DeepSeek V4 Flash building and Gemini 3.5 Flash planning and judging.

The first was a small thing on a canvas site where programming languages show up as stars in the night sky: make Bash stand out, magenta, a subtle pulse. 31 minutes, 31 requests, no errors. The plan had a step that made no sense for a canvas (a CSS animation for a label that is not a DOM element); the builder noticed and did the right thing anyway. The judge gave phase one a D, because only the label was colored and not the star. One fix round later it was an A, and it was right: the star pulses now.

The second was unattended, on the shared chassis of my agent fleet: push a notification to my phone when a bird run ends, with the right priority for a failure. 113 minutes, 70 requests, no errors, every request under the 32K budget. The code reads well. It mirrors the discipline the chassis already has: the same set of OK statuses, the payload built by a JSON encoder because the detail can contain model-chosen text, a bearer token that is never echoed, a best-effort curl that cannot slow a run. Docs updated, the fleet screen shows “ntfy on”. Three phases, three A grades.

Then I ran the new function against my real ntfy server. The message arrived as raw JSON text. No title, no priority, no tag. ntfy only parses a JSON body when you post it to the server root with the topic inside; post it to the topic URL and the body is the message. The builder did not know that, the judge did not know that, the gates cannot know that. A person with a phone found it in one try. So I typed the finding as a new request, the way I would in the TUI, and let scoby fix its own work.

Fifty minutes and 43 requests later: title, priority and tags as headers, the body still plain text built by the same encoder, and a comment that explains why. I sent three fake outcomes through it to the real server. A failure came in high with a warning tag, a quiet run came in low, a run with finds came in at default priority, quotes and newlines in the detail intact. And while testing I found that an older ping in the same chassis had been failing silently all along: the topic answers 403 without a token, and the curl swallows it by design. Best-effort notifications fail quietly. That is what they are for, and also why you test them from the phone side.

What I can defend

Claude wrote the code; these are the parts I checked myself. The build engine is a state machine with no model in it, tested on its own. Failover, the resume after an outage, the budget and the plan approval are tested end to end against mock providers that return 429 and 503 on cue, and a scripted human speaking pi’s RPC protocol. Every request writes its budget, its estimate, its real token count and what was cut into the session file. Every number in this post comes from those files, not from memory.

What it is not

It is not a sandbox. pi’s bash runs as me, in my repo, and the files go to whoever is behind the API. I run it on repos I would publish anyway, on a branch, and I read the diff. The free tiers change weekly and I promise no maintenance. It works today, on my machine, and the repo says exactly how.

What I got out of it is not the tool. It is a clearer picture of where the value in Kimchi’s shape actually sits. The three models matter. The gates matter. But the one thing that caught the real mistake, twice now, was a human reading the plan before any code existed. That is the cheapest step in the whole loop, and the one I would keep if I had to throw everything else away.