A dialogue concerning checkboxes

Persons of the dialogue: PAULUS, who has been writing software with an agent for some months. TIRO, a young programmer who wants to start. The agent, which Paulus calls Claudius, is at work in the next room. The scene is forsitan modulare, a plugin of modules for VCV Rack. Rome, late October.

TIRO. I looked at your repository. 1059 commits, 1058 with your name on them. They say you no longer write the code.

PAULUS. They are right. Claudius writes it.

TIRO. Then teach me how you talk to it. What do you put in your prompts?

PAULUS. The prompt is the smallest part. Tell me: when an agent hands you code that looks right but is wrong, how do you notice?

TIRO. I read it carefully.

PAULUS. Nine hundred commits in one summer. Will you read them all carefully?

TIRO. No. I would test it, I suppose.

PAULUS. Then you already see the shape of the work. Writing code is what you delegate. What you keep is building the conditions in which a wrong result cannot hide. I can give you eight precepts. None of them is about prompts.

I. Give it the right memory

PAULUS. Suppose you correct Claudius on Monday. What does it remember on Tuesday?

TIRO. What I told it on Monday?

PAULUS. Nothing. Every session starts from zero. So where must the correction live?

TIRO. Somewhere it reads every time.

PAULUS. In my case, a file called CLAUDE.md at the root of the repository. The agent reads it at the start of every session. Mine is 520 lines long.

TIRO. You wrote 520 lines of instructions before starting?

PAULUS. Not one of them before. Each rule arrived after something went wrong. Read it carefully and it is not a style guide: it is a record of scars.

TIRO. Then why not keep it short? “Do this, never do that.”

PAULUS. Because a rule without its reason gets applied to the letter in the wrong case, and ignored in the right one. Every rule in that file carries its why, often with the name of the module where the trouble happened. The agent is not obedient, Tiro. It is reasonable. Give it reasons.

II. Never let it guess

PAULUS. Here is one of those scars. When Claudius builds a test patch for me, a script must say things like “turn this knob” or “plug this cable into that input”. Inside Rack, how do you think a knob is identified?

TIRO. By its name?

PAULUS. By a number: its position in the order the module’s author happened to declare things. On the standard VCO, which number would you expect the frequency knob to be?

TIRO. The first. Zero.

PAULUS. It is 2. On the VCA, the audio input is 2 as well. Both had been guessed, and both were wrong. Now: what happens when such a guess is wrong?

TIRO. An error, I hope.

PAULUS. Nothing. The patch opens, sound comes out, no complaint. But the cable sits in the wrong socket, and I am listening to something other than what the test describes. I might even approve it.

TIRO. That is worse than a crash.

PAULUS. Much worse. A crash tells you. A plausible guess lies to you quietly. So now nothing is addressed by number: every knob and socket is called by its name, the names are read from Rack itself, and a name that matches nothing stops the script. Wherever the agent might fill a gap with something plausible, take the gap away.

III. Give it hands and eyes

TIRO. You said the names are read from Rack itself. But Claudius cannot see your screen.

PAULUS. No. And Rack is made to be played with a mouse: it has no interface for a program. So what did we have to do?

TIRO. Build one?

PAULUS. We built limen. It is a module like the others in the plugin, except it makes no sound: three units wide, a green light and little else. It opens a local network port and answers questions in JSON. Which modules are in the rack, what each knob is called and where it is set, which cables connect what. It also takes orders: turn this knob, move this module, quit.

TIRO. Why a module? A script outside Rack could read the patch file.

PAULUS. A file tells you how things were when it was saved. A module inside Rack tells you how they are now. That is how the names came out right: a script loads each module through limen and asks it what its knobs and sockets are called.

TIRO. And “quit”? It seems an odd thing to need.

PAULUS. Another scar. These tools run against my real Rack setup. Rack saves my settings on the way out, and killing the process can damage them. So every tool that starts Rack closes it through limen, and waits. When you give an agent hands, Tiro, give it good manners too.

TIRO. So the lesson is: build the agent its own instruments.

PAULUS. An agent that can only read code will reason about code. An agent that can observe the running system will reason about the system. Hours spent on instruments are never wasted.

IV. What yields a number, a machine checks

TIRO. Now testing. How do you know a module is finished?

PAULUS. There are two layers. The first Claudius writes: a smoke test for every module, which runs its audio code outside Rack and checks that no NaNs come out, that levels do not run away, that things oscillate when they should and decay when they should. Next to it, probes that print measurements.

TIRO. And the second?

PAULUS. The audition: a file of about a dozen tests that I do by ear. Two lines each: what to set, what to expect.

TIRO. Only a dozen? I would want to check everything.

PAULUS. Then you will write fifty items, and you will never finish them. An audition nobody finishes tests nothing. So there is one rule that divides the two layers. If a test yields a number, where does it belong?

TIRO. In the audition, written down, so you can compare?

PAULUS. It seems reasonable. Suppose you write in the audition that a filter cuts 12 dB at a certain setting. Next week the code changes. Who notices that the figure is now wrong?

TIRO. Nobody, until someone measures it again.

PAULUS. And nobody measures it again, because it is written down and looks settled. A number measured by hand is measured once, against that afternoon’s build, and it is wrong by the next commit.

TIRO. So it goes in the harness.

PAULUS. Where it is measured again every time, and fails loudly when it drifts. The audition gets shorter, the harness gets longer. The machine does what a machine does well, and my ears are spent only on what a machine cannot judge.

V. Keep one copy of the truth

TIRO. When you sit down to audition, who prepares the patch for each test?

PAULUS. Claudius. Rack opens with the patch already built and the text of the test in a notes module beside it. But tell me: how would you be sure the patch it builds is the one the test describes?

TIRO. I would check the patch against the description.

PAULUS. Every time, for every test? There is a simpler way: make them the same text. We invented a small Python language for describing patches, and the code lives inside the audition file, next to the words. A test for radix looks like this:

- [ ] 1.3. **Source** on Self, then step **Law** through its five positions:
      five different kinds of instability, none of them a plain tone.  ```python
      r.set(src="Self", clock=0.537) # about 4 kHz ```

TIRO. For you it is a checklist. For the machine it is a program.

PAULUS. One file, both things. At the top sits the bench every test starts from, each section adds its own setup, each test changes a line. vcv.module("VCO") adds a module, a["sine"] >> b["left"] connects two sockets by name, m.set(amt="90%") turns a knob. And vcv.modulate(...) replaces “sweep the knob slowly” with an LFO at a written-down rate, because a gesture by hand can be neither repeated nor described.

TIRO. Why not keep a short description in words too, for the human reader?

PAULUS. Because two copies of the same thing drift apart, and the first to go quietly wrong is the prose. When we cleaned up duplicated descriptions in the artifex audition, a line that put a scope on the left output went with them. Two tests still said “watch the scope”. There was no scope. Wherever there are two copies, delete one.

VI. Turn what you do twice into a recipe that fails early

TIRO. And releases? Surely those the agent can do by following a list.

PAULUS. Let us see. The plugin manifest holds 34 links to the manuals, and each must point at the tag of the version being released, never at a branch. Otherwise a user with an old version reads today’s manual. How would you keep 34 links right by following a list?

TIRO. Carefully?

PAULUS. Carefully works 33 times out of 34. The rule in CLAUDE.md is: never edit them by hand, run the script. sync_version.py rewrites them all, and with --check it fails if any is stale. The same for the module tags: the VCV library rejects unknown tags only after you have pushed the release. An expensive way to find out.

TIRO. So there is a script that checks them first.

PAULUS. Against Rack’s own list, before anything leaves the machine. The panel images too: nothing in the image folder is made by hand, a script drives Rack and takes the screenshots. Do you see the pattern?

TIRO. Every step the agent could get slightly wrong becomes a script that cannot.

PAULUS. And that fails loudly, as early as possible. An agent is very good at following a procedure, and very good at being slightly wrong at step seven without telling you. A script is not good at anything, except being the same every time.

VII. Write your taste down in numbers

TIRO. The panels of your modules all look alike: dark, yellow badges, the same font. Do you draw them?

PAULUS. Claudius does. It writes the coordinates of every knob, socket and label, and a script generates the SVG. Does that surprise you?

TIRO. A little. Layout is a matter of taste, and taste is yours.

PAULUS. Then how do you hand over taste?

TIRO. I don’t think you can.

PAULUS. Part of it you can, if you can say it in millimetres. At least 1.5 mm between any two elements. Every label at a fixed distance below its control: 7.5 mm under a socket, 8.5 under a small knob. The title always 3.2 mm tall: the width of a panel is no reason to shrink it, only lack of space is. A script, panel_audit.py, measures all of it against the real component sizes and the real width of the font, and fails if anything is off.

TIRO. That is geometry, not taste.

PAULUS. Then listen to this one. When a knob has both a CV input and a trimmer, the two sit on one row beneath it, trimmer left and socket right, and the three form a triangle. That row sits 18 mm under a big knob, 15 under a small one. Use one distance for both, and the rows under the big knobs look tighter. On a panel with both sizes, it shows.

TIRO. Would Claudius notice on its own?

PAULUS. It sees coordinates, not a panel. Someone noticed once and wrote it down, and now a script checks it on every panel, forever. That is the trick: every time you catch yourself saying “it looks wrong”, ask why until the answer is a number. The part of your taste you can state in numbers, you can delegate. The rest stays with you.

VIII. Yours is the final word

TIRO. Seven precepts, and in each one the machine takes over a little more. What is left for you?

PAULUS. Look at the audition files. Next to every test there is a checkbox. Who do you think ticks them?

TIRO. Claudius could. It built the patch, it knows what the test expects.

PAULUS. There is a line in CLAUDE.md that forbids it: the tick marks belong to the user. Claudius may fix the wording of a test, but it must anchor the edit on the text without the box, and it must never tick one on my behalf.

TIRO. It sounds like bureaucracy.

PAULUS. It is the border. Claudius can write the module, the tests, the manual, the presets and the release. It cannot declare that a module has been listened to. Some tests are even marked “Decide”: open questions of judgement, resting on numbers the harness prints. The numbers are its work. The answer is mine.

TIRO. Then let me ask it directly. Claudius! Does the last module you wrote sound good?

CLAUDIUS (from the next room). It passes every check in the harness. No NaNs, no DC, nothing runs away.

TIRO. That is not what I asked.

CLAUDIUS. Then I do not know.

TIRO. And who knows?

CLAUDIUS. Whoever ticks the boxes.

PAULUS. There you have the whole method, Tiro. Give it the right memory, take away its guesses, give it instruments, let machines check numbers, keep one copy of the truth, turn what repeats into recipes, write your taste in numbers. Then go and listen.

The end.