Verification is the product

The agent said done. The test said no.

That should be the least dramatic sentence in this whole series. It is also the line that separates a coding agent from a code-flavored chatbot. If the harness cannot make the model prove the change, the human becomes the verification layer by default. That works for a demo. It does not work for a workflow.

The trap is that agents are very good at producing completion-shaped language. Completion sentences all sound final: "Implemented support for split tax lines," "Added tests," "All checks pass." Those words arrive with the same confidence whether the command ran or not. A decent harness should make that confidence expensive.

Pi already gives the agent a shell. The missing decision is which commands count. For tiny-ledger, the answer is not mysterious: parser changes need fixtures, tests, and npm test. If the agent touches the parser and does not run the gate, it has not finished the task. It has merely edited the repo.

This is where extensions stop being a plugin story and start being the harness story. A verification gate is just a small piece of project taste made executable.

Start with manual verification

In Pi interactive mode, a shell command prefixed with ! runs and sends the output back into the model context:

!npm test

That is useful when the model needs to read the result and decide what to do next.

A command prefixed with !! runs without stuffing the output into model context:

!!npm test -- --watch

That is useful when the command is for the human or for long-running noise. Context is part of the harness budget. Dumping every watcher tick into the model makes the transcript worse on purpose.

At this stage, manual verification is enough. Tell Pi the rule in AGENTS.md:

# tiny-ledger agent rules

- Parser changes need a fixture and a test.
- Run `npm test` before claiming done.
- If `npm test` fails, report the failing command and the next smallest fix.

Then make the agent earn the summary:

Add support for subtotal lines. Keep the parser small, add a fixture, add a test, and run npm test before you say done.

A useful trace is boring and specific:

agent: edited src/parseReceipt.js
agent: added fixtures/cafe-subtotal.txt
agent: added test/parseReceipt-subtotal.test.js
agent: ran npm test
shell: 2 tests passed
agent: summary

If the summary comes before the shell evidence, the harness is letting prose outrun proof.

Make the gate callable

Manual rules are a start. Project-level commands are better. A Pi extension can register a tool that runs the exact check for this repo.

Put this in .pi/extensions/verify.ts:

import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
import { Type } from "typebox";

export default function extension(pi: ExtensionAPI) {
  pi.registerTool({
    name: "verify_tiny_ledger",
    label: "Verify tiny-ledger",
    description: "Run the tiny-ledger verification gate and return stdout/stderr.",
    parameters: Type.Object({}),
    async execute(_toolCallId, _params, signal) {
      const result = await pi.exec("npm", ["test"], { signal });
      const output = [result.stdout, result.stderr].filter(Boolean).join("\n");

      return {
        content: [{ type: "text", text: output || `npm test exited ${result.code}` }],
        details: { code: result.code, killed: result.killed },
      };
    },
  });
}

Reload Pi:

/reload

Then ask the agent to use the gate:

Fix the subtotal parsing bug. Use verify_tiny_ledger before summarizing.

This gate is a named piece of evidence. It does not prove the code is good; it proves the configured check ran and returned a result. That matters because the harness can now point the model at the same command every time instead of hoping the prompt ritual sticks.

Use warnings before cages

The next temptation is to block every risky edit. Blocking everything feels responsible. Sometimes it is. Often it creates a little process prison where the agent spends more time negotiating the harness than doing the task.

Start with warnings.

import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";

export default function extension(pi: ExtensionAPI) {
  pi.on("tool_call", async (event, ctx) => {
    if ((event.toolName !== "edit" && event.toolName !== "write") || !("path" in event.input)) {
      return;
    }

    const path = String(event.input.path);
    if (!path.startsWith("src/parser/") && path !== "src/parseReceipt.js") return;

    ctx.ui.notify("Parser edits need a fixture and a test before you claim done.", "warning");
  });
}

If you later need a hard gate, tool_call handlers can return:

return { block: true, reason: "Parser edits need a fixture and a test." };

Do not start there unless you know the rule is safe. A parser edit might be paired with a test edit in the next tool call. Blocking the first call because the whole diff is not visible yet is how safety theater becomes friction.

What the gate proves

The gate proves that the configured command passed. Nothing more.

That sounds smaller than the marketing version, because it is. Passing tests does not prove the parser is good. It does not prove the API is pleasant. It does not prove the model understood the domain. The repo accepted this evidence now; that is the whole claim.

That limited proof still matters. In practice, most coding-agent failures are not metaphysical. They are boring:

• the agent edited the wrong file
• the test command was never run
• the output failed but the summary ignored it
• the new behavior had no fixture
• the agent fixed the symptom by deleting the check

Verification gates catch a lot of that. Not all. Enough.

Exercise

Extend tiny-ledger with subtotal lines.

Fixture:

cat > fixtures/cafe-subtotal.txt <<'EOF'
Coffee 3.50
Bagel 4.25
Subtotal 7.75
EOF

Target output:

[
  { name: "Coffee", price: 3.5 },
  { name: "Bagel", price: 4.25 },
  { type: "subtotal", amount: 7.75 },
]

Run the exercise in three passes:

1. Ask Pi to implement it with only the AGENTS.md rule.
2. Add verify_tiny_ledger and ask Pi to use the tool before summarizing.
3. Add the parser warning hook and watch whether it changes the shape of the session.

Compare the traces. Ignore the scoreboard for a minute and watch where evidence entered the loop.

The annoying but useful conclusion

The harness does not need to know truth. It needs to know what evidence the repo accepts.

That is less glamorous than "autonomous software engineer," but it is much closer to the thing people need. A model that can edit code is interesting. A harness that forces the edit to meet the repo's definition of done is useful.

Next primitive

The next post is about hooks — the place where the boring rules become hard to forget without turning the harness into a process prison.