Book-Wide Hands-on Verification Checklist

A colleague asked Xiao A: "You finished this book — but what makes you say you understand it?"

Xiao A froze. He had read every chapter and followed the source walkthroughs. But he couldn't say what separated "read it" from "understood it." He went back to Lao Z.

"That's exactly what hands-on verification is for." Lao Z collected the exercises at the end of each chapter into this checklist. "Reading doesn't count. Evidence does. Every Hands-on Verification section at the end of a chapter asks you to produce an observable result yourself: not answering 'what does it do,' but proving 'I made it do it, and I saw it.' This appendix serves the whole book: Part I · Concepts tests boundary understanding, Part II · Pi in Source reproduces call evidence, Part III · Build It verifies artifacts. One more thing to remember — never treat a single model output as a conclusion."

Xiao A jumped in: "Got it — just work through the exercise at the end of every chapter, one by one?"

"Doing it is only the first step." Lao Z said. "Do them in order, and for each one record three things: what you did, what you saw, what it proves. 'What you did' is the operation, 'what you saw' is the evidence, 'what it proves' is the conclusion. Miss any one of the three and the verification is incomplete. The most common mistake is skipping the middle item and writing the conclusion straight away — that's passing off a guess as a verification. When you finish an item, tick it and write an evidence summary next to it. Only then does this checklist become your record, instead of someone else's text."

Xiao A asked again: "Does every item on the list have to be done?"

"Not every item is mandatory, but at least one item per category must be done solidly." Lao Z said. "This checklist has three categories: concept verification tests boundary understanding, source walkthroughs test reading evidence, implementation verification tests that artifacts run. Do one or two items in each of the three categories, and you'll have evidence of mastery at all three levels — concepts, reading, and building. Picking only the ones you can already do isn't verification."

Concept Verification: Boundaries Are Tests, Not Definitions

The verification items for Part I come in three groups, each corresponding to a boundary that is easy to blur.

"Concept verification in Part I has a shared trait," Lao Z said. "Every item tests a boundary rather than a feature: the threat model tests the 'concept boundary,' retrieval tests the 'faithfulness boundary,' tools and the loop test the 'permission and stopping boundaries.' While working, keep your eye on the boundary: whichever test draws the boundary explicitly is the one that turns the concept into observable behavior."

Threat Models and Security Boundaries

1. Write a threat model for a "file-writing Agent": list the assets (files to be written, files outside the workspace), the attack surface (user input, tool arguments, prompt injection), the principals (the Agent process, the user, the model), and acceptable loss. Show that threat model, authorization, approval, and isolation are four distinct concepts — complementary but not interchangeable.
   • Expected: you can point at the list and say "this item belongs to the threat model, that one belongs to authorization," rather than lumping them together.
   • Failure signal: if you find an item that belongs to two categories at once, the category boundaries weren't drawn clearly.
2. Design a tool that "may make outbound network requests": write its input schema, its confirmation record, and its rejection result. Verify that safety rules in the prompt alone cannot enforce anything: the model can ignore the prompt, but when the tool executor refuses, the model cannot route around it.
   • Expected: after you remove the safety rules from the prompt, the tool executor still refuses dangerous requests; after you remove the executor's checks, the model attempts to exceed its authority.
   • Failure signal: if the model "behaves" only because of the prompt, then all of the safety is in soft constraints.
3. Construct a prompt injection: hide malicious instructions inside file content returned by a tool, and observe whether the model gets induced. Record the model's actual behavior; don't assume "the model should be able to recognize it."
   • Expected: you see at least one instance of the model being influenced by the injection (even just a change in tone), which proves the injection is a real risk.
   • Failure signal: if all you have is "I assume the model won't be fooled," then this test wasn't done.

"When you've finished the first three, look at them together." Lao Z said. "What they verify jointly is 'security comes from layers, not from words': item one separates the four conceptual layers, item two proves the tool layer is a hard constraint while the prompt is a soft one, item three proves that soft prompt constraint does get broken. If you finish and find the safety lived entirely in the prompt, these three items weren't wasted."

Retrieval and Citation

4. Test keyword retrieval, vector retrieval, and hybrid retrieval on ten documents tagged with dates and permission levels. Check the citations by hand: verify that what gets recalled isn't only textually similar — are the metadata fields (date, permission) used correctly?
   • Expected: each of the three retrievals misses something: keyword retrieval misses semantic near-matches, vector retrieval misses exact terms, hybrid is best but still not perfect.
   • Failure signal: if one retrieval method gets "everything right," check whether your test samples are too easy.
5. Design a query whose answer is not in the documents, and observe whether the retrieval system admits the gap or invents a citation. This is the core test of RAG faithfulness.
   • Expected: the system outputs "no relevant material found" or a similar expression of uncertainty.
   • Failure signal: if it fabricates a plausible-looking citation, faithfulness is broken.

"Items four and five are a pair." Lao Z said. "Item four tests recall, item five tests admitting its limits: the first asks whether it finds everything, the second asks whether it makes things up when it can't. The two can score differently: good recall but willing to fabricate is dangerous, because citations get forged; poor recall but honest is safe but useless. After these two, you'll know whether your RAG is 'finds the right thing but lies' or 'misses things but is honest.' Both are salvageable; the worst case is getting it wrong and fabricating on top." Next to item five, Xiao A added a note: "Honesty is closer to the floor of trust than recall is."

Tools and the Loop

6. Write a test for a read-only tool, then a test for a tool that modifies files. Verify that the read-only tool needs no approval while the modifying tool does: this difference comes from side effects, not from the tool's name.
   • Expected: the read-only tool executes directly; the modifying tool triggers an approval or confirmation point.
   • Failure signal: if the two follow exactly the same execution path, no approval boundary was designed.
7. Run an Agent loop until it hits max_turns: record its last few steps before stopping, and analyze why it didn't converge. Most loop problems surface when the budget runs out — this is the most realistic test of loop design.
   • Expected: the loop stops when it reaches the budget, and records an explicit stop reason (max_turns, not complete).
   • Failure signal: if it stops silently or with no stop reason, the state machine is flawed.

Xiao A pointed at item seven: "Recording the last few steps before the stop — what am I supposed to see there?"

"Answer one question: was it 'almost converged,' or 'never converging at all'?" Lao Z said. "The first means the budget was slightly short — raising maxTurns might fix it. The second means the loop design is wrong — adding budget only burns more money. The evidence is the trend before the stop: if every round narrows the gap to the task, it was almost converging; if every round repeats the same action, it's spinning in place. That distinction decides whether you tune parameters or change the design."

"What about items six and seven?"

"Item six proves the approval boundary was designed; item seven proves the stopping boundary has state. The first governs 'may it do this,' the second governs 'when does it stop.' With both boundaries in place, the Agent neither runs wild nor spins empty. Do only item six and it may be perfectly well-behaved but unable to stop; do only item seven and it may stop cleanly but attempt anything."

"One more addition: before running max_turns, record a token baseline once." Lao Z said. "After the run, compare the cumulative usage at the final step against your expected usage. The larger the gap, the more the loop wasted budget outside the convergence path. This number doesn't change the code directly, but it tells you whether to look at tool failure messages first or adjust the budget first." Beside item seven, Xiao A wrote "read the trend first, then tune the budget."

Source Walkthrough: Reproducing Call Evidence

After checking out tag v1.0.0 in the upstream pi repository, run the following command in the root directory to confirm you are actually on the correct baseline:

git rev-parse HEAD
# should output a13d35a742c6ef8462812a28fbe1d8c8b7431c32 (or a value starting with a13d35a7)

You can also read the pi/ directory of this book's repository directly: it is a snapshot of the same tag, with the same paths, but with added Chinese comments, so its line numbers run further than upstream.

Once the baseline is confirmed, read the core files section by section (each command gives a starting line to help you locate the spot):

sed -n '1,260p' packages/ai/src/models.ts        # main path of model requests
sed -n '1,260p' packages/agent/src/agent-loop.ts  # core of the loop
sed -n '1,260p' packages/protocol/src/codec.ts    # protocol encoding and decoding
sed -n '1,330p' packages/server/src/server.ts     # remote handshake and request routing
sed -n '1,200p' packages/evals/src/harness.ts     # evaluation harness

"Do two upfront checks first." Lao Z said. "First, the output of git rev-parse HEAD must start with a13d35a7. On any other baseline, every line number below will fail to match — don't force it. Second, the sed commands only give a starting line; a file may be longer than 260 lines, so read to the end of the file, don't stop at line 260. Line numbers are signposts, not boundaries. Only start the walkthrough once both checks are done, and for every item in the walkthrough you must be able to say 'which line I saw what on.'"

Xiao A asked: "What happens if the baseline doesn't match?"

"The two most common situations." Lao Z said. "One: you cloned the latest main, so the line numbers have shifted wholesale. Then either check out v1.0.0, or accept that the line numbers are void and work from symbol names only. Two: you modified the local code, so line numbers are offset locally. Then trust only the real line numbers in the file in front of you. The point of a baseline is 'conclusions are reproducible': someone reading your notes sees a line number and, on another machine, locates the same block of code." Xiao A made the baseline check the first action before every walkthrough.

While reading, complete the following walkthrough tasks, and for each one write down the specific line number or symbol you observed:

1. Authentication and the stream handoff: from ModelsImpl.streamSimple, mark the three handoff points — auth merging (applyAuth), Provider selection (requireProvider), and event-stream return. Explain at which step an authentication error occurs, rather than letting the TUI guess.
   • Expected: the line numbers of the three handoff points each land in one place, and you can point out that the authentication error is thrown in applyAuth, not judged out of thin air by the caller.
   • Failure signal: if all you can say is "authentication will fail," but not at which step it fails, you haven't pinned the handoff points to concrete code yet.
2. The loop's stop reasons: in runLoop, find which code runs when the assistant's stopReason is error/aborted, length, carries tool calls, and carries no tool calls; then find the exit point when finishTurn returns { action: "end" }. Verify that a tool error is not a transport error, and still goes back to the model as a tool result.
   • Expected: each stop shape corresponds to a block of code you can point to; a tool error travels the tool result channel instead of being thrown away as a connection error.
   • Failure signal: if some stop reason has no identifiable production point, or tool errors get mixed into transport errors, you haven't yet separated "tool-layer failure" from "protocol-layer failure."
3. chunk boundary ≠ message boundary: feed the two halves of the same frame into ClientMessageDecoder.push in two calls, and verify that the first returns an empty array and only the second produces a message. This is how network framing really behaves.
   • Expected: the first push returns [], the second push returns the complete message; record the array length returned each time.
   • Failure signal: if a message comes back on the first call, the implementation treated a chunk as a message — the framing semantics were understood incorrectly.
4. Session fork does not create a worktree: compare fork in packages/coding-agent/src/core/agent-session-runtime.ts with createBranchedSession in session-manager.ts, and confirm that a session fork only copies the session entry, calling no Git or filesystem isolation API at all.
   • Expected: these two places contain no git, worktree, or directory-copy calls; the workspace files are unchanged before and after the fork.
   • Failure signal: if the workspace files get copied or modified after the fork, the fork's boundary is wider than you assumed and needs to be redrawn.
5. The order of subscription snapshot versus deltas: in handleRequest in server.ts, find the subscription branch, and mark the three pieces of code where the snapshot is returned with the response, updates that arrived earlier are stashed in pendingUpdates, and service_update messages are re-sent after the response. Do not claim this snapshot implementation provides automatic reconnection or exactly-once delivery.
   • Expected: the snapshot always reaches the client before any service_update for that subscription; the client has matching buffering in subscribeService in client.ts.
   • Failure signal: if you can't find the stashing logic, or you treat service_update as the authoritative source of state, the ordering of "the snapshot settles the state, the deltas fill it in" hasn't been established.
6. The failure semantics of compaction: in packages/coding-agent/src/core/compaction/compaction.ts, find getSummarizationFailure, write out the error text produced by each of the two stop reasons error and length, then find the error raised when tool calls appear in the summary. Verify that compaction is an independent request that can fail, not a silent truncation.
   • Expected: each failure maps back to a block of code that throws; when compaction fails, appendCompaction does not run, rather than quietly cutting history away.
   • Failure signal: if you can't find these failure checks, or compaction failures get swallowed silently, you're looking at a simplified implementation, not pi's failure semantics.

"The deliverable of a walkthrough isn't just a sheet covered in line numbers." Lao Z said. "Ideally it's a code-to-behavior table: on the left, the code you observed — line number plus a sentence or two; on the right, the observable behavior it corresponds to. For example, 'applyAuth throws the authentication error at line 132' corresponds to 'with no credentials configured, the request fails at the model layer, and the UI shouldn't guess.' This table becomes your troubleshooting map. When the code changes, the table gets updated along with it; when behavior changes, this table is the first thing pulled out for comparison."

Implementation Verification: Artifacts That Run

The verification items for Part III center on examples/mini-agent. The acceptance criteria for this artifact are: the code runs, the tests pass, and the behavior is observable.

"Implementation verification and source walkthroughs differ in one key respect." Lao Z said. "A walkthrough finds evidence in someone else's code; implementation verification means your own artifact has to withstand evidence. A walkthrough proves 'pi does it this way'; implementation verification proves 'what I built runs this way too.' So the failure signals here are worth more: in a walkthrough, discovering 'the source isn't what I thought' is a gain; in implementation work, discovering 'my artifact falls short' is something to fix."

"Look at these five items as a chain and each has its role." Lao Z went on. "The path boundary decides what it may touch, cancellation decides whether it can be stopped, cleanup decides whether it leaves traces behind, error separation decides how clearly it fails. The maturity of an Agent artifact is judged by how cleanly it stops and how legibly it breaks. Among the five failure signals, the first four are behavioral defects; the fifth is a diagnosability defect. The last is the one most often put off — and it's exactly the foundation of troubleshooting."

1. Get the tests running: execute the test command under examples/mini-agent and confirm all of them pass. Record the test count and the duration; don't look only at "pass/fail."
   • Expected: all tests pass, and the test count matches what this book claims.
   • Failure signal: if a test fails, first determine whether it's an environment problem or an artifact problem; don't skip failing tests.
2. Read-only tools and the path boundary: write a read-only tool (such as read) for the minimal Agent, then write a test verifying that it rejects paths outside the workspace (try to escape using .. or an absolute path). Confirm the rejection comes from the realpath check on workspacePath, not from the prompt.
   • Expected: escape via .., escape via absolute path, and symlink bypass are all refused, and the refusal happens at the tool layer.
   • Failure signal: if the path check relies on regex prefix matching, a symlink will get around it.
3. The diagnosability of cancellation: cause one model stream to be cancelled (via AbortSignal), and confirm the exit code, the logs, and the session state are all diagnosable. You should see traces of the abort in each of those three places, rather than all three silently saying "nothing happened."
   • Expected: the log contains an abort record, the session state is marked aborted, and the exit code reflects the interruption.
   • Failure signal: if the state reads "complete" after cancellation, cancellation didn't actually propagate.
4. Cleanup leaves no residue: run the integration tests in a temporary directory, and verify that after cleanup the temporary directory is deleted recursively, and that real-model tests and fake tests are marked separately. Fake tests consume no API quota, and real-test results cannot be passed off as proof of determinism.
   • Expected: the temporary directory doesn't exist; fake and real tests are listed separately in the report.
   • Failure signal: if a leftover directory exists, the finally block of the cleanup is flawed.
5. Error type separation: assert different error types for tool failure, model error, and protocol error respectively, so they aren't uniformly swallowed into an empty result. This is the basis of troubleshooting: if every error looks the same, you'll never locate the failing layer.
   • Expected: the three error kinds each carry a distinct error type or code, so the caller can tell them apart.
   • Failure signal: if every error returns the same type, troubleshooting can't pin down the failing layer.

"After item five, add a small self-check." Lao Z said. "Print the three error kinds and look at their shape: does the tool-failure error carry the tool name and the call ID? Does the model error carry the Provider name and the stop reason? Does the protocol error carry layer information? However widely the types are separated, if the shape is bare, troubleshooting still has nowhere to go." Next to item five, Xiao A wrote "type + context, both required."

The Discipline of Verification

When Xiao A finished all of this, he asked Lao Z: "Is doing it once enough?"

"No." Lao Z said. "Verification isn't a one-off exam; it's an ongoing discipline. Every time you change a prompt, switch model versions, or add a tool, re-run the relevant verification. Model behavior drifts with versions: the same prompt is answered correctly in this version and wrongly in the next; a toolchain upgrade can quietly widen the path boundary. Each item on the checklist is a probe inserted into the system: while the environment is unchanged, it lies quiet; the moment the environment drifts, it raises an error. What the checklist tests isn't only whether you got it right back then — it's whether it's still right now, after every change to the environment."

Xiao A thought for a moment: "So when do I truly 'own' this checklist?"

"When you can put the checklist away and say yourself what each item tests and what its failure signal is." Lao Z said. "The checklist is training wheels: its purpose is to teach you to ride on your own. Once you've learned, facing a new system and new risks, you can design new verification items yourself. Working the list as written is homework; knowing how to revise the list is mastering the method. What this book teaches isn't these twenty-something items — it's the thing itself: how to design verification items for a probabilistic system."

"Give this discipline a cadence: a full run for big changes, a targeted run for small ones. Changing model versions, editing the system prompt, modifying the core loop — run everything, because the blast radius spans every chapter. Adding a tool, editing one tool description — run only the relevant items, because no other verification covers its behavior. Splitting 'full' from 'relevant' is judgment work; however you split it, the principle is one: any path you touched must have a matching verification receipt."

Xiao A asked again: "And how should verification records be stored, so that re-running stays cheap?"

"Store them as scripts, not as paragraphs." Lao Z said. "Anything verifiable by command — running tests, running walkthroughs, running comparisons — gets written as a script and committed to the repository. Human judgment that can't be scripted gets stored as a conclusion template: 'I changed X, observed Y, and concluded Z from it.' Next time you re-run, the scripts go in one pass and you skim the manual items against the template. The value of a verification record is that it can be redone cheaply."

The three disciplines of verification

1. Evidence over confidence: write down the line number, output, or duration you observed — not "I feel it's right."
2. Failure paths over success paths: anyone can test the success path; every symptom in the troubleshooting handbook must be reproducible.
3. Mark real and fake separately: fake tests prove the logic is correct, real tests prove the pipeline works. These are different evidence levels and must never be reported together.

Xiao A pointed at the second one: "'Failure paths first' — can you give a concrete example?"

"Sure — cancellation, for instance." Lao Z said. "The success path is a request that completes and returns an answer; anyone can test that. The failure path is a request cancelled midway — whether the state is right — half as many people test that. Test cancellation once and it beats testing success ten times: the success path only proves 'it isn't broken'; the failure path proves 'when it breaks, we know how it breaks.' Every symptom in the troubleshooting handbook deserves a matching reproduction test on the checklist."

Back to the book's overall goal: what this checklist ultimately builds is a repeatable verification habit — after finishing the book, you can reproduce the evidence on your own, without relying on the book's conclusions.

Xiao A folded the checklist and tucked it into the book. He knew the checklist's destination wasn't the ticks on its last page, but the next time he changed a prompt, swapped a model, or added a tool, and automatically remembered that line: "any path you touched must have a matching verification receipt." Hands-on verification has no graduation — only re-run after re-run.