Part II Chapter 11: A New Exam Hall Every Time — pi-evals

Chapter guide: Stop ⑩ on the source route (the evaluation layer). We look at the pi-evals package and read src/harness.ts, src/plan.ts, src/report.ts, and the cases under evals/. This stop throws a bucket of cold water: "it ran" is not "it got better". See how the eval harness uses a temporary environment and artifacts to leave re-checkable evidence, where faux tests and real-model tests differ in evidence level, and how the "with docs versus without docs" controlled experiment keeps an accident from being read as a conclusion.

Xiao A finally got his own Agent running and let out a long breath: "It works! This time it should be fine, right?"

"Running once and running well are two different things," Lao Z said. "Protocols and remote state can be tested, but 'did the model do a good job' cannot rest on one successful run. In this chapter we follow the pi-evals harness to see how the environment is isolated, how the model is chosen, and what a judge can actually prove. Do not read eval scores as a universal capability ranking."

All source evidence in this chapter comes from the upstream pi tag v1.0.0 (commit a13d35a7, packages/evals/, package version 1.0.0, private: true). v1.0.0 reorganized this package: runner code lives in src/, cases and fixtures in evals/, image build files in docker/. README.md splits the cases into two kinds: *.docs.eval.ts are documentation-lift evals, and the remaining *.eval.ts are ordinary host evals that run on your own machine.

The harness sets up one isolated run

"Start with the run environment." Lao Z opened packages/evals/src/harness.ts: "createPiCodingAgentHarness wraps a harness using createHarness from vitest-evals/harness, and the run callback enters runPiCodingAgent. Every run creates a temporary root directory with mkdtemp(join(tmpdir(), "pi-eval-")), and inside it places a separate workspace, home, agent directory, and session directory."

// packages/evals/src/harness.ts, runPiCodingAgent (v1.0.0, excerpt)
const root = await mkdtemp(join(tmpdir(), "pi-eval-"));
const workspace = join(root, "workspace");
const isolatedHome = join(root, "home");
const agentDir = join(isolatedHome, ".pi", "agent");
// …
const restoreEnvironment = applyIsolatedEnvironment(isolatedHome, agentDir);

Metaphor: a new exam hall every time

A temporary cwd is like changing exam halls for every attempt: the last one's scratch paper does not stay on the desk.

"applyIsolatedEnvironment temporarily points HOME, USERPROFILE, and PI_CODING_AGENT_DIR at this root directory, and first deletes every environment variable starting with PI_EVAL_, restoring all of them when the run ends," Lao Z said. "A case can also pre-place fixture files with workspaceFiles; seedWorkspace rejects absolute paths and relative paths that escape the workspace."

Xiao A guessed: "Then the temp directory locks the model service in as well, right? During a test nothing touches the outside?"

"The model service is exactly off that list." Lao Z pointed at the try block. "It first reads the stored credentials for the selected provider from the host agent directory's auth.json, puts them into an InMemoryCredentialStore, and uses that to call ModelRuntime.create. So the temporary directory isolates the workspace, the configuration, and the session; credentials are deliberately brought in from the host, and model requests still go over the real network. What this isolation suppresses is the contamination of test cases sharing a working directory — it is not the same thing as a run having no side effects."

"There is one stricter layer of isolation that only turns on inside a container," Lao Z continued. "When PI_EVAL_SANDBOX_UID and PI_EVAL_SANDBOX_GID are set, runPiCodingAgent first deletes the host's auth.json, hides the credential environment variables from process.env, and then calls enterToolSandbox: it requires the process to start as root, chowns the temporary root directory to the sandbox user, drops privileges permanently with setgid/setuid, and finally self-checks that after the drop it can no longer read the Vitest-transpiled case module. The comment and the README say the same thing: after the privilege drop, the case definitions, the judging helper code, and the fixtures are unreadable to the Agent."

Xiao A: "What about cleanup? In what order does it delete?"

"The order is fixed, and it has to line up with the order of artifact collection." Lao Z read on down the source. In the finally block it first reads sessionManager.getSessionFile(): no session file at all counts as one cleanup error, 'Pi eval produced no session file.'; if there is one, the whole JSONL is stored via setArtifact(PI_SESSION_SNAPSHOT_ARTIFACT, ...). Only then session?.dispose(), then rm(root, { recursive: true, force: true }), and last the environment variables and the hidden credential variables are restored. Artifact reads must come before rm — once the JSONL is deleted it cannot be read again. Each step has its own try/catch, and every error is pushed into cleanupErrors; even if dispose throws, rm still runs.

end of try block
 → read session JSONL into the artifact (a missing file is itself recorded as an error)
 → session.dispose()
 → rm(root, recursive)
 → restore environment variables and credential variables
 → merge runError with cleanupErrors, decide to throw or return

"runId is recorded right after the session is created, through setArtifact("runId", sessionManager.getSessionId())," Lao Z added. "A failed run still leaves the runId trail; it does not go completely silent because of rm."

Xiao A: "When a cleanup error and a failed run collide, how does the report merge them?"

"Two kinds — don't mix them up." Lao Z went back to the tail. "Run succeeded but cleanup failed: a single error is thrown directly, several are wrapped in AggregateError("Agent cleanup failed."). Run failed and cleanup also failed: it is wrapped together with the run's error into AggregateError("Agent run failed and cleanup also failed."). If a transcript and usage had already been collected before the failure, attachHarnessRunToError also hangs that partial result onto the error. The reason a run failed is always the first-hand evidence; a cleanup failure is only extra information."

createPiCodingAgentHarness → createHarness (vitest-evals)
   │ run callback
   v
runPiCodingAgent
   ① signal.throwIfAborted / resolveModelSelection / mkdtemp / isolate environment variables
   ② host credentials → InMemoryCredentialStore → ModelRuntime.create → getModel / getAuth
   ③ (inside a container) delete auth.json, hide credential variables
   ④ createAgentSessionServices → SessionManager.create → setArtifact(runId)
   ⑤ createAgentSessionFromServices → check for no unexpected extensions
   ⑥ (inside a container) enterToolSandbox drops privileges
   ⑦ stepwise prompt / reload; signal abort bridged to session.abort()
   ⑧ toTranscriptEvents → usage → verifySystemPrompt → output()
   ⑨ finally: read JSONL → dispose → rm(root) → restore the environment
   ⑩ merge errors, or return the result with timings

The boundary between fake and real models

Xiao A: "Which model does a test use? Just pick one?"

"You can't pick casually," Lao Z said. "resolveModelSelection requires an explicit model, or both PI_PROVIDER and PI_MODEL set; afterwards, if the model cannot be found, or the model has no configured authentication, it fails outright."

// packages/evals/src/harness.ts, resolveModelSelection (v1.0.0)
const provider = (explicitModel?.provider ?? environment.PI_PROVIDER)?.trim();
const id = (explicitModel?.id ?? environment.PI_MODEL)?.trim();
if (!provider || !id) {
	throw new Error("Select a harness model explicitly or set both PI_PROVIDER and PI_MODEL as defaults.");
}
return { provider, id };

Xiao A jumped in: "Then what about faux? I'd guess it's just a fake model for show — no real run?"

"No run." Lao Z opened packages/ai/src/providers/faux.ts. "createFauxCore maintains a pendingResponses queue; when stream is called it shift()s out one pre-loaded response and uses streamWithDeltas to cut it into small chunks that simulate a stream. withUsageEstimate estimates tokens by dividing character length by 4, so the usage is an estimate, not real billing. When the queue is empty it produces one error message whose text is 'No more faux responses queued'. faux tests control flow and event sequences; it never touches model reasoning."

Xiao A: "Then why doesn't the harness just default to faux?"

"Because a faux pass does not mean a real model passed," Lao Z said. "resolveModelSelection accepts no implicit default: it would rather let the test fail than let you mistakenly believe you tested a real model. faux is used in pi-coding-agent's unit tests, for example test/suite/harness.ts; packages/evals contains no faux reference at all — everything goes through ModelRuntime.create() plus the real getModel(provider, id). The two paths correspond to two evidence levels: unit tests prove control flow, pi-evals proves the real chain."

Key point: two evidence levels

A real model is affected by server-side version, sampling, rate limits, and price; a faux provider can stand in for deterministic unit tests, but cannot prove real API compatibility or quality. Conversely, a real-model smoke test proves that one actual chain worked, but cannot prove the correctness of all prompts.

When does a prompt count as "complete"

Xiao A: "What is promptAgent's standard for calling a run successful?"

"Three hard failures." Lao Z opened promptAgent. "If no new assistant message is found when the run ends, it throws 'Agent run completed without an assistant message.'; if stopReason is neither stop nor toolUse, it throws with errorMessage or a fallback text; if it ends with stop but there is no text, it throws 'Agent run produced no assistant text.'. error, aborted, and length all make the run fail."

Xiao A pressed: "That strict? What if the product itself wants to accept truncated output?"

"Then this path reports a false failure," Lao Z said. "These checks prove that 'this run finished strictly', which suits a regression condition; 'output truncated but still partly useful' is a separate product policy, and this does not cover it."

"There are also two self-checks, one around the session and one after the run," Lao Z continued. "Once the session is built, getExtensionPaths() must hold nothing besides the inline extension the harness injects itself, or it throws 'Isolated eval loaded unexpected extensions'. After the run, verifySystemPrompt checks that the system prompt still contains a <rules> section, and that the presence or absence of the <docs> section matches this variant. The first keeps host extensions out; the second keeps the two variants of a controlled experiment from quietly not being separated at all."

Xiao A thought of the interruption case: "What about the signal?"

"The entry point calls signal?.throwIfAborted() first, checks again after the services are built, and then bridges abort to session.abort()," Lao Z said. "The Promise returned by abort() is stored in abortPromise and awaited in the inner finally. So the effect of an interruption is: the current prompt throws, wait until the session actually stops, then do artifact collection and rm; a signal cannot skip cleanup."

Xiao A: "When is the cost in usage computed?"

"It is decided against the model's cost table, not filled in blindly," Lao Z said. "estimatedCostUsd goes into the metadata only when the model declares a non-zero cost; a zero-cost model does not get a fabricated 0-dollar bill. provider/model come from the Model the runtime returned, and the token and tool-call counts come from session.getSessionStats(). That is why evals/smoke.eval.ts can assert that result.usage's provider and model equal the environment variables, proving this run really used the specified model."

Stage	Evidence produced	Normal path	Error / boundary
Model selection	provider / model	explicit, or both environment variables	missing, not found in the catalog, no auth → fail
Session	temp cwd, session, runId	prompt completes	unexpected extension, signal triggers abort()
Result	output, events, usage, system prompt summary	ends with stop or toolUse	no assistant, no text, unexpected stop reason
Teardown	session artifact	dispose, recursive delete of root	a missing session file is also an error; cleanup errors merged into the report

The controlled experiment: with docs and without docs

Xiao A: "What are *.docs.eval.ts?"

"That is a class of eval newly added in v1.0.0. The question it asks is: does showing the Agent pi's own documentation raise the completion rate?" Lao Z said. "createPiDocumentationEvalHarness can only be used inside the container sandbox; if PI_EVAL_CONTAINER is not 1 or there is no sandbox identity, it throws immediately. The variant is decided by PI_EVAL_VARIANT: without_docs uses excludePiDocumentation to delete the <docs> section from the default system prompt, with_docs keeps it as is. By default the tools are only read, write, edit, grep, find, ls — no shell."

"src/plan.ts expands the cases into tasks." Lao Z opened createTaskPlan. "Each case, each repetition round, produces two tasks, one per variant; odd rounds run without_docs first, even rounds run with_docs first — the README says this reduces order bias. By default each variant runs only once; to test stability you have to explicitly add --runs-per-variant."

"src/report.ts then pairs the two results that share the same case, the same model, and the same round," Lao Z said. "The README is blunt about it: a pair counts toward the pass-rate lift only when both sides get exactly one score; missing, duplicated, skipped, unscored, or errored, and the pair is blocked. If even one pair is blocked, the overall pass rate for that eval set is not published and the process exits with a non-zero code. The report also flags no-lift, negative-delta, control or experimental arm saturation, and flaky."

"One repetition cannot prove stability," Lao Z emphasized. "The README's exact words are 'One repetition cannot establish stability.' The controlled-experiment design exists so that you do not treat one accidental pass as evidence that documentation helps."

The README also records one boundary honestly: model requests still need the container to have network access, so Docker alone cannot prove that arbitrary code the Agent writes never touches the network.

What a judge can show

Xiao A: "What exactly does the harness hand to the judge?"

"The final output, the transcript events, the token and tool usage statistics, and the session snapshot artifact." Lao Z unfolded toTranscriptEvents:

// packages/evals/src/harness.ts, toTranscriptEvents (v1.0.0, excerpt)
events.push({
	type: "tool_result",
	toolCallId: message.toolCallId,
	name: message.toolName,
	content: message.content.every((part) => part.type === "text") ? text : toJsonValue(message.content),
	...(message.isError ? { error: { message: text || "Tool failed" } } : {}),
});

Xiao A wanted to see a concrete one: "What does a judge look like?"

"Look at evals/extensions.docs.eval.ts." Lao Z turned to it. "It has the Agent first write an extension with a hello tool, reload, then use that tool to greet Bob. The output callback pulls four facts fresh out of the session at the end of the run: the final answer, the extension load errors, whether the hello tool is loaded, and the last successful hello tool result. Then it uses two ready-made judges: StructuredOutputJudge requires those four fields to equal the expected values exactly, and ToolCallJudge requires hello({ name: "Bob" }) to have appeared in the trace."

"Notice judgeThreshold: null," Lao Z said. "The README's account is: when scoring is used for comparison, a low score is data, not an infrastructure failure; Vitest assertions are reserved for 'the case itself is broken'. So a judge's result goes into the report and participates in pairing, instead of hanging the test."

Xiao A: "Then can I assert that the model 'understood the requirement'?"

"No," Lao Z said. "A single output cannot prove that the model 'understood the requirement', that no edge case was missed, that it is safe on all inputs, or that future model versions will keep the same quality. What an assertion can prove is format, expected files, a tool-call trace, or a set of explicit facts. For open-ended quality, keep the inputs, the model version, the prompts, the judging rules, and human sampling, and report behavioral regressions separately from subjective quality."

Xiao A: "Can a tool result in the transcript be treated as evidence that 'the tool really executed successfully'?"

"Separate two layers." Lao Z pointed back at toTranscriptEvents. "It attaches the error field only when message.isError is true. isError: false is not the same as the tool's side effect succeeding — it only means the tool returned a non-error result. The transcript is 'the tool-call trace visible to the model', not proof of 'the real state of the external world'. To prove a file really got written, read that file inside the output callback; extensions.docs.eval.ts checking whether the extension actually loaded is exactly that idea."

Evidence	Where it comes from	What it can prove	What it cannot prove
The object output returns	the output() callback	format, fields, facts read fresh inside the callback	the reasoning process, inputs not covered
transcript events	toTranscriptEvents(session.messages)	message sequence, tool call arguments and results, the isError flag	that a tool's side effect really succeeded, model quality
session snapshot artifact	the whole JSONL from getSessionFile()	replaying inputs, reproducing edge cases, auditing the run	the current state of the external world
the paired report	report.ts	the pass-rate difference between two variants over the same batch of tasks	stability under few repetitions

"The difference is how far each one stands from the scene," Lao Z explained. "output is an assertion in code you wrote yourself, closest to the result; the transcript is the trace inside the model's view; the artifact is the raw JSONL, and the later it sits the better it reproduces; the paired report is statistics built on top of all these. But none of them can prove 'the real state of the external world'. Turning a score into a quality conclusion still takes human sampling."

Summary

The core duty of an eval harness is to build re-checkable evidence for one Agent run. pi-evals uses mkdtemp to create a separate temporary root directory for every run, puts the workspace, home, agent directory, and session inside it, and temporarily rewrites the related environment variables; at the end it saves the session JSONL first, then disposes, deletes recursively, and restores the environment. Credentials are deliberately brought in from the host and model requests go over the real network, so this reduces contamination between runs — it is not the same as having no side effects. Container mode goes further: hide the credentials, run with dropped privileges, and make the cases and the judging code unreadable to the Agent. Model selection must be explicit; if the model or its authentication is missing, the run fails rather than choosing implicitly.

This chapter kept returning to the difference between evidence levels. A faux provider can stand in for deterministic tests and prove control flow, but cannot prove real API compatibility or quality; a real-model smoke test proves that one actual chain worked, yet is subject to server-side version, sampling, and rate limits. The documentation-lift eval uses paired tasks, alternating order, and the rule that one blocked pair suppresses the overall result, to stop an accident from becoming a conclusion — though the README also states plainly that one repetition cannot prove stability.

Isolation and evidence are two sides of one thing. The temporary directory buys cleanliness at the price of deleting the scene: afterwards you can only go back through the artifact, the runId, and the facts explicitly retained in the output callback. That is why the cleanup order is fixed as "read the JSONL first, then dispose, then rm", with each step in its own try/catch, and run failures reported separately from cleanup failures. Between faux and real models, between transcript and external side effects, between scores and quality, there is a clear evidence boundary in each pair.

Source Walkthrough

1. Read mkdtemp, applyIsolatedEnvironment, and the cleanup and artifact-saving paths in src/harness.ts, and write down at which step the environment variables are restored.
2. Remove PI_PROVIDER/PI_MODEL before running, and verify that model selection fails rather than selecting implicitly.
3. Read promptAgent and list the three situations that make a run fail.
4. Read evals/smoke.eval.ts and list what it can prove and what it cannot.
5. Read createTaskPlan in src/plan.ts, and write out the four tasks one case expands into with --runs-per-variant 2, together with their order.
6. Find resolvePair and publishHeadline in src/report.ts and explain under what conditions the overall pass rate is hidden.