Troubleshooting Handbook for Part III · Build It: Why Agents Fail

Xiao A's agent ran overnight. The early-morning log held three entries: a model stream cut off mid-way, a tool calling the same file over and over, and state that didn't line up after a session restore. He stared at the screen, not knowing which one to chase first.

"Don't panic, and don't guess entry by entry," Lao Z said. "Use this as a troubleshooting handbook: read the symptom first, then find the evidence, then verify the root cause. Never the other way around. This appendix serves Part III · Build It. Consult it when something goes wrong while you implement or verify the minimal agent. Each entry lists symptom, evidence, root-cause hypothesis, and verification steps separately, so you keep whatever is reproducible first. The source locations are leads for your investigation, not product guarantees."

"Why separate evidence from root cause first?" Xiao A asked.

"Because a root cause can lie to you. Evidence doesn't," Lao Z said. "The symptom is 'it broke'. The evidence is 'which line it broke on, and what state it carried'. The root cause is your interpretation of the evidence. The same evidence can support two root causes, and one of the interpretations will be wrong. But evidence is the key to reproduction. Record the evidence before you discuss root causes, and the only thing you throw away later is a hypothesis. Do it in reverse, and you throw away the whole investigation."

"One more general principle: verify one hypothesis at a time," Lao Z said. "Test two together, and even success won't tell you which one earned it. Change one thing, reproduce once, look at the result — that is the same discipline as controlling variables in a lab. Troubleshooting rewards whoever deletes hypotheses most cleanly."

Model requests fail or get cancelled

Symptom: No final answer, or a visible cancellation. Evidence: Record the provider, model ID, AbortSignal, stop reason, and the raw error. Root-cause hypothesis: Authentication not configured, a network failure, the model returning an error, or the user cancelling. Verification: Reproduce it first with a minimal request that calls no tools; check ModelsImpl.applyAuth, streamSimple, and whether the caller passes the same signal all the way down. Never treat a cancellation as a successful empty response.

On the mechanics, one warning about the pit people fall into most often. Lao Z said: "AbortSignal is designed so that one signal runs the whole length of the call stack. If any layer in between fails to pass the signal down, then when you abort at the top, the innermost request doesn't stop — it keeps running. You think the cancel worked. In fact nothing got cancelled, and the request is still burning money." What Xiao A has to check is whether the signal travels all the way down the chain from the caller to streamSimple, not just whether the outermost layer aborted.

Xiao A: "If even the minimal request can't reproduce it, the problem is somewhere else?"

"Then look downstream," Lao Z said. "If the minimal request passes, the model layer is fine and the problem is in the assembly layer: a tool description so long it blows up the request, bad data stuffed into the context, or a message structure the provider won't accept. The minimal request is a watershed. Problems before it belong to the model layer; problems after it belong to the assembly layer. It lets you cross out most of the 'the model is broken' guesses."

"There's another class of failure people miss," Lao Z said. "The model returns an error, but the stop reason gets recorded as complete. On a partial failure, some providers emit the error event first and the end event after it. Read only the last stop reason, and one failed run gets scored as a success. So the evidence must hold the raw error and the stop reason together. When the two disagree, information was lost inside the adapter."

"For the 'no final answer' symptom, besides failure and cancellation, is there anything else?" Xiao A pressed.

"Yes, and it hides deep: the stream stops and nobody knows," Lao Z said. "The connection reports no error, the signal never fires, and delta events simply stop coming. It may be a provider disconnecting silently, or an adapter swallowing the end event. This neither-success-nor-failure state is the most grinding, because it isn't on any error path. Debug it with a heartbeat: if no event arrives for a set window, declare the timeout yourself instead of waiting forever. A wait without a timeout isn't waiting. It's hanging."

"Timeouts and cancellations use the same mechanism, right?"

"No — don't conflate them," Lao Z said. "A cancellation is an outside party actively asking to stop: the user pressed cancel, or a higher layer sent an abort. A timeout is a passive death sentence: no activity inside the agreed window. The evidence differs too. A cancellation leaves the trace of a signal firing; a timeout leaves only timestamps. Keep the verification separate as well: test cancellation with AbortSignal, test timeout with a short timeout parameter. Which kind of failure it is decides which branch's code you read."

Repeated tool calls and write conflicts

Symptom: The same tool executes repeatedly, or edits to the same file are lost. Evidence: Keep the tool call ID, a digest of the arguments, the tool result, the turn number, and the file path. Root-cause hypothesis: The description or error result is too weak for the model to change strategy, or concurrent writes hit a shared resource. Verification: Set a turn budget; make tools return failures explicitly; check that withFileMutationQueue serializes writes on the same path, rather than assuming every tool call is safe to run concurrently.

Lao Z kept the two root causes apart: "Repeated execution and write conflicts look like one symptom, but their roots sit in two layers. Repeated execution is mostly a model-layer problem: the failure message is too vague — say it just returns ‘Error’ — so the model doesn't know why it failed and retries with different arguments, staying on the wrong track however it changes them. Write conflicts are an execution-layer problem: multiple tool calls land on the same file at once, there's no serialization queue, and the later write overwrites the earlier one. The fixes stay separate too. For the first, put a more specific failure reason into the tool result. For the second, use withFileMutationQueue to chain writes on the same path. Neither layer covers the other's back. However clear the failure message, it won't stop concurrent trampling. However tight the queue, it won't stop a model guessing blindly."

Xiao A: "In tonight's log, how do I tell which one it is?"

"Look at the timestamps," Lao Z said. "Repeated execution is serial: the same tool shows up again and turn after turn, each turn slower than the last, until it hits the turn limit. Write conflicts are concurrent: two tools touch the same file at almost the same instant, and their order in the log is random. Serial means the model layer; concurrent means the execution layer. When you can't tell, check concurrency first — concurrent problems are harder to reproduce, so locking one down early is worth more."

"Repeated calls can also hide in the tool description," Lao Z added. "If the description doesn't say when the tool should be used and when it shouldn't, the model treats it as a cure-all and keeps reaching for it. That isn't a bug, but the symptom is identical. Read the description before you dig through the log — it can save a night of archaeology."

Xiao A jumped in: "Then put every write behind a queue and the conflicts disappear, right?"

"Before the queue, split once more: concurrent writes, or sequential overwrite," Lao Z said. "Concurrent writes are two tool calls landing at the same instant; serializing them with withFileMutationQueue fixes that. Sequential overwrite is the model itself writing the same file twice in a row, the second write based on stale content from the first. That's a model-layer problem, and a queue won't help. Timestamps separate them: two writes appearing almost together means concurrent; writes spread across several turns means sequential. The two fixes are completely different."

"What people miss easiest is inside a single turn," Lao Z said. "One model request can produce several tool calls. The host runs them all in parallel, and if they share a resource, the conflict blows up inside that turn. When you investigate, check whether the multiple tool calls in one turn shared a file or state. If they did, either execute them serially or draw a clear boundary between them. Parallelism isn't always what the model asked for — the host may be manufacturing the concurrency itself."

Context too long or compaction failure

Symptom: The request exceeds the window, or a summary drops key constraints. Evidence: Save the messages before and after compaction, the token estimate, the retained boundary, and the error text from the failed compaction. Root-cause hypothesis: A large tool output entered the context, or the summarizing model was aborted or failed. Verification: Read prepareCompaction, compact, and getSummarizationFailure in packages/coding-agent/src/core/compaction/compaction.ts; test separately whether, on a summarization failure, the original history is preserved and reported to the user.

Lao Z told Xiao A to split "compaction failure" into two kinds first and never investigate them as one: "One kind is compaction that can't happen: the summarizing model crashed or was aborted. The other is compaction that happened but happened wrongly: the summary dropped key constraints. The first is a runtime error — the summary request's stop reason and error text locate it. Both error and length, truncated by the token cap, will fail the compaction. The second is a semantic error. The log looks perfectly clean, and from the next turn on the model simply forgets the hard constraints you gave it. To investigate the second, diff the messages before and after compaction and check whether the retained boundary compacted away the system prompt or early user constraints as well."

"For the 'compacted but compacted wrongly' kind — how bad can it get at worst?"

"As bad as looking normal," Lao Z said. "No error, no interruption. The request goes out as usual, and the model just works every turn in a state missing constraints. The dangerous part is that no signal at all says it happened. The only remedy is to keep the pre-compaction history, even if it's only one copy, so that after the fact you can compare 'the constraints the model thinks it has' against 'the constraints you actually gave'. Without a pre-compaction copy, this error can never be found. So the 'messages before and after compaction' column in the evidence isn't a suggestion — it's a requirement."

"Prevention starts at the source," Lao Z said. "First find out what blew the context up. Nine times out of ten it's one large tool output: read on a file of tens of thousands of lines, or list on a giant directory. Compaction cleans up the mess; controlling large output is what keeps the mess from appearing. Do both. The first treats the symptom; the second treats the cause."

"Token estimation has its own subtlety," Lao Z said. "'Estimating by characters' and 'counting by tokens' are two different budgets, and you can't mix them. Some implementations count characters roughly — cheap, but it can overestimate or underestimate. Others count tokens precisely — accurate, but it costs an extra counting call. When you debug, confirm your own budget unit first. Apply a character budget against a token window and your conclusions will never line up. Write the unit into the evidence; it is far more reliable than 'I think it's about right'."

"The trigger point for compaction is itself tunable," Lao Z continued. "Compact at 80% of the window, or only when you hit the wall? An earlier threshold puts less pressure on each compaction. Too late, and the history is already huge when compaction runs, so the summary burden grows as well. That threshold balances wasted context against the risk of losing information to the summary. It's a parameter you tune to the task, not a default you inherit." Xiao A wrote the number "80%" in the margin of the handbook.

State is wrong after a session restore

Symptom: History is missing after a restart, or file state after a fork differs from what you expected. Evidence: Save the JSONL entries, the session ID, the fork point, and the filesystem state. Root-cause hypothesis: Session entries were mistaken for messages, or a session fork was mistaken for a Git worktree. Verification: Compare against fork in packages/coding-agent/src/core/session-manager.ts and agent-session-runtime.ts; check the Git / workspace isolation mechanism independently.

Lao Z corrected a common misunderstanding first: "The JSONL file holds a message stream, not executable state. On restart you get a replay of past messages, not the in-memory state of a tool call that was half-executed in the last turn. So after a restore, if an unfinished side effect is still dangling — a tool call that sent a network request whose result never came back — that state can't be recovered. You have to treat it as 'unknown', not as 'it didn't happen'. Mistaking a session fork for a Git worktree is another common misalignment: a fork copies the session history, not necessarily the workspace files. Check the isolation boundary of each separately."

"'Treat it as unknown instead of as it didn't happen' — what does that look like operationally?"

"After a restore, take an explicit inventory of side effects first," Lao Z said. "At startup, list the write operations that may still be pending and ask about each one: is last run's temporary file still there? Did the result of the request we sent come back? Then either complete it, roll it back, or mark clearly 'the state of this branch is unknown and needs human confirmation'. The worst habit is pretending nothing happened and just continuing. That is letting the agent keep acting in a world it believes is clean, while the world still has half-finished operations hanging off it."

"Session entries come in so many types — how do you decide which ones are worth reading while debugging?"

"Sort them by restore semantics," Lao Z said. "Message entries — user, assistant, tool — decide what the model sees. Configuration entries decide how tools and models are wired up. Compaction entries decide how much of the old history survives. When state looks wrong after a restore, ask one question first: is what the model sees wrong, or is what we assembled wrong? For the former, read message entries; for the latter, configuration entries. Use entry types as signposts instead of reading every entry from the top."

Remote UI out of sync with the server

Symptom: The progress display disagrees with the final transcript, or after a disconnect the request's outcome is unknown. Evidence: Save the request ID, attachmentId, the subscription snapshot, each service_update received, and the connection close reason. Root-cause hypothesis: Treating stitched increments as the authoritative state, not re-subscribing after a reconnect, or retrying blindly while the commit state is unknown. Verification: Re-subscribe and rebuild the view from a fresh snapshot; read the subscription branch in packages/server/src/server.ts and requireAttachment in session-router.ts. Do not claim that this snapshot of the implementation auto-reconnects or delivers exactly-once.

Xiao A: "Progress and the transcript disagree. Is that a UI bug or a server bug?"

"Don't assign blame yet," Lao Z said. "Ask first: which source computed 'the current state'? If the UI builds state by accumulating events one at a time, one lost event corrupts everything. That is the error of treating increments as authority, not a bug in any one render function. The correct source is the snapshot you get when you subscribe, with service_update events applied on top in order. After a disconnect, the client's subscription is cleared, so you must re-subscribe and take a fresh snapshot. Decide which one is the authoritative state first, then go find the rendering code."

"Disconnect behavior needs its own verification," Lao Z added. "If a request has already gone out and its result is still unknown, the interface must surface that state explicitly. 'Connection lost' and 'request outcome unknown' are two different messages: the first only says the connection is gone; the second says the request may not have completed. The root cause of many disconnect incidents is that the interface reported only the first. The user, or the logic above it, read 'no result' as 'it didn't happen', while the side effect had already executed. Mark unknown as unknown. Never mark it as not-happened."

"Reconnecting has discipline too: you may retry, but you may not retry blindly," Lao Z said. "You don't know whether the last commit succeeded, so a retry can produce a duplicate side effect — a duplicate message, a duplicate charge. Before reconnecting, settle whether the last commit actually succeeded. For a state you can't settle, prefer human confirmation over automatically doing it again. The precondition for retry is an idempotency guarantee, not a default setting."

"So between increments and snapshots, which do you trust?"

"Start from the snapshot," Lao Z said. "A snapshot is the complete state at a moment; increments only mean something on top of it. Reverse the order, and after a disconnect the UI will treat incomplete increments as the truth. The reason you record the post-reconnect snapshot in the evidence is to answer, after a disconnect, 'where did the state I'm looking at now come from?'" Xiao A copied this line into his troubleshooting checklist.

Protocol input rejected

Symptom: The decoder reports ProtocolValidationError. Evidence: Record the frame length, the connection phase, and the truncated error. Do not record the payload itself, which may hold secrets. Root-cause hypothesis: Length over the limit, corrupt CBOR, a schema mismatch, or a half-frame at EOF. Verification: Reproduce with ClientMessageDecoder; check FrameDecoder, decodeCbor, and the schema. After a failure, create a new decoder and new connection state instead of continuing to push bytes into it.

Xiao A asked tentatively: "The decoder errored — can't you just retry?"

"No. After a decoder fails you must rebuild it; you cannot keep feeding it bytes," Lao Z said. "A protocol decoder is stateful. Internally it tracks which frame it's on and how many bytes of the length prefix it has read. Once it errors, that state is no longer trustworthy, and pushing more into it only carries the corruption into every later decode. The correct move is to discard the old decoder, build a new one, and reset the connection state. This matters most for half-frames: at EOF you may have read only part of one. A half-frame is not an error — it's an incomplete operation. But once the decoder has classified it as an error, the only recovery path left is to rebuild."

"Of the four root-cause hypotheses, how do you tell which one it is?"

"Look at which stage the error happened in," Lao Z said. "Before FrameDecoder is the length layer: over-limit lengths and half-frames at EOF live there. Between FrameDecoder and the schema is the encoding layer: corrupt CBOR lives there. After schema validation is the semantic layer: a message with the wrong shape lives there. The log should say which layer threw the ProtocolValidationError. Locate the stage first, then guess the root cause — that alone cuts the options in half."

"Security adds one hard requirement: the evidence never records the payload itself," Lao Z said. "Frame contents may carry keys, user data, or an entire prompt. What troubleshooting needs is 'how long it was, which layer broke, did it truncate' — not 'what was inside it'. Logging sensitive content to chase one protocol bug trades one bug for another."

Runaway cost or a surprise bill

Xiao A hit this in the first week of Part III · Build It. The model kept calling the same tool inside the loop, one task burned through hundreds of requests, and the bill multiplied. Lao Z asked only one question at the time: "What did you set maxTurns to?"

Symptom: Token usage or cost for a single task far exceeds expectations. Evidence: Record tool calls per turn, input and output tokens, stop reason, and total turn count. Root-cause hypothesis: There is no turn budget, or the tool's failure information isn't specific enough for the model to change strategy, so it retries the same action over and over. Verification: Check maxTurns in runAgent. Check whether the tool's error results are specific enough ("file not found: src/a.ts" is likelier to make the model change paths than "Error: ENOENT"). Add logging on the loop's token usage instead of reading only the total bill.

"Why isn't the total bill enough?"

"Because the total bill doesn't tell you which stage spent the money," Lao Z said. "The same triple cost could be an infinite loop, which is a turn problem. It could be context growth, with every message resending the whole history. It could be cache misses from an unstable prefix. Or one tool could produce enormous output, where a single tool result costs more than a hundred turns. Those four root causes need completely different fixes, yet the total bill looks identical for all of them. Record tokens per turn, and you can localize 'money on fire' to 'which turn, which tool, which stop reason'."

"Retry itself is also easy to underestimate," Lao Z said. "Every attempt the model makes sends a full request again, and the longer the history, the more expensive each one. So failed turns deserve more attention than successful ones. A lot of runaway bill curves are the accumulation of failed attempts, not the cost of the task itself. Keep a separate ledger for failed turns and you'll see the burn much sooner."

Memory anchor: "a turn cap plus readable failures" are the two handles on cost control. Without maxTurns, however smart the model is, it will burn money in an infinite loop. If error messages aren't specific, the model can only guess.

The prompt cache never hits

Symptom: cacheRead stays at zero on the bill, or request latency doesn't come down. Evidence: Record the prefix structure of the request you send, the cache fields, and the provider's hit reporting. Root-cause hypothesis: The stable prefix changes often — for instance a timestamp, a random ID, or a project path spliced into the beginning of the system prompt — so the prefix can't be reused. Verification: Put stable rules (safety rules, output contracts) in the prefix and dynamic content (the task, the time) at the end. Check the provider's documented matching rules and validity window. Compare the usage fields across two requests with identical prefixes.

Xiao A: "Besides prefix order, what are the common ways to kill the cache? I'd guess timestamps."

"You got one. Four are most common," Lao Z said. "First, timestamps or random numbers: different on every request, so the prefix is dead. Second, project paths: run the same prompt on a different machine and the prefix forks. Third, tool definition order: change the tool list and the prefix changes with it. Fourth, tiny differences in message format — whitespace, newlines, quotes look irrelevant, but the hashes are completely different. A cache hit requires stability at the byte level, not stability of meaning. When you investigate, compare the raw bytes of two requests, not two prompts that look alike."

"Know the boundary too: a cache hit is provider service behavior, not your code's behavior," Lao Z said. "When it expires, at what granularity it matches, and how long a prefix it bills for are all set by the provider's documentation. The verification isn't 'I think it should hit'. It's sending two identical requests and checking whether the cache-related values in the usage fields changed. Test the provider's cache as a black box. Don't tune it like local configuration."

Memory anchor: the cache hits on a stable prefix. Change the prefix and everything after it is wasted. This isn't a cost-saving trick — it's a structural issue.

Troubleshooting principles

Xiao A worked through all these scenarios and asked Lao Z: "Is there one ordering that covers every investigation?"

Lao Z gave him four:

1. Read the stop reason first: complete / max_turns / aborted / error decides which path you investigate.
2. Reproduce minimally first: with the smallest request that calls no tools, is the problem still there? If yes, it's the model layer. If no, it's the tool or assembly layer.
3. Keep structure in the evidence: record provider, model ID, stop reason, and call ID. Don't record only "it errored".
4. Never treat a cancellation as success: aborted is not an empty response, truncation by length is not a complete answer, and max_turns is not a normal finish. Three states, three handling paths.

"Use these four together with the scenarios above," Lao Z said, "and most agent failures land in a layer: model, tool, loop, session, protocol — each in its own place."

"Is there a last resort in troubleshooting?"

"Yes, and the plainest one imaginable: write a reproduction script that pins the failure down," Lao Z said. "A crime-scene investigation can only be done once. A reproduction script can be run a thousand times, and after every change you run it to see whether the failure is still there. Every item in this handbook deserves a reproduction script. A debugging conclusion with no script behind it won't be believed a month later."

"One last reminder: keep the record of the debugging itself," Lao Z said. "Write down how you localized it this time, and which hypothesis which evidence killed. Next time the same failure comes, you may jump straight to the verification step. The time you save is the interest this record earns you."