Chapter 16: When There Is More Than One Route — Workflow & Graph
Xiao A had recently come across three things at once: the Agent loop he was writing himself, an "approval flow" a colleague owned, and an article about "graph orchestration". A little panicked, he took his coffee and went to find Lao Z.
"Do I have to pick one of these three, and is it over if I pick wrong?" he asked.
"Hold on. Retrieval decides what material comes along; it does not answer who decides the next step," Lao Z said. "The three things you are looking at — the loop, the workflow, and the graph — are not rivals waiting for you to choose one. They are three different arrangements of control and state. Work out what each one governs, and you will naturally know how to use it."
This chapter compares Loop, Workflow, and Graph. It commits to no particular framework, and it does not mistake a graphical representation for a reliability guarantee.
Aligning the Names First
| Term | Full English name or abbreviation | Working definition in this book | What it is not |
|---|---|---|---|
| Agent loop | Agent Loop | bounded iteration driven by model output and tool results | unbounded autonomous execution |
| workflow / graph | Workflow / Graph | fixed steps / nodes and edges with routing expressed | two strictly mutually exclusive categories |
| state machine | state machine | an explicit model of states, events, and transition conditions | just drawing a flowchart |
| checkpoint / replay / idempotency | checkpoint / replay / idempotency | the contracts for saving a recovery point, executing again, and avoiding repeated side effects | automatic transactional guarantees |
Three Arrangements of Control
"So who exactly decides the next step?" Xiao A asked.
"Look at this table and the question becomes clear. The key is who decides the next step:"
| Form | Who mainly decides the next step | The boundary it fits |
|---|---|---|
| Loop | the runtime iterates on model output and stop conditions | exploratory tasks, which must carry a budget and an abort |
| Workflow | the developer fixes the steps up front | approvals, ETL, predictable business processes |
| Graph | nodes and edges define the reachable routes, conditions pick the branch | multi-path, recoverable state flows |
"And the stop conditions — who is responsible for each form's stop?" Xiao A pressed.
"That is the part people overlook most easily," Lao Z said. "A Loop's stop condition must be given explicitly by the host: turn count, tokens, cost, or timeout. The model is not responsible for hitting the brakes. A Workflow ends at a defined terminal node — walk into it and you stop. A Graph stops when a node has no outgoing edge or a termination condition is met. Whichever form you choose, you have to be able to answer 'when does it stop, and what state is left when it does'. Otherwise recovery and auditing are out of the question."
"Is a Graph necessarily more advanced than a Workflow?" Xiao A asked.
"No. A Graph need not contain a model, and a Workflow can contain branches. The difference between them is in how they express things and how they manage state, not two mutually exclusive categories: a Workflow lays its steps out in order, a Graph draws branches explicitly as nodes and edges. The same business logic fits either one. The difference only shows up in what you touch when you change one branch: in a Workflow you edit the step definition, in a Graph you edit the routing condition."
"What if I pick the wrong form at the start — can I switch later?"
"You can, at different costs," Lao Z said. "Going from a Workflow to a Graph usually just means rewriting the fixed steps as nodes and transitions. Going from a Graph to a Loop hands the routing authority from your code to the model, so the stop conditions and guardrails all have to be redesigned. The work of switching forms goes almost entirely into moving the routing authority from one side to the other. So ask before you start: how fixed is the path in this task, really?"
"Whether the path is fixed — I read that as whether all the branches can be written out in advance?" Xiao A jumped in.
"Right. It is exactly whether the branches can be enumerated ahead of time," Lao Z said. "'Retry on failure, ask for missing material, finish when done' — if every outcome is written out clearly, the path is fixed, and a Workflow expresses it well enough. When the branches are numerous and the conditions themselves are only known at runtime, the path is not fixed. That is when a Graph or a Loop earns its place. What you are judging is whether the branches are predictable, not whether the task is complex."
"Can we line up a few common tasks against that?"
"Sure. Here are a few common tasks side by side:"
| Task | Are the branches predictable? | Fitting form |
|---|---|---|
| form approval (pass / return / reject) | yes | Workflow |
| release process (check → approve → deploy → verify) | mostly | Workflow / Graph |
| support tickets (route by type, then follow up on reply quality) | partly | Graph + bounded Loop |
| exploring a new feature (generate options → verify → iterate) | no | Loop + human confirmation |
"By that table, the same person can use all three forms at the same time?"
"Exactly. Different tasks inside one team often do use all three at once," Lao Z said. "Rather than sticking a 'form label' on every task, record clearly who holds the routing authority for each one."
Metaphor: the route map
A Workflow is a subway on a fixed schedule, a Graph is a map with transfers, a Loop is a taxi with no fixed route. None is better; it only depends on where you want to go.
The Engineering Profile of Each Form
"Beyond 'who decides the next step', is there any real engineering difference between the three?" Xiao A asked.
"There is, and the form choice usually hides it," Lao Z said. "Put observability, cost, failure recovery, testing, and debugging side by side and the differences are plain:"
| Dimension | Loop | Workflow | Graph |
|---|---|---|---|
| Observability | each turn's events can be logged, but the next step is not known | steps are known in advance, progress at a glance | nodes and edges are visible, the current position is queryable |
| Cost | every iteration incurs a model call | fixed steps, cost estimable | more branches, more test combinations |
| Failure recovery | resume from the stop reason; intermediate state may be lost | replay from the failed step, needs idempotency | resume from a checkpoint; the path can be rebuilt |
| Testing | needs many boundary input samples | each path tested separately | each node transition tested separately |
| Debugging | the problem may be buried in a model decision | the problem localises to a specific step | the problem localises to a specific node |
"Just from the testing row, the three have completely different things to prove?"
"Completely different," Lao Z said. "A Loop can go wrong at any model decision, so its exam is the stop condition and the boundary inputs — you need samples that cover 'did it stop when it should have'. A Workflow's exam is each fixed path: input A takes 1, 2, 3; input B takes 1, 2, 4; a path walked wrongly means a definition written wrongly. A Graph's exam is the transition conditions: from the same node, which edge to take when the condition holds, and which edge when it does not. One more thing — the number of test combinations is itself a hidden cost you pay when choosing a form."
"Is there a common pattern of building one form first and switching later?"
"Yes, and it is very common," Lao Z said. "Most systems start as a Workflow: fixed paths, testable, cost controllable. Only when task complexity climbs and the branches stop being enumerable do you turn parts of it into a Graph or a Loop. The direction of travel is usually from the deterministic toward the flexible; the reverse is rare, because the cost and the auditability the deterministic parts save you are precious assets you should not drop lightly."
State, Routing, and Recovery
"Whatever shape you draw it in, what should engineering actually care about?"
"Five questions: what the current state is, who may change it, where the branch conditions come from, where you recover after a failure, and whether the action is idempotent." Lao Z drew a simplified schema:
receive request → validate → [enough data?] ─no→ request material → validate
│yes
↓
execute task → record result"Why are those five questions questions about 'state' rather than about 'shape'?"
"Because the diagram is for people to read; the state is for the program to use," Lao Z said. "You can draw the same process any way you like, but the state has to land somewhere persistable — a database, a file, an in-memory object. Otherwise one process restart and the flow no longer knows where it got to. To judge whether an orchestration design is reliable, don't check whether the diagram is right; check whether each of those five questions has a place where it actually lands."
"What engineering part does each of the five questions correspond to?"
"One to one." Lao Z laid the five questions out as a table:
| State question | The engineering part behind it | What it looks like when missing |
|---|---|---|
| where the state lives | persistent storage (database / file / memory) | the flow is lost after a process restart |
| who may change it | write-permission control | several processes mutate the same state concurrently |
| where conditions come from | the branch logic (rules / model / external input) | the wrong branch is taken and cannot be traced back |
| where you recover from | checkpoint + a recovery entry point | after a failure you can only start over |
| is re-running idempotent | idempotency key / de-duplication logic | side effects repeat after a replay |
"That table is the checklist for whether an orchestration implementation is complete," Lao Z said. "Only when all five questions have an answer does the implementation close the loop. A question you cannot answer names the failure point: not that it might go wrong there, but that when it does, you have no means in hand to deal with it."
"What if the 'execute task' node in the diagram dies?"
"If 'execute task' writes to an external system, you cannot simply re-run it on recovery — record an idempotency key, or use a compensating action," Lao Z said. "The diagram draws the route; it does not automatically solve duplicate writes, racing updates, or model misclassification. Concretely: one node does 'create order' and writes to an external system, and the process crashes before the local checkpoint is written back. On replay, with no idempotency key, you get a second order; with the key, the external system recognises it as the same call."
"What is a compensating action?"
"Compensation is the action that 'gives back' a side effect that already happened: an order created, so you send a cancellation; an email sent, so you issue a retraction notice. But compensation is not rollback. The external system may already have executed, may have executed halfway, or may never have received the request at all. Compensation can only converge as best it can; it cannot guarantee you get back to the state before execution. So the design order is always: define idempotency first, then talk about compensation."
"Then does more uncertainty mean you must use a Loop?"
"Not necessarily. You can fence the uncertain part inside one node and keep the outer layer a deterministic workflow, so approvals, permissions, and delivery stay protected," Lao Z said. "The choice turns on predictability, the cost of failure, and observability — not on which one is smarter. For a fixed process whose failure is very expensive, even when a model takes part, you would rather lock the model's decisioning inside one budgeted node than let the whole process improvise."
"The system is already running. If I want to evolve it toward flexibility, where is the safest place to start?"
"Start with a local replacement," Lao Z said. "Don't overturn the whole Workflow. Pick one branch point and change its fixed decision into a budget-limited model decision. For example, where it used to say 'exceptions split into three fixed categories', make it 'the model classifies the exception first, then the original branches apply'. Replace one point at a time and leave the rest of the structure alone; the replacement point needs its own budget, stop condition, and fallback path."
| Evolution move | Risk | Mitigation |
|---|---|---|
| swap one Workflow step for a model decision | the model's judgement is unstable | constrain inputs and outputs, give a budget, set a fallback |
| turn fixed branches into Graph routing | routing conditions not fully tested | keep the original branches as a fallback first |
| allow free improvisation inside a node | the node runs away | cap the turns, force a stop, keep human confirmation as the backstop |
| replace the whole process with a Loop | you lose auditability | almost never taken; usually only local replacements |
"And the reverse? Too many model decisions and I want to take them back."
"Going backwards is simpler," Lao Z said. "Put a rule check in front of the model's output first; anything that fails the check returns to the fixed branch. Once the rules cover enough ground, remove the model decision entirely. Retreating from flexibility to determinism works by rules replacing the model step by step, and every step needs data showing the rules reach the quality of the model's judgement. You cannot cut by feel."
Key point
The reliability of orchestration comes from the answers to the five state questions, not from the flowchart itself: where the state lives, who may change it, where the conditions come from, where you recover from, and whether re-running is idempotent.
Hybrid Orchestration
"So can I mix the three?"
"You can, and it is common," Lao Z said. "One usual combination: deterministic routing picks the task type first, then the open-ended analysis goes to a Loop with a bounded number of turns. Or a graph node calls retrieval, tools, and models. Every layer needs its own input and output schema, timeout, budget, and audit events. Disguising one large Loop as a 'node' without exposing those boundaries only makes recovery and testing harder."
"So what gets mixed is the form; not one boundary may get mixed?"
"Right. Boundaries are exactly what you look for," Lao Z said. "Every segment of a hybrid orchestration has explicit inputs and outputs, a budget, and a stop condition. 'No orchestration' is the whole process smeared together — no segments and no boundaries. The test is simple: can you point at a segment and say 'this part runs at most 5 turns', 'this part has human confirmation', 'this part takes a fixed branch'? If yes, it is hybrid orchestration. If not, it is a mess."
| Trait | Hybrid orchestration | Muddled orchestration |
|---|---|---|
| segment boundaries | each segment has an input/output contract | segment responsibilities are fuzzy |
| budget | each segment has its own turn/token limit | no budget anywhere |
| stop conditions | each segment has an explicit stop reason | depends on the model behaving itself |
| audit | each segment has event logs | only the final result |
| recovery | each segment can resume from a checkpoint | a break means starting over |
"Isn't the maintenance cost higher than a single form?"
"It is — but higher where the higher cost pays off," Lao Z said. "What you maintain is no longer the logic of one form but the contracts of several segments, the switch points between them, and each one's recovery strategy. The cost is real. The return is that every segment is expressed in the form that suits it, and every segment can be tested on its own. If a segment's complexity does not justify maintaining it separately, do not open a new form for it. Most systems get by with one or two forms; a third is bought with extra complexity and cost."
"Draw me a hybrid structure?"
"Sure. Here is a simplified sketch:"
entry router (deterministic) ─── task type A → fixed approval flow (Workflow)
└── task type B → bounded investigation node (Loop, at most 5 turns)
↓
graph orchestration: pick a branch from the findings
↓
human confirmation (required)"In this structure each form owns one segment," Lao Z said. "The entry router and the approval flow are the deterministic parts; they handle authorisation and delivery. The investigation node is a bounded Loop; it handles the open-ended analysis. The graph chooses the next step once the findings are in. Every boundary is explicit: the Loop has a turn cap, the Workflow has a human gate, the graph has definite branch conditions."
"Why not just hand the whole task to a Loop?"
"Because the costs differ," Lao Z said. "Every extra turn of a Loop buys one more model call, a little more context cost, and a little less predictability in behaviour. Express the deterministic parts in a deterministic form: cheap and auditable. Give the model only the parts that genuinely need open judgement, and the budget goes where it counts. Hybrid orchestration, said plainly, is drawing a finite box around the uncertainty."
"So where should that box be drawn?"
"Wherever a failure would get amplified," Lao Z said. "Approvals, payments, permission changes — those fail expensively and irreversibly, so they belong inside the deterministic workflow, backed by human gates and auditing. Open-ended analysis, information gathering, generating options — those fail cheaply and can be retried, so they belong inside a bounded Loop. The measure is how much damage a failure here causes and whether it can be redone, not whether a model should be used here."
"Within one task, does the order of the deterministic and the open parts matter?"
"It does, and the shape is usually fixed: deterministic outside, open inside," Lao Z said. "The entry uses a deterministic router to split the traffic, the middle uses a bounded Loop to analyse, the exit returns to a deterministic process to close things out — the approval and the write-back happen in that last segment. The reverse, an open outer layer with a deterministic inner one, is rare: that lets the model decide the skeleton of the whole process, and the deterministic parts end up as the model's execution details."
Recovery Points Belong to State, Not to the Diagram
"Why do you say recovery points belong to state and not to the diagram?"
"Because a recovery point has to be readable back by the program. Drawing it on the diagram does not count," Lao Z said. "Take an expense-report process: 'generate draft' can be re-run safely; 'submit the report' must first check an idempotency key, or hand off to a human. Loop, Workflow, and Graph can be mixed: a deterministic workflow handles authorisation and delivery, the graph handles branching, and a turn-limited Loop handles only the open-ended analysis. You give up a little flexibility and buy a control plane that can be recovered and tested."
"Then what exactly does a node's definition have to spell out?"
"Check four things," Lao Z said,
The four definitions of a node
- Input: what data the node needs to start, and what happens when something is missing
- Output: the result the node produces on success, for downstream to consume
- Commit point: where the side effect lands — re-runnable before it, idempotent after it
- Idempotency strategy: what makes running twice different from running once — an idempotency key, querying external state, or handing off to a human
"The most common mistake is drawing the recovery point in the diagram while keeping it only in your head," Lao Z said. "The diagram says 'submitted', but the program has no field for it — one process restart and it is as good as never submitted. The recovery point has to be read out of persistable state, and that diagram cannot do it for you."
Design Recovery Before Side Effects
"Is there one view that covers all three forms?"
"There is. All three land on a state machine: states, events, transition conditions, plus a persistable checkpoint," Lao Z said. "A Workflow usually writes its transitions as ordered steps; a Graph records nodes and edges explicitly; a Loop delegates 'the next step' partly to the model, but a turn budget and a stop state are still mandatory. The difference is who owns the routing authority, not whether it can be drawn as a graph."
"How exactly are a state machine's states and transitions defined?"
"A state is 'where the process is right now'; a transition is 'what event happens and what condition holds for it to move from here to there'," Lao Z said. "Take the expense-report process; a minimal state machine looks like this:"
draft ──(submit)→ submitted ──(approve)→ approved ──(pay out)→ paid
│ │
└──(return)→ draft └──(reject)→ rejected| State | Meaning | Allowed transitions | Event that triggers them |
|---|---|---|---|
| draft | draft, editable | → submitted | the user submits |
| submitted | submitted, awaiting approval | → draft / approved | returned / approved |
| approved | approved | → paid / rejected | payment made / approval denied |
| paid | paid, terminal state | none | — |
| rejected | rejected, terminal state | none | — |
"The value of defining a state machine is that illegal transitions become visible," Lao Z said. "A ticket that is already 'paid' should never go back to 'draft'. The state machine has no such transition, so the code cannot take it even if it wanted to. Without a state machine, you avoid those illegal states by developer self-discipline. Miss one, and your data keeps a record that is 'paid but still awaiting approval'."
"Then a state machine and a graph are the same thing?"
"They are not," Lao Z said. "A state machine emphasises the legality of states and transitions — which transitions are allowed. A graph emphasises the expressiveness of nodes and paths — which paths are reachable. The same business can carry both views at once: use the graph to express the routing structure, use the state machine to constrain transition legality. They are two complementary check surfaces, not substitutes for each other."
"Give me one worked example?"
"Take creating a release ticket as the running example," Lao Z said. "Checkpoint 'validation done' first, then make one external create call carrying an idempotency key. On replay, read the checkpoint and the key; never create again blindly. There are three failure paths. One, the process crashes after the write — recovery has to query external state. Two, the model routes wrongly — the Graph should send it to a human or to a safe terminal. Three, the checkpoint was never written — you must test that risk together with the order of the side effects. Hybrid orchestration can wrap the fixed approval in the outer layer and put a bounded investigation Loop inside a node, but a node that uses a model does not thereby get to give up its replay, audit, and recovery contracts."
"Those failures — how does each one recover?"
"Sort them by whether the side effect has already landed," Lao Z said,
| Failure position | Side-effect state | Recovery | Counter-example |
|---|---|---|---|
| before the checkpoint write | no side effect | re-run the current node | if the node itself writes externally, it still needs idempotency |
| after the checkpoint write | the side effect has landed | replay from the checkpoint, de-duplicate with the idempotency key | no idempotency key → duplicate writes |
| the external system already executed | invisible to you | query external state, use the idempotency key, or hand off to a human | blind resend → duplicate orders |
| model routing error | the wrong branch was taken | send to a human or a safe terminal | let the model self-correct → it may route wrong again |
"The counter-example column is the important one," Lao Z said. "Every recovery method has an assumed-safe version that goes wrong: replaying because you assumed replay was safe, resending because you assumed resending was harmless. Recovery design lists 'how this goes wrong' first, then picks the recovery method — not the other way around, writing the recovery code and praying nothing happens."
The Gap Between the Diagram and the Implementation
"I noticed something: a lot of flowcharts look great, but the code does not match them."
"That is the trap Workflow and Graph make you step into most easily," Lao Z said. "The diagram is for people; the implementation runs the machine. Nothing guarantees the two stay consistent. When drawing, people naturally leave out error handling, timeouts, and idempotency, because a diagram shows the main path. The implementation cannot leave them out: every node has to handle failure, and every edge needs somewhere to go when it fails."
"Then what is the diagram actually for?"
"As a starting point for communication and design, not as the end point of acceptance," Lao Z said. "A diagram's value is that everyone in the discussion sees what the main path looks like, who decides the next step, and where the human gates are. But a correct diagram is not a correct implementation; whether implementation and diagram agree is checked by tests. 'The diagram says so' proves nothing."
| Diagram layer | Implementation layer | Typical gap |
|---|---|---|
| node | function / service / subprocess | the node does more in code than on the diagram (hidden side effects) |
| edge | function call / message / route | one edge on the diagram may be retries, degradation, and timeouts in code |
| branch condition | if / switch / routing rule | the diamond on the diagram may be several nested checks in code |
| termination | return / exit / exception | the exception paths the diagram omits must be drawn in the implementation |
"Can the gap be narrowed?"
"Two ways," Lao Z said. "One, annotate the non-main paths on the diagram: use dashed lines or notes to mark where timeouts, failures, and human intervention lead, so the boundaries show up while you discuss. Two, check the diagram against the implementation: once the code is written, back-fill each node's actual behaviour into the diagram and see whether the diagram's promises match the code's implementation. The gap cannot be eliminated, but it can be made explicit. What is stated in the open is a known boundary; what is hidden becomes a failure nobody defends."
"Can the diagram express a node where a model is involved?"
"It can express 'this node calls a model'. It cannot express 'how the model will decide'," Lao Z said. "For a model node, what you can draw is its inputs, outputs, and stop condition — not its decisions. So annotate model nodes with particular care: draw the boundary, not the behaviour. The boundary is the budget, the stop condition, the input/output contract. The behaviour is what the model decides inside that boundary; you cannot draw it, and you should not pretend you can."
Common Orchestration Anti-patterns
"Are there orchestration designs that look reasonable and then actually crash?"
"Three typical kinds," Lao Z said,
Three orchestration anti-patterns
- Squeezing the whole task into one big Loop and pretending it is a node: nothing outside can see its internal state, and the budget, the turns, and the stop reason are all hidden, so recovery and auditing have nowhere to start
- Letting a graph node improvise freely with a model and no budget at all: the graph only guarantees routing determinism, not that the node's interior stays under control
- Treating every model call as replayable: model calls are probabilistic, so a replay yields a different result; they cannot be re-run as safely as deterministic computation
"What do these three anti-patterns have in common?"
"All three mistake one layer's constraint for the whole system's guarantee," Lao Z said. "A Loop having a stop condition does not make the whole process safe. A Graph having routing does not stop a node from running away. A replay having an idempotency key does not make every node idempotent. Each layer guarantees only its own boundary. Assumptions that cross layers must be declared explicitly and verified explicitly."
"When you design orchestration, where should you start thinking?"
"Start from the worst case," Lao Z said. "Ask first: 'what is the most expensive failure of this process' — a double charge, a wrong notification, a wrong document generated. Then reason backwards from the most expensive failure: which layer does it happen in, what stops it there, and what happens if nothing stops it. Write the worst case down, then choose the form. The form is just where that thinking lands."
"To close this out: how do I know I have thought the orchestration through?" Xiao A asked.
"When you can answer three questions at once," Lao Z said. "Who holds the routing authority at each step — code or model? What are each node's budget and stop condition, so it cannot spin forever? After each failure, where do you recover from, and do side effects repeat? With all three answered, whichever form you choose stands up. Miss one, and however right the form looks, it collapses in some corner."
Summary
Loop, Workflow, and Graph are three tools for expressing control and state. They are not mutually exclusive, and you choose among them by how fixed the path is. An Agent Loop hands control to the model, letting it decide the next step for itself at every point, which suits open-ended problems whose path cannot be fixed in advance. A Workflow pins the path down: what each step does, in what order, and what condition must hold before it proceeds — all written in the definition — which suits processes where mistakes are costly and you need auditability and replay. A Graph sits in between, expressing the possible branches with nodes and edges: it allows conditional routing while keeping the overall structure observable. Their engineering profiles differ sharply — observability, cost, failure recovery, and test combinations are all different — and when choosing a form these hidden costs usually matter more than the name of the form.
Checkpoints and replay are the companion capabilities of Workflow and Graph: saving a recovery point lets the process stop and continue afterwards, and replay requires every step to be idempotent, otherwise an interrupted re-run charges twice. Recovery splits four ways by whether the side effect has already landed: re-run before the checkpoint, de-duplicate with the idempotency key after it, query external state or hand off to a human when the outside world already acted, and send to a human or a safe terminal when the model took the wrong branch. All three forms land on a state machine — states, events, transition conditions, and a persistable checkpoint. The state machine's value is that illegal transitions become visible, and the graph and the state machine are two complementary check surfaces, not replacements for each other.
Hybrid orchestration is the norm: a deterministic router splits the traffic at the entry, a bounded Loop analyses in the middle, and a deterministic process closes things out at the exit, with every open node carrying its own budget and stop condition. The three common anti-patterns are squeezing the whole task into one big Loop and pretending it is a node, letting a graph node improvise with a model and no budget, and treating every model call as safely replayable. What they share is mistaking one layer's constraint for the whole system's guarantee: each layer guarantees only its own boundary, and cross-layer assumptions must be declared and verified explicitly.
Hands-on Verification
- Draw an expense-report process as nodes, states, and conditions, and mark an idempotency strategy for every external write.
- Change one of its open-ended "explain the anomaly" nodes into a Loop of at most three turns, and define the stop reasons.
- Inject one timeout and one duplicate delivery, and verify that the process recovers and does not submit twice.