Chapter 2: The Truth About the Next-Token Machine — LLM

Xiao A took on an assignment: write the first version of the question-and-answer endpoint for the company's AI customer service. The requirement was simple — send the user's question to the model, display the answer. He finished the code in half a day and cheerfully ran a few batches of tests. Then he noticed something: for the same question asked twice, the first answer was perfectly well behaved, while the second produced an API that does not exist at all, and laid out its usage with a straight face.

Same parameters, same model, same code — yet the results differed this much. He pasted the two outputs side by side on the screen. The longer he looked, the less it made sense, so he dragged Lao Z over.

"Did I write something wrong?" he asked, pointing at the screen.

Lao Z did not answer. He stared at the screen for a while, then asked back: "What do you think it is actually doing?"

"Answering... the question?" Xiao A was not sure. "It has learned a lot of material. It knows the answer."

Lao Z neither agreed nor disagreed. He turned the laptop around and typed one line into the terminal: "Once upon a time there was a mountain, and in the mountain there was a __". Enter. The model supplied the missing word: "temple".

"Again." Lao Z typed the same line a second time. The model said "temple" again.

Xiao A did not get it. "What does that prove?"

"It proves it does not know the answer," Lao Z said. "Swap yourself in: see 'Once upon a time there was a mountain, and in the mountain there was a', and you would also write 'temple'. You did not look anything up — 'temple' is simply the smoothest continuation. That is the one thing it does: read all the text you have already written, and guess what is most likely to come next."

Metaphor: the next-token machine

It is a typewriter that does exactly one thing: read all the text you have already written, then type the next most likely token. It has no idea whether what it writes is code or poetry. It only knows that, read as a continuation, this is what belongs here.

"But it can write code and solve math problems," Xiao A protested. "How does guessing the next token produce any of that?"

"Repeat 'guess the next token' a few thousand times and see what happens." Lao Z drew a loop in the terminal:

input tokens ──> probability distribution over the next token ──> sample one token
^                                                                                │
└───────────────────── append the result back to the input ──────────────────────┘

"Given the tokens it already has, a language model outputs a probability distribution over the next token. One token gets selected, appended back to the input, and the process continues. That is autoregressive generation: it chains onto its own output, gathering mass like a rolling snowball."

Xiao A stared at the diagram and said nothing for a long time. He remembered his own line — "it has learned a lot of material, it knows the answer" — and suddenly it looked hard to defend.

"So the 'code' it outputs is continued token by token?"

"Right. That is why it sometimes hands you an API that does not exist: it did not look anything up before writing. Read as a continuation, this is where an API call 'most belongs'. So a fluent, lifelike answer is not evidence of fact." Lao Z paused. "'Predicting the next token' is a confirmed fact at the level of the model's objective and of the inference interface. On its own it cannot prove that the model has no internal representations, or that it cannot carry out reasoning. The more useful engineering conclusion is this: the output is text produced probabilistically, not the result of a database query."

Key points

• The essence of a large model is predicting the next token, not "looking up the answer"
• The generation mechanism is settled (autoregressive); the capability is trained into it. Do not merge the two into one sentence: "the model can think"

Align the names first

Term	English / Abbreviation	Working definition in this book	What it is not
Large language model	Large Language Model, LLM	A model or service that generates a probability distribution over subsequent tokens from the input tokens	A database of facts or a tool executor
token / tokenizer	token / tokenizer	Text encoded into the discrete units a model processes	A character count, a word count, or a letter count
Autoregressive generation	autoregressive generation	Once a token is selected it is appended as a condition and generation continues	Fetching a complete answer in one shot
Parameters	parameters / weights	The numbers optimized during training	Memory that every request writes into
Training	training	The process of updating model parameters with a training objective and data	An ordinary model request
Model inference	inference	Processing new input with fixed parameters to produce output	The reasoning process discussed in Chapter 3: Leaving a Scratch Paper for Thinking — CoT
Context window	context window	The token budget jointly visible to one request	A session that persists automatically across requests
Sampling / temperature	sampling / temperature	The process of choosing a token from the candidate distribution / a common parameter that shapes that distribution	A correctness switch, or a scale that means the same thing across vendors
Hallucination	hallucination	A fact, citation, or code detail that looks plausible but is not true	Wording that is merely imprecise
Prompt caching	prompt caching	A service mechanism that reuses computation for a reusable input prefix	Correctness, or automatic memory on the application side

2.1 What it does is conditional continuation

That afternoon Xiao A kept chewing on the phrase "chaining", turning it over from every angle, and it still felt wrong: the model writes code, translates, solves problems. How could it be "just chaining"?

He could not help going back to Lao Z. "You say all it does is chain tokens. But it plainly gets all this work done."

"Then think: when I say 'chaining', which image is closest to the one in your head?"

"Um... a typewriter?" Xiao A ventured.

"Right, a typewriter. The 'repeat it a few thousand times' from the opening is exactly what this typewriter is doing," Lao Z said. "I use 'chaining' as a metaphor so you grasp the generation mechanism first. Capability comes from training — a separate matter. Those two are the easiest things to merge into one sentence: 'the model can think.'"

Key point

"Chaining" describes the generation mechanism (how the next token is produced), not the ceiling of capability. Capability is a separate question, and the next few chapters open it up.

2.2 Tokens are a unit of measure, not a character count

Back at his desk, Xiao A started adding usage accounting to the endpoint. He had planned to keep the books in characters: the user sent a few hundred, the model replied with a few dozen, obvious at a glance. He opened the billing detail and froze. The unit listed was not characters but something called a token, and the number did not match the characters he had counted.

He took the bill to Lao Z. "What is a token? Why is the count higher than my character count?"

"What it processes is not 'Chinese characters' and not 'words'. They are tokens cut out by a tokenizer," Lao Z said. "How the splitting works varies by model: one word, one snippet of code, one punctuation mark — any of them can be a token, and the same string yields different token counts under different models' tokenizers. Estimate with 'one Chinese character equals one token' and you will get the books wrong sooner or later."

"So what does it actually affect?"

"Three things: how much input the context can hold, how long the output can be generated, and how most APIs meter and charge." Lao Z counted on his fingers. "Do you have to count first when you write the program? You only need to when you are close to a model limit, chasing down a cost, or designing a truncation policy. Day to day it is enough to treat long logs, whole repositories, and repeated rules as context that costs something, not as free strings."

"Do token counts differ much between languages?"

"A great deal. A tokenizer is designed around the statistical regularities of its training corpus: in English a common word may be a single token; a frequently used Chinese character usually lands near one token, and a rare one may split into two or three; in code, indentation, brackets, and identifiers each get split their own way. For text of the same length, switch the language or the content type and both the token count and the cost can differ several-fold."

Content type	Rough tokens per character	Notes
Common English words	about 0.25 (4 characters ≈ 1 token)	The cheapest case
Chinese	about 0.5-1.5 (1 Chinese character ≈ 0.5-1.5 tokens)	Varies with the tokenizer and with character frequency
Code	highly variable	Indentation, brackets, and long identifiers split separately
JSON / structured data	higher	Large amounts of punctuation and quotation marks split on their own

"Are these ratios fixed?"

"No. They are only rough orders of magnitude; the exact value depends on the specific model's tokenizer," Lao Z said. "For a precise estimate, use the token-counting endpoint the target API provides, or run the matching tokenizer locally. There is no universal 'characters ÷ N = tokens' formula. N moves with the model and with the content."

"So what does that mean for managing the context?"

"It directly affects the truncation policy," Lao Z said. "Truncate the message history by character count and you may cut early while the token budget still has room — space wasted — or you may fail to cut at all once the tokens are already over the limit, and the request gets rejected. Truncating by tokens is more accurate, but only if you can count tokens. Without counting ability, leaving a safety margin is steadier than riding right up against the limit."

2.3 Parameters, training, and inference

The next day, while debugging, Xiao A asked offhand: "It is so good at continuing text. Was it born with that?"

"No. Parameters — also commonly called weights — are the numbers the model learns during training," Lao Z said. "Training updates the parameters through vast amounts of samples and an optimization process. Inference uses the fixed parameters: it runs a forward computation over new input and generates output."

"Could I train one myself?" Xiao A's eyes lit up for a second.

"You can, but keep the responsibilities straight. Application development calls an inference service; it does not change model parameters on every request." Lao Z laid out a table for him. "Training and inference differ completely in the resources they take, in cost, and in what you get to control."

Stage	Main action	What an application engineer usually controls
Training	Update the parameters against a training objective	Data, the training scheme, or the model choice (if you take part in training)
Inference	Generate output with fixed parameters	Input, model, sampling, tools, and stopping conditions

"How much do the two sides differ in cost?"

"Several orders of magnitude," Lao Z said. "Training one large model takes a great many GPUs running for weeks or months, and the cost runs into millions of dollars. One inference request is usually billed per token: a few cents to a few tenths of a dollar. An application developer's job is to get the task done inside an inference budget, not to retrain the model."

"What about fine-tuning? I see a lot of people talking about it."

"Fine-tuning sits between the two: you continue training an existing model on a smaller dataset and adjust part of its parameters. The cost is far below training from scratch, but still far above an inference call," Lao Z said. "It suits the case where you have a large body of domain data and the general-purpose model falls short on your tasks. But it is not free performance: a fine-tuned model can regress on general capability, and it needs extra evaluation and maintenance."

Approach	Cost scale	Suitable for	Risk
Training from scratch	Extreme (millions of dollars)	Foundation model research	Not an application developer's business
Fine-tuning	Moderate (needs data and GPUs)	Plenty of domain data, general model insufficient	General capability may regress
Inference calls	Low (billed per token)	The overwhelming majority of application scenarios	None (you are just calling a service)

"Then is a model with more parameters necessarily smarter?"

Lao Z shook his head. "'More parameters must mean smarter' is not a reliable rule. Model architecture, data, training method, inference budget, and task type all affect the result. Base model selection on evaluation against your target task."

"What specifically should I watch out for in evaluation?"

"Measure on your task, not on the leaderboard," Lao Z said. "A leaderboard score is computed over someone else's task distribution. Your scenario may be long text, or tool calling, or Chinese output. A one-point gap on the leaderboard can reverse completely once it is your task. Build your own sample set for evaluation: a few real tasks, a few failure paths, one fixed probe. Run the same set each time and check whether the output satisfies the contract. Then when you switch models you are comparing 'which one fits my scenario better'."

Key point

As an application developer you only do inference: you call a service that has already been trained. Training is the model vendor's business. The two costs differ by several orders of magnitude.

2.4 The context is one request's workbench

On the morning of the third day, Xiao A came rushing over to Lao Z, excited: "Yesterday I went twenty-some turns with the model and it remembered all of them! You said it cannot remember. Weren't you wrong?"

Lao Z did not argue with him directly. He only asked: "In the message you sent today, did you carry yesterday's conversation along with it?"

Xiao A blinked, then looked through his own code: his implementation stuffed the entire history into every request.

"It does not remember," Lao Z said. "Every model call has one finite context: system instructions, user messages, message history, tool results, and the current question all occupy it. The model itself does not automatically keep the previous conversation between two HTTP requests. What you see as 'remembering' is the program carrying the relevant content into the new request again — something the application does deliberately."

"Then it still remembers." Xiao A was not ready to give up.

"Think of the context as a workbench," Lao Z said, drawing a box on paper. "The desk holds only so much. What will not fit forces a trade-off. A new request means a brand-new desk."

system rules + dialogue history + tool results + current input + reserved output
└───────────────── all jointly bounded by the context window ──────────────────┘

"So maxTokens is the size of the desk?" Xiao A felt he had found the answer.

"Only half right. maxTokens (or a similarly named field) usually caps the output you expect from this call; the context window caps the total budget for input and output — and different services define the reserved space and the field names differently." Lao Z warned: "When you implement this, read the target API's current documentation, and reserve room for the output before sending the request. Do not look only at the input length."

"When both input and output grow, which gets cut first?"

"Cut what can be rebuilt," Lao Z said. "Among the system rules and the message history, which parts can be reconstructed and which must stay? The usual pruning order is: project material that can be retrieved again goes first, then the conversation history from oldest to newest, and the system rules last. On every request you can ask one question: if this piece of context were gone, could the model still finish the current task? If no, keep it. If yes, prune it first."

"What if the context is full? Is just cutting it off a bad thing?"

"Cutting loses information, but there is something worse than cutting: not cutting, and having the request rejected outright," Lao Z said. "Truncation is an explicit boundary — you drop the old content and make that known. Not truncating means the request fails for exceeding the limit, and the task never starts at all. Between the two there is compaction — squeezing the older conversation into a summary — but compaction loses detail too. Truncate, compact, enlarge the window: three paths ordered from lowest cost to highest, and none of them free."

"Which category do tool results fall into? They are on the desk too."

"They count, and they are the most overlooked thing on it," Lao Z said. "The logs, file contents, and search results a single tool call returns all occupy context, and they should be cleared once that round is used up. Like the conversation history, they can be rebuilt — rerun the tool and you get them back — so they sit high in the pruning order. But rebuildable does not mean rebuilding is free: a rerun pays the cost again. Cache what can be cached first, and for anything you really must prune, keep the citation location for the audit before you drop it."

Key points

• Context window: the finite budget visible to one request (the size of the desk)
• maxTokens: the output cap for a single call (how long it can write in one breath)
• The two govern separate things — do not conflate them; the model itself does not remember across requests

2.5 Sampling means there is no single answer

Two days later Xiao A ran into a new problem: the same request, run twice, gave different answers. He suspected the network and was about to go and investigate when Lao Z stopped him.

"Don't check the network yet. Put the two outputs side by side. Apart from a few words, do they roughly line up?"

Xiao A did as told and found that they did.

"That is the point: the model produces a probability distribution, not one answer," Lao Z said. "Once the model gives the distribution, the service selects tokens by its sampling rule. Parameters such as temperature, top-p, and top-k change exactly that candidate distribution or candidate set."

"Does turning the temperature down make it 'accurate'?"

"Not necessarily. Lower randomness usually helps format stability and repeatability; higher randomness may add variety to the wording," Lao Z said. "But neither is a guarantee of an 'accurate mode' or a 'creative mode'. A wrong fact can still be repeated reliably under low randomness. If the interface supports a seed, it only helps reproduce results under certain conditions — it should not be read as a promise of determinism across models or versions."

"What does each of temperature, top-p, and top-k control?"

"They control how the pick is made out of the candidates, and they work at three different levels," Lao Z said. "Temperature adjusts the flatness of the distribution: the higher the temperature, the more easily a low-probability token gets chosen, and the more divergent the output. Top-p — nucleus sampling — keeps the smallest candidate set whose cumulative probability reaches the threshold, and picks from inside it. Top-k keeps only the K most probable candidates. All three change the sampling path, not the model's own capability. Push the parameters to whatever extreme, and the model still cannot suddenly do what it could not do to begin with."

Parameter	What it controls	Effect of raising it	Common misuse
temperature	The flatness of the distribution	More divergent, more varied	Expecting it to "correct" factual errors
top-p	The cumulative-probability threshold of the candidate set	Candidates concentrate in the high-probability region	Stacking it with top-k without thinking
top-k	The size of the candidate set	Keeps only the K highest-probability candidates	Fixing K and ignoring the shape of the distribution

"Should these parameters be tuned together?"

"Do not stack them casually. Adjusted at the same time they interfere with each other, and you can hardly say which layer produced the final effect," Lao Z said. "In practice you usually move only one or two: lower the temperature for format tasks, raise it for exploratory tasks, and keep the defaults in most cases. Also, not every provider supports every parameter — read the documentation before you decide."

Xiao A wrote "move only one or two" in his notebook.

"Then how do I verify that the output is stable?"

"By testing, not by parameters," Lao Z said. "Run a fixed task several times and check whether the format is consistent and whether the key facts are right every time. When you need reproducibility, use a seed plus a fixed parameter combination, and record them as part of the request. Stability is a conclusion you measure, not a guarantee you tune."

Xiao A was not done yet: "And temperature at 0? Then valid format has to be guaranteed, surely."

"No guarantee. Temperature 0 only means it picks the highest-probability token every time. The model can still generate JSON whose brackets never close — just with a lower probability, not zero," Lao Z said. "Format validity is a validation problem, not a sampling problem. When you need a strict format, constrain it with a schema plus runtime validation, or have the model produce output through a tool that follows a controlled template."

Key point

Sampling parameters change "how the choice is made", not "whether it is right". For stable format, tighten the randomness and test; for factual correctness, rely on evidence and validation.

2.6 Hallucination is a verification problem, not a wording problem

After two days of detours, Xiao A finally pulled the topic back to the original bug: "The API it produced that does not exist — does that count as 'not well enough worded'?"

"It is not a wording problem," Lao Z said. "When a model generates a fact, citation, or code detail that looks plausible but is not true, that is commonly called a hallucination. It shows up as invented APIs, misread files, wrong attribution, and fabricated sources. The model generates a likely continuation; a prompt saying 'do not hallucinate' cannot eliminate the risk."

"So what can I do?" Xiao A remembered the pits he had stepped in, and his tone turned serious.

Lao Z listed a few:

Engineering countermeasures against hallucination

1. Supply the raw source for any checkable fact, and require a citation location
2. Have the model read, search, or run verification through bounded tools
3. Put rule checks, tests, or human approval on high-risk results
4. Keep "unsure" and "verification failed" as visible states, instead of forcing the model into a conclusion

Xiao A copied the four into his notebook. "'Keep an unsure state' — how exactly does that land?"

"It means treating 'I don't know' and 'not verified' as legitimate output categories," Lao Z said. "Many tasks force the model to a conclusion: not answering counts as answering wrong. But in engineering, the model stating plainly 'I cannot confirm this from the evidence' is far safer than a hard-coded answer. When the application shows that state on its own instead of swallowing it as a failure, the user knows to go and check rather than to act on it. Allowing the model to say 'I don't know' is part of the validation chain."

Xiao A pointed at the second item on the list: "Then if the model goes and 'looks it up', is that necessarily right?"

"Not necessarily," Lao Z said. "What a tool returns can itself be wrong or contaminated. A tool is not a 'truth button'. It is one link that makes the chain of evidence checkable."

Xiao A was still not at ease: "Is hallucination only ever 'inventing a whole block'?"

"No. Partial distortion is harder to guard against," Lao Z said. "A wholly invented block is obvious at a glance, but what slips through more often is this: a citation to an article that really exists, but with its conclusion misassigned; a number that is correct while the scope it applies to is stated wrongly; an API name that is real but parameters that were made up. In that kind of hallucination every fragment looks verifiable on its own, and it only goes wrong when you put them together. Verification cannot stop at 'does this thing exist'. It must also ask 'is it used correctly here'. The first blocks fabrication; the second blocks distortion."

"Can the verification itself go wrong?"

"It can. Verification is code too," Lao Z said. "A regular expression may match the wrong span, the scraped page may itself be wrong, and a human reviewer may miss things. So a verification conclusion must keep a reference to the original evidence: what the model cited, what you checked, and what you concluded. Store the three as one record, and verification stops being a one-off transaction."

2.7 Cost and prompt caching

At the month-end reconciliation, Xiao A stared at the model bill, dumbfounded, and went to Lao Z: "How is this so expensive? I only had a few dozen exchanges."

"You resend the entire history every time, and you carry several long documents along, right?" Lao Z said. "The cost of one call usually depends on the input, the output, and the provider's caching rules. A long context raises both latency and price, and it dilutes the instructions that actually matter. Many services offer prompt caching or prefix reuse: when a stable prefix satisfies their matching and validity conditions, the service can reuse part of the computation."

"So if I stuff all the material into the prompt, a cache hit saves me money?" Xiao A's eyes lit up.

"Do not park unrelated material in the prompt for a long time just to feed the cache," Lao Z cut him off. "As a general design rule, put the stable instructions in the prefix and the frequently changing information at the end. But cache optimization cannot replace context management. Exact prices, hit conditions, and retention periods are product policy: take the official documentation and your bill at the time of the call as the authority."

"Are there pitfalls with the cache?"

"The biggest one: one character different in the prefix and the whole cache misses," Lao Z said. "Prompt caching matches the stable prefix exactly. Touch one spot in the prefix and everything after it is recomputed. If you put fields that change every time — a session ID, a timestamp — at the front, the cache will never hit, and the bill will not show you anything is wrong. To judge whether a prefix is stable, ask whether it is byte-for-byte identical on every request. Put what changes at the back, keep it out of the prefix, and only then can you hold the hit rate up."

Key point

Prompt caching hits on the "stable prefix": the prefix must match before it saves money. Do not treat cache optimization as a substitute for context management.

2.8 How one generation becomes an observable request

The following week Xiao A invited Lao Z over to his desk and demonstrated the "API that does not exist" bug once more.

"Which step should I start investigating from?" he asked.

Lao Z turned the laptop around and pointed at the log. "Land 'chaining' in a program and there are at least three distinct stages: the input is split by the tokenizer; the service computes over the context it already has and selects the output step by step; the application receives text, usage, or an end signal. The first two stages belong to the model service; the last stage is under the application's control. Merge them into 'the model answered once' and troubleshooting loses its bearings."

user task → app assembles the context → service generates the token sequence → stream events → app validates/displays
                        ↑                                ↑                           ↑
               controllable input              probabilistic output          auditable result

"And the failure paths are more than 'a wrong answer', aren't they?"

"Right. The input may exceed the budget and be rejected, the output may be interrupted by the length limit, and a streaming connection may break after you have received part of the text," Lao Z said. "The UI should mark a partial result as incomplete, and must not store it in the same state as an answer that ended naturally."

"How do the stages hand off to each other without losing information?"

"What passes between stages is a structured record, not a sentence," Lao Z said. "The application's assembly stage records an input summary and a version number; the service's generation stage records usage and the stop reason; the application's handling stage records the validation result and where the data was stored. Each stage's output is the next stage's input. If the stage before leaves only 'the model finished answering', the next stage has no evidence to work with. Break 'finished answering' into 'how many tokens it used, how it stopped, and whether it was validated', and troubleshooting can answer different questions."

"If the user does not want to wait any more, can we cancel?"

"You can, and cancellation must be kept clearly apart from 'completed'," Lao Z said. "Cancellation is the application interrupting on purpose: it emits an abort signal, the service stops generating, and the application marks the part already received as incomplete. That is not the same thing as the truncation and the broken stream you saw earlier. Truncation means the output hit its cap and was cut; a broken stream means the connection failed; cancellation means the result was voided deliberately. All three say 'the answer did not finish', but for different reasons, so the records must store them separately. Otherwise 'how many answers were interrupted' can only be guessed."

Illustration: the user asks for "list the test commands in this repository". If the application provides no repository files and no retrieval tool, the model can only keep generating from what is common in its training data. Even if the commands read fluently, that does not show they exist. If the application reads the project configuration first and then asks the model to cite what it read, the conclusion still has to be confirmed by an actual run or a human check.

2.9 Keep mechanism and conclusion apart

Over the next few days Xiao A finally straightened out the whole chain. He made a table and pinned it up beside his desk:

Confirmed mechanism	Engineering action it justifies	What you cannot infer from it
Tokens are generated one by one	Design states for length, cancellation, and partial output	That the output went through complete reasoning
Fixed parameters take part in inference	Do not treat one chat session as model training	That more parameters must mean more accuracy
A finite context window	Budget, truncate, or compact the context	That every input gets used correctly
Sampling affects the selection	Tighten randomness and test for format tasks	That low temperature must mean factual correctness
Prefix caching is a service policy	Separate the stable prefix from dynamic content	That the cache will hit, or that it saves money

"So the bug from the beginning — it produced an API that does not exist — was not a network problem, and not my code being wrong either?" Xiao A ran the whole chapter through his head.

"No. When a colleague says in a repository walkthrough 'read the configuration, then answer', that only shows the application put evidence into the current context. The conclusion should still be confirmed by execution or a manual check." Lao Z summed up: "Finishing normally while citing a nonexistent API is a fact-verification failure, not a network failure. Input over budget, output at the length limit, and an interrupted stream call for shrinking the input, marking the result incomplete, and stopping then recovering respectively; you cannot store all three as success."

"When I investigate, how do I quickly tell which stage the problem is in?"

"Sort it by 'who holds the evidence'," Lao Z said. "For an error in the application's assembly stage, the log shows the input fields are wrong — go look at the application code. For a problem in the service's generation stage, look at the usage fields and the stream events — check the request parameters and the server-side state. For a problem in the application's handling stage, look at the UI and the stored records — check your display and persistence logic. Each class of error puts its evidence somewhere different. Troubleshooting them all mixed together is guessing."

"Anything else to keep in mind at the end?"

"Two pairs people mix up," Lao Z said. "First, 'the model raised no error' is not 'the result is correct': finishing normally only means text was generated, and the content still needs verification. Second, 'truncated' and 'rejected' are different states: truncated means the output ran past the limit and was cut off; rejected means the input was over budget and the request never went out. Store each state separately, and you can tell them apart when you troubleshoot."

Summary

The first step in understanding an LLM is to let go of the intuition that "it thinks". At its core it is a next-token machine: working token by token, it predicts what the next token should be from what it has already seen. A token is not a character, a word, or half a word — the tokenizer decides, and the same word can split into different fragments under different tokenizers. Once the model selects a token it appends that token to the conditions and generates the next one; that is the entire meaning of "autoregressive". Training compresses a vast corpus into parameters, and inference uses those fixed parameters to handle new input. The two are completely different processes — do not conflate them.

This next-token machine has two fundamental limits. The first is the context window: for content beyond the window, the model has not "forgotten" it — it "never received" it. The window is the budget visible all at once at request time. The second is the probabilistic nature of sampling: parameters such as temperature, top-k, and top-p change the distribution of the output, not the truth or falsity of a fact. No sampling parameter can guarantee that what gets generated is real. Probabilistic sampling makes outputs vary; turn the temperature as low as you like and the model may still repeat a wrong fact reliably. Holding hallucination down takes evidence, execution, and validation.

Hands-on Verification

Run the same model on the same question twice: once asking for the conclusion only, once asking it to list local evidence you can check or official sources. Compare the share of verifiable claims in the two answers. Then add a long stretch of irrelevant text to the input and record the change in latency, in the usage fields, or in output quality. If the service does not expose those fields, do not use this to guess how it bills internally.