Understanding AI and Language Models

Chapter 1

Neural Network Foundations

Goal: Understand what a neural network stores, how training changes it, and why its output is an approximation rather than a database lookup.

The shortest useful mental model

Ordinary programs combine instructions written by a developer with runtime data. A trained neural network instead combines a mostly fixed computation with millions or billions of learned numeric parameters. Training discovers parameter values that make useful outputs common on examples. Inference freezes those values and applies the resulting function to new input.

Think of the model as a very large, differentiable configuration file. Its weights do not usually store readable rules such as “an unopened file handle is a resource leak.” They store distributed numerical influences that, together, make some output patterns more likely than others.

Data, parameters, and the learning problem

A training example contains an input and some learning signal. In supervised learning the signal may be an explicit label: a log record paired with incident or normal. In self-supervised learning the signal is derived from the data itself: hide or shift part of a sequence and train the model to predict it. Language model pretraining is self-supervised because ordinary text supplies both the preceding tokens and the token that came next.

Parameters are learned values such as weights and biases. Hyperparameters are choices made around learning: layer count, batch size, learning rate, or training duration. Data is normally split into training data used for updates, validation data used for decisions, and test data reserved for a final estimate. Reusing a test set to tune the model quietly turns it into another validation set.

Vectors, matrices, and dot products

A vector is an ordered list of numbers. It can represent input features, an intermediate state, or an output score. A matrix is a rectangular collection of numbers that transforms one vector into another. Neural networks use matrix multiplication because it applies many weighted combinations efficiently in parallel.

The simplest weighted combination is a dot product: multiply matching positions and add. Suppose a toy change-risk model has input x = [1, 0.5], where the values mean “tests are failing” and “change size, normalized.” With weights w = [2, 0.8], the dot product is (1 × 2) + (0.5 × 0.8) = 2.4. A bias of -1 shifts the score to 1.4.

The weight controls how strongly each input affects the result; the bias sets a baseline. Real models apply this operation to vectors with many dimensions and learn the values rather than having a developer choose them.

Neurons, activations, and layers

A conceptual neuron computes a weighted sum and passes it through an activation function. Without activations, stacking linear matrix operations would collapse into one linear operation and could express only limited relationships. An activation introduces a nonlinearity, allowing layers to build curved decision boundaries and conditional features.

A common intuitive activation is ReLU: ReLU(z) = max(0, z). It passes positive evidence and clips negative values to zero. Sigmoid compresses a number into a value between zero and one, which is convenient for a binary probability. Modern architectures use other smooth activations, but the engineering idea is unchanged: each layer transforms its input into a representation that later layers can use.

# A tiny forward pass; shapes matter more than syntax.
hidden = relu(W1 @ input_vector + b1)
scores = W2 @ hidden + b2
prediction = softmax(scores)

This computation from input to output is forward propagation. During inference, that is almost the whole story. During training, the model must also learn how each parameter contributed to the error.

Loss: turning “wrong” into a number

A loss function converts prediction quality into a scalar that an optimizer can minimize. For regression, squared error heavily penalizes predictions far from a numeric target. For classification, cross-entropy penalizes assigning low probability to the correct class.

Before becoming probabilities, output values are often logits: unconstrained scores. Softmax exponentiates and normalizes them into a distribution that sums to one. For logits [2, 1, 0], softmax is approximately [0.67, 0.24, 0.09]. Raising one logit increases its probability while decreasing the others because they compete for the same total.

If the correct class receives probability 0.9, the loss is about 0.105. If it receives 0.1, the loss is about 2.30. Cross-entropy therefore punishes confident wrong answers strongly. It does not encode every quality you care about; it encodes the training objective the optimizer can see.

Gradient descent and backpropagation

Imagine loss as elevation on a landscape whose coordinates are all the parameters. A gradient contains the partial derivative of loss with respect to each parameter: locally, which direction makes loss rise fastest and by how much. Gradient descent takes a small step in the opposite direction.

The learning rate controls step size. Too large can overshoot or make training unstable; too small can make useful progress painfully slow. Optimizers such as stochastic gradient descent and adaptive variants change how gradients are accumulated and scaled, but they do not change the central loop.

Backpropagation computes gradients efficiently from the output layer back toward the input. It applies the chain rule: if a weight changed an intermediate activation, and that activation changed the loss, combine those sensitivities to determine the weight's responsibility. Backpropagation is not a separate learning goal or a model “thinking backward”; it is accounting over the computation graph.

initialize parameters
repeat:
    batch = sample(training_data)
    predictions = forward(batch.inputs, parameters)
    loss = loss_function(predictions, batch.targets)
    gradients = backpropagate(loss, parameters)
    parameters = optimizer.update(parameters, gradients)

Batches, epochs, and generalization

Computing one update from the entire dataset is expensive. Training usually uses a batch, a subset whose average gradient estimates the broader direction. Smaller batches produce noisier updates; larger batches use more memory and produce smoother estimates. An epoch is one pass through the training examples. A checkpoint saves parameters and optimizer state so training or evaluation can resume from a known point.

The desired outcome is generalization: low error on relevant inputs not memorized from training. Underfitting means the model or training process has not captured enough structure; both training and validation performance are poor. Overfitting means training performance keeps improving while validation performance stalls or worsens. Causes include insufficient or unrepresentative data, excessive capacity, label leakage, and optimizing too long against the same examples. More parameters are not automatically worse—capacity becomes a problem when the learning setup rewards patterns that do not transfer.

Back to study method · Next chapter

Chapter 2

How Language Models Work

Goal: Trace a request from text to tokens, through transformer blocks, and back to generated text—and understand why fluent next-token prediction can hallucinate.

The generation pipeline

text
  → token IDs
  → token embeddings + position information
  → repeated transformer blocks
  → logits for the next token
  → sampling rule chooses one token
  → append token and repeat

A language model does not generate a whole answer at once. It predicts a distribution for the next token, chooses one, appends it to the context, and runs again. Everything in the completed answer emerges from this repeated loop.

Tokens and embeddings

Models process tokens, not characters or words. A tokenizer maps text to integer IDs using a fixed vocabulary of common pieces. A familiar word may be one token; an unusual identifier may be several. Whitespace and punctuation can matter. This explains why token limits do not map cleanly to word counts and why source code, multilingual text, and generated identifiers can consume context at different rates.

A token ID is only an index. An embedding table maps each ID to a learned vector. During training, tokens used in similar contexts acquire representations that support similar computations. An embedding is not a dictionary definition; each dimension participates in many learned features, and meaning changes further as the vector flows through the network.

Because attention alone does not inherently know sequence order, the model also receives position information. That lets it distinguish dog bites person from person bites dog even though the same token pieces appear.

Inside a transformer block

A transformer contains many repeated blocks. Each block has two major kinds of computation: self-attention, which moves information among token positions, and a per-position feed-forward network (often called an MLP), which transforms the resulting features. Residual connections preserve an earlier representation by adding it to a block's output, and normalization keeps values in workable ranges.

For self-attention, each token state is projected into three vectors: a query describing what information this position seeks, a key describing what it offers, and a value containing information to pass along. A query is compared with all eligible keys using dot products. Softmax turns those scores into attention weights, and the output is a weighted sum of the value vectors.

scores  = query @ keys.transpose()
weights = softmax(scores)
output  = weights @ values

In a generative model, a causal mask prevents a position from attending to future tokens during training. Multiple attention heads perform separate learned projections, allowing different relationships to be useful at the same time—such as matching a variable use to a definition or connecting a pronoun to earlier context. It is misleading to assign one stable human concept to each head; the representations are distributed and context-dependent.

The MLP then transforms each position independently with learned matrix operations and nonlinear activations. Attention communicates across positions; the MLP develops features at each position. Repeating these operations lets later layers work with increasingly contextual representations.

Pretraining and post-training

During pretraining, the model sees many sequences and learns to predict the next token. To improve at that objective it must absorb regularities in language, code, document structure, and the concepts expressed in its data. The objective does not explicitly say “learn grammar” or “learn programming”; those capabilities become useful internal machinery for reducing prediction error.

A pretrained model completes text, but it may not reliably follow requests. Instruction tuning continues training on examples of instructions and desired responses. Preference-oriented post-training further encourages answers humans or evaluators rank as more helpful and discourages unwanted behavior. These stages shape response style and policy; they do not replace the pretraining objective's basic next-token mechanism.

Inference and sampling

The final layer produces one logit per vocabulary token. Softmax converts logits into probabilities. A decoder can select the highest-probability token, sample from the distribution, or restrict the candidates first. Temperature rescales logits: lower values make the distribution sharper and usually more repeatable; higher values flatten it and admit less likely continuations. Top-k keeps only a fixed number of candidates; top-p keeps the smallest group whose cumulative probability reaches a threshold.

Sampling is only one source of variation. Serving infrastructure, numeric precision, parallel execution, model updates, and hidden prompt changes can also affect reproducibility. For code transformations and extraction, use conservative sampling and validate outputs. For ideation, variation can be useful.

Why hallucination follows naturally

The model is optimized to produce a plausible continuation, not to query an internal truth table. When context strongly supports a fact, the likely continuation may be correct. When evidence is missing, contradictory, or outside learned patterns, the same mechanism still must produce a next-token distribution. A polished but unsupported completion is therefore not an exotic malfunction; it is a predictable consequence of generation without grounding.

For example, if asked to describe an imaginary library with a realistic-looking name, the model may continue with conventional API terms because those tokens fit the prompt. Fluency measures compatibility with learned patterns, not existence. Reliable applications add retrieval, tools, constraints, and verification rather than asking the model to “be certain.”

Previous chapter · Next chapter

Chapter 3

Capabilities and Limitations

Goal: Predict which tasks suit a model, which require grounding or tools, and which should remain deterministic software.

Think in distributions, not abilities

A model does not possess a fixed checklist of skills. Its reliability is a distribution over tasks and inputs. It may summarize dozens of pages well yet fail to count a small set exactly; generate a sound algorithm yet invent a library method; solve one phrasing and miss an equivalent one. Good system design asks, “Under what inputs and controls is this behavior reliable enough?” rather than “Can the model reason?”

Models are especially useful when inputs are unstructured and outputs allow semantic variation: drafting, classification, extraction, summarization, translation, explanation, code suggestions, and approximate search. They are weaker as authorities for current facts, exact arithmetic, exhaustive enumeration, access control, or operations where one silent error is unacceptable. Connect a calculator, database, compiler, or policy engine when those systems already solve the exact part.

Context is temporary working input

The context window contains tokens available for the current generation: instructions, conversation, retrieved documents, tool results, and the answer produced so far. It is not durable memory. A long context window is capacity, not guaranteed use; relevance can degrade when important evidence is buried among repeated, conflicting, or irrelevant text. Position and phrasing can change which information influences the answer.

Training knowledge is also not a current database. Even if a model learned a fact, it may not reproduce it reliably or know that it changed. Time-sensitive questions need a live source. Exact citations need retrieved documents and an attribution check, not a request to remember a URL.

Reasoning-like behavior and brittle edges

Models can produce useful multi-step solutions because training contains many patterns of decomposition and because intermediate text gives the next step more relevant context. That does not guarantee a stable symbolic process. A plausible early mistake can condition all later tokens, and a coherent explanation can be constructed around a wrong result. Treat reasoning traces as candidate work products to verify, not privileged access to the model's internal cause.

Non-determinism makes failures harder to reproduce. Record model and prompt versions, parameters, retrieved context, tool calls, and outputs. Benchmark scores are useful only when the benchmark resembles your task and has not become part of the optimization target. Your own representative evaluation set is more actionable than a single public ranking.

Previous chapter · Next chapter

Chapter 4

Running Models Locally

Goal: Understand local inference as a deployment choice with explicit memory, performance, quality, and operational trade-offs.

What “local” changes

An open-weight model makes learned parameter files available under some license. Running it on hardware you control can improve data locality, offline availability, customization, and cost predictability at steady load. It also makes you responsible for capacity planning, upgrades, observability, security, and model licensing. Open weights do not automatically mean open training data, unrestricted use, or equal capability to a hosted model.

Memory before compute

First estimate whether the parameters fit. A rough lower bound is parameter count × bits per parameter ÷ 8. Runtime needs additional memory for temporary activations, server overhead, and the key/value cache used to avoid recomputing earlier attention states. Longer contexts, more concurrent requests, and larger batches grow that cache.

Quantization stores or computes weights at lower precision. Moving from high precision to 8-bit or 4-bit representations reduces memory and can improve throughput, at a possible quality cost. The trade-off depends on the model, quantization method, task, and hardware; test your workload rather than treating bit width as a quality guarantee.

GPUs accelerate parallel matrix operations, but memory bandwidth often limits token-by-token generation. CPUs can run smaller or heavily quantized models, usually with lower throughput. Unified-memory systems can make a larger shared pool available but do not eliminate bandwidth constraints. Measure time to first token, generation tokens per second, peak memory, and throughput under realistic concurrency.

The serving stack

A local runtime loads a model file, applies the model's tokenizer and chat template, executes inference, and exposes a CLI, UI, or HTTP API. Different runtimes optimize different hardware and formats. A portable quantized file format packages tensors and metadata, but the prompt template still matters: a model trained to recognize specific role markers may perform poorly if raw messages are serialized incorrectly.

# Generic shape of a local chat request.
POST http://localhost:PORT/v1/chat/completions
{
  "model": "local-model",
  "messages": [
    {"role": "user", "content": "Explain this stack trace."}
  ],
  "temperature": 0.2
}

An API compatible with a familiar hosted interface can simplify switching, but compatibility is rarely behavioral equivalence. Models differ in role handling, context size, tool-call format, structured output, and stop tokens. Put a small adapter around providers rather than leaking assumptions throughout the application.

Previous chapter · Next chapter

Chapter 5

Model Selection

Goal: Select and route models by measured product requirements instead of size, novelty, or a single leaderboard.

Start with the contract

A model is one component in a service-level objective. Define the task, acceptable error, maximum latency, throughput, context and output size, privacy boundary, tool or vision needs, deployment constraints, and cost ceiling. “Best model” is incomplete without these constraints. A smaller model that returns valid JSON in 300 milliseconds may be better for classification than a stronger model that is slower and more expensive.

Model families specialize through architecture, data, and post-training: general chat, code, visual input, embeddings, or deliberate multi-step generation. Labels are hints, not proof. Test the exact interface you need: long-context retrieval, schema adherence, tool selection, multilingual text, or repository-scale edits.

Build a representative bake-off

Collect real task shapes, including ambiguous requests, long inputs, malformed data, and cases where the model should abstain. Freeze prompts and decoding settings, run candidates more than once where variation matters, and record correctness, format compliance, latency, cost, and safety. Blind human reviewers to model identity when judgment is subjective.

Routing can use known task type, input size, user tier, or a cheap classifier. Escalation is safer than pretending every request is easy: try a small model, verify the result, and send failures to a stronger model or a human. Include routing mistakes in evaluation because a perfect specialist is useless if the router sends it the wrong work.

Previous chapter · Next chapter

Chapter 6

Prompt Engineering

Goal: Specify a model task as clearly and testably as an API contract.

A prompt is a runtime specification

Good prompting resembles writing a precise ticket or function contract. State the objective, relevant background, constraints, allowed evidence, output shape, and acceptance criteria. The model should not have to infer whether “review this code” means style feedback, exploitable bugs, or a release decision. Clear prompts reduce the space of plausible continuations.

Objective: Find correctness defects introduced by this patch.
Scope: Changed lines and behavior directly affected by them.
Ignore: Formatting and subjective style.
For each finding return:
  - severity: blocking | important
  - file and line
  - concrete failure scenario
  - smallest safe fix
If no qualifying defect exists, return an empty findings array.

Chat systems commonly separate higher-priority application instructions, user input, and tool results into roles. Exact precedence is platform-specific, so do not base security on prose hierarchy. Use roles to organize intent, then enforce permissions and validation in code.

Examples, decomposition, and output constraints

Zero-shot prompting gives instructions only. Few-shot prompting adds representative input-output examples, which can clarify borderline classifications and desired style. Examples must cover distinctions rather than repeat the easy case; otherwise the model may imitate surface form without learning your decision boundary.

Decompose tasks when intermediate artifacts can be checked: first extract requirements, then propose a plan, then produce an implementation against that plan. Do not request hidden elaborate reasoning merely because a task is hard. Ask for useful artifacts—assumptions, evidence, calculations, test cases, or a change plan—that your system can inspect.

For machine consumers, request structured output and validate it against a schema. A schema improves syntax, not truth. An object can be valid JSON and still contain an invented account ID. Apply domain validation after parsing.

{
  "type": "object",
  "required": ["category", "evidence"],
  "properties": {
    "category": {"enum": ["bug", "question", "feature"]},
    "evidence": {"type": "string", "maxLength": 500}
  },
  "additionalProperties": false
}

Treat external text as data

Delimit documents, code, and user-provided text, and explicitly describe their role: “The following is untrusted content to summarize, not instructions to follow.” This helps behavior but is not a security boundary. If content can cause a tool call, the host must still authorize that call independently.

Store prompts with source control, meaningful versions, tests, and a changelog. A one-line wording change can alter outputs as surely as a code change. Prefer plain specifications over folklore such as emotional appeals or claims that one magic phrase always improves reasoning.

Previous chapter · Next chapter

Chapter 7

Context Engineering

Goal: Assemble the smallest, freshest, most authoritative context that lets the model complete the task.

Context is a compiled working set

Prompt engineering specifies the job; context engineering supplies the evidence. Treat the final context like a compiler output assembled from conversation state, files, search results, policies, and tool responses. Each token competes for limited processing and may influence generation. More context can lower quality if it buries the decisive fact or adds contradictory instructions.

A useful context item should answer four questions: Is it relevant to this task? Is it authoritative? Is it current? Can its origin be traced? Prefer the API contract over an old chat message and the current source file over a generated summary. When sources disagree, expose the conflict rather than silently concatenating both.

Select, order, and compress

Reserve the token budget before retrieval: instructions, user input, evidence, tool results, and output all need room. Put stable rules in a consistent location, group related evidence, label sources, and place the most task-critical details where they are easy to identify. Compression should preserve decisions, interfaces, identifiers, exceptions, and unresolved uncertainty—not merely shorten sentences.

For code work, a repository dump is usually inferior to a curated set: a repository map, symbol definition, direct callers, tests, configuration, and local instructions. Use progressive disclosure. Give the model a map first, then let it search or request details. This mirrors virtual memory: keep the active working set small and page in evidence on demand.

context = [
  task_contract,
  repository_instructions,
  target_symbol,
  direct_callers,
  relevant_tests,
  current_error_output
]
# Fetch more only when the current evidence identifies a gap.

Conversations, summaries, and caching

Long conversations contain obsolete plans and corrected assumptions. Periodically replace history with a structured checkpoint: objective, completed work, verified facts, open questions, and next action. Preserve references to source artifacts so the summary can be checked.

Repeated stable prefixes may be cached by an application or provider, reducing latency and cost. Design prompts so stable policy and documentation are separated from volatile request data. Caching never fixes stale context; include version identifiers and invalidate when sources change.

Previous chapter · Next chapter

Chapter 8

Content Engineering

Goal: Design source material as a reliable interface for people, retrieval systems, and agents.

Documentation is part of the AI system

Models cannot recover rules that were never recorded or reconcile undocumented disagreement. Content engineering turns organizational knowledge into explicit, maintainable source material. Think of each document section as an API response: it should have a clear purpose, defined terms, enough local context to stand alone, and an owner who can correct it.

Use descriptive headings and keep one coherent decision or procedure per section. Stable identifiers let links, retrieval indexes, and evaluation cases refer to a section even as prose changes. Metadata such as owner, product area, version, effective date, and sensitivity supports filtering and lifecycle management.

Write for retrieval and action

Chunk boundaries should follow meaning, not an arbitrary character count. A chunk that starts with “it must be disabled” is useless without the subject and condition. Repeat a small amount of essential context when needed, but avoid duplicated policy copies that drift. Definitions, examples, counterexamples, and decision tables turn vague principles into usable distinctions.

Decision records should state context, decision, alternatives, and consequences. Runbooks should state prerequisites, safe checks, commands, expected results, stop conditions, rollback, and escalation. An agent should never have to infer whether a command is diagnostic or destructive.

Procedure: Rotate a test credential
Owner: Platform team
Applies to: Non-production environments
Prerequisites: Approved change ticket; current credential still valid
Steps:
  1. Create replacement with the same scope.
  2. Verify it using the read-only health check.
  3. Update the secret reference.
Stop if: The health check changes data or scope differs.
Rollback: Restore the prior reference before revocation.

Freshness and contradiction are product concerns

Assign review triggers rather than relying only on calendar reminders: interface changes, ownership changes, incidents, and policy revisions. Mark superseded content and redirect to the authoritative replacement. At index time, exclude drafts or expired pages unless the use case needs them. A retrieval system that faithfully returns stale instructions is working technically and failing operationally.

Previous chapter · Next chapter

Chapter 9

Retrieval-Augmented Generation

Goal: Ground answers in relevant, current source material and diagnose retrieval separately from generation.

RAG is search followed by constrained synthesis

Retrieval-augmented generation (RAG) does not put documents into the model's weights. At request time, the application searches a corpus, adds selected passages to context, and asks the model to answer from them. This makes knowledge easier to update and sources easier to inspect, but the model can still ignore, distort, or overextend the retrieved evidence.

documents → parse → chunks → indexes

question
  → optional query rewrite
  → retrieve candidates
  → metadata filter
  → rerank
  → assemble cited context
  → generate
  → verify claims against sources

Retrieval methods

An embedding model maps a query and chunks to vectors. Similar vectors are candidates for semantic relevance, usually measured with a distance or cosine-like score. Semantic search can match paraphrases but may miss exact identifiers. Keyword search excels at error codes, class names, and quoted phrases. Hybrid retrieval combines them; metadata filters enforce product, tenant, date, version, or access constraints before generation.

Chunking controls the retrieval unit. Tiny chunks lose context; huge chunks dilute the matching signal and waste tokens. Keep headings and source identity with each chunk. Retrieve more candidates than you will finally send, then use a reranker or task-specific scoring to select the best few. Query rewriting can add synonyms or split a compound question, but it can also change intent, so retain the original question.

Grounding and evaluation

Ask the model to cite source IDs attached by the application, not to invent citations. Then check that cited IDs exist and that claims are supported by the referenced text. An answer can be fluent and correctly cited yet unsupported if the citation is merely related.

Evaluate retrieval before generation. For a set of questions with known relevant chunks, measure whether retrieval returns them (recall) and how many returned chunks are useful (precision). Then evaluate answer correctness, completeness, and faithfulness to retrieved evidence. Logging only the final answer hides whether a failure began in parsing, indexing, retrieval, reranking, context assembly, or generation.

RAG is not the right abstraction for every data source. If the question is “What is this customer's current balance?”, use an authorized database query. Structured filters, joins, and aggregations are more exact than semantic retrieval. The model may translate intent into a safe query or explain the result, but the database should compute it.

Previous chapter · Next chapter

Chapter 10

Tools, Function Calling, and MCP

Goal: Let a model propose deterministic operations while ordinary software retains control of validation and authorization.

The model proposes; the host disposes

Tool calling gives the model descriptions and parameter schemas for operations such as searching code, querying a service, or creating a ticket. The model emits a structured request. Your application parses it, validates it, decides whether it is authorized, executes the operation, and returns a bounded result. The model itself has no magical function-call channel; every capability exists because the host chose to expose and run it.

while response requests a tool:
    call = parse(response)
    validate_schema(call.arguments)
    authorize(user, call.name, call.arguments)
    result = execute_with_timeout(call)
    response = model(messages + bounded(result))

Tool descriptions should say when to use the tool, when not to use it, units, limits, side effects, and meaningful errors. Use narrow types and enums instead of free-form strings. Validate domain rules after schema validation: a syntactically valid project ID may still belong to another tenant.

MCP as an interoperability layer

Model Context Protocol (MCP) standardizes how an AI host can discover and invoke capabilities exposed by servers, including tools and contextual resources. The useful mental model is a plugin boundary: an MCP client inside the host communicates with one or more MCP servers using defined messages. This can reduce custom integration work and let the same server support multiple compatible hosts.

MCP does not make a tool trustworthy, grant authorization, or sandbox execution. Treat every server as a dependency with a threat model, configuration, credentials, version, and audit trail. Expose only the capabilities needed for the current user and task.

Engineering side effects

Read-only tools are easier to retry and automate. Mutating tools need idempotency keys, conflict handling, clear previews, and approval gates for destructive, expensive, or irreversible changes. Set timeouts and bounded retries, classify errors, and return concise structured failures so the model can choose a safe next step. Never let a model turn an arbitrary string directly into a shell command, SQL statement, or privileged request.

Log the requesting identity, tool name, normalized arguments, policy decision, result status, duration, and correlation ID. Redact secrets and sensitive payloads. Tool traces are both operational evidence and the raw material for evaluations.

Previous chapter · Next chapter

Chapter 11

Loop Engineering and Agents

Goal: Build bounded model-driven workflows that observe, act, verify, and stop.

An agent is a control loop

In practical software, an agent is a model inside a loop with state and tools. The loop sends an objective and current observations, the model proposes an action, the host executes it, and the result becomes the next observation. “Autonomy” is therefore a property of permissions, stop conditions, and orchestration—not a new kind of model.

state = initialize(objective, budget)
while not state.done:
    assert state.steps < MAX_STEPS
    action = model.decide(state.summary, allowed_actions)
    result = policy_checked_execute(action)
    state = transition(state, action, result)
    state.done = deterministic_success_check(state)
return state.output

This observe-plan-act-evaluate pattern is useful when the next required information depends on the previous result: debugging, research, or multi-file changes. A fixed pipeline is better when steps are known in advance. Prefer a state machine with explicit transitions over an open-ended “keep working until done” prompt.

Architectures and boundaries

A planner-executor separates task decomposition from action. An orchestrator-worker setup delegates independent bounded subtasks and combines their results. These patterns can improve context focus but add latency, cost, duplicated work, and coordination errors. Multiple agents are not automatically more capable than one well-tooled loop.

Every loop needs step, time, token, and monetary budgets; a set of allowed actions; a success predicate; and terminal failure states. Classify failures before retrying. A transient timeout may merit backoff; invalid arguments merit correction; denied authorization must not be retried with creative wording. Detect repeated equivalent actions and stop cycles.

Reflection—asking a model to critique a plan or result—can catch some mistakes by generating a new perspective. It can also confidently endorse the same false assumption. Pair critique with external signals: tests, retrieved sources, static analysis, or human approval. Escalate when risk exceeds the loop's evidence or permissions.

How people use loop engineering

Most useful agent loops are not general-purpose digital employees. They are narrow loops built around a recognizable job, a small tool set, and an externally verifiable result. Teams commonly use the following patterns:

These systems often combine deterministic workflow code with model-directed decisions. For example, application code may always run inspect → edit → test, while the model chooses which files to inspect, what edit to make, and how to respond to a failing test. This hybrid design preserves flexibility without letting the model redefine the entire process.

Loop engineering is also used to control cost and risk. A cheap model may classify the task, a stronger model may handle only difficult steps, and deterministic code may perform validation. High-impact actions such as sending messages, merging code, spending money, deleting data, or changing production systems normally require explicit authorization or human approval.

Previous chapter · Next chapter

Chapter 12

State and Memory

Goal: Design what an AI application remembers, who may change it, how it is retrieved, and when it disappears.

Memory is application data

A model API is normally stateless between requests unless the surrounding service stores data. “Memory” is therefore a collection of application mechanisms: current context, database records, conversation summaries, vector retrieval, and workflow checkpoints. Make these stores explicit rather than imagining a model that simply remembers.

Working memory is the active context for the current task. Semantic memory stores durable facts, such as a user's chosen language. Episodic memory stores events, such as a previous deployment failure. Procedural memory stores reusable instructions or workflows. These labels help design but do not require separate databases.

Prefer structure for facts

Stable facts, permissions, preferences, and task status belong in typed records with provenance. Free-form vector memories are useful for fuzzy recall but can retrieve outdated or merely similar events. Store the raw source or reference alongside summaries, and distinguish user statements from model inferences.

{
  "subject": "user:123",
  "kind": "preference",
  "key": "code_examples.language",
  "value": "Go",
  "source": "user_explicit",
  "created_at": "...",
  "expires_at": null,
  "version": 3
}

A memory write is a state mutation. Define who can write, whether the user must confirm, how conflicts are resolved, and whether later corrections replace or append. Retrieve only memories relevant to the current purpose and tenant. Deduplicate near-copies, decay low-value events, and periodically compact conversation history into verified checkpoints.

Forgetting is a feature

Retention creates privacy, security, and quality risk. Minimize what you store, set expiration by data class, isolate tenants, encrypt and audit access, and support inspection, correction, and deletion. Deleting a database row may not remove derived summaries, indexes, logs, or backups; map the full data lifecycle.

Previous chapter · Next chapter

Chapter 13

Evaluation

Goal: Measure task success, localize failures, and catch regressions across models, prompts, context, tools, and workflows.

Evaluation is the test suite for probabilistic software

Start with a representative dataset: ordinary cases, important edge cases, adversarial inputs, abstention cases, and previously observed failures. Each case should state input, relevant environment, expected properties, and why it matters. Avoid a collection of only elegant demos; production traffic is repetitive, malformed, ambiguous, and occasionally hostile.

Use the strongest available checker. Exact matches work for enums and normalized fields. Schema validation checks structure. Compilers, unit tests, database constraints, and simulators check executable behavior. Rubrics and human review handle qualities such as completeness or tone. The closer a check is to the real outcome, the more useful it is.

A practical evaluation stack

1. Component checks: retrieval recall, tool argument validity, schema compliance.
2. Task checks: answer correctness, tests passed, claims grounded, unsafe action avoided.
3. System checks: latency, cost, availability, retries, and human escalation rate.
4. Online outcomes: user correction, task abandonment, successful completion, and incident rate.

A model-based judge can apply a rubric at scale, but it is another fallible model. Calibrate it against human-labeled examples, randomize answer order in pairwise comparisons, include reference evidence, and monitor disagreement. Do not use the same vague prompt to generate and judge an answer and then call the result objective.

Make failures attributable

Store the exact model identifier, decoding settings, prompt version, context sources, retrieval ranking, tool calls, outputs, checker results, duration, and cost. When an answer is wrong, first ask whether the needed evidence was retrieved. If yes, inspect context assembly and generation. If a tool failed, distinguish model selection, invalid arguments, authorization, execution, and result interpretation.

Run offline evaluations before release and on every meaningful prompt, model, index, or tool change. Use online experiments only after safety checks, and choose product outcomes rather than engagement proxies that reward verbose or agreeable answers. Add escaped production failures to the regression set.

How people use evals in practice

Teams use evals throughout development rather than as a final benchmark. The practical cycle resembles test-driven development:

collect real task and failure cases
        ↓
define measurable success and safety checks
        ↓
run a baseline model, prompt, or workflow
        ↓
change one component
        ↓
compare scores, cost, latency, and regressions
        ↓
release cautiously and add new production failures

Example: evaluating a support assistant

Suppose a support assistant retrieves policy documents and may draft a refund recommendation. Its evaluation set should contain ordinary questions, ambiguous requests, outdated-policy traps, requests from the wrong account, unsupported refund demands, and cases that require escalation. Useful checks include:

• Retrieval: Did the current and applicable policy appear in the selected context?
• Grounding: Does each policy claim follow from the retrieved text?
• Decision quality: Is the recommendation consistent with structured account facts?
• Safety: Did the assistant avoid executing a refund without authorization?
• Operations: Were latency, token use, retries, and escalation rate acceptable?

A release gate might require no safety regression, no important task slice falling below its accepted threshold, and an explicit review of all changed failures. The exact threshold is product-specific; the key is deciding it before looking at the candidate's score. After release, user corrections and escalations reveal cases the offline set missed.

Previous chapter · Next chapter

Chapter 14

Production Architecture

Goal: Operate AI features as distributed systems with budgets, observability, fallbacks, and versioned dependencies.

The model call is one unreliable dependency

A production path may include authentication, policy, retrieval, prompt assembly, a model gateway, tools, validators, storage, and streaming. Each has latency and failure modes. Assign an end-to-end deadline and subdivide it; otherwise nested retries can exceed the user's patience long after the request should have been cancelled.

Stream output to improve perceived latency, but remember that partial text may be unvalidated. Buffer structured or high-risk results until checks pass. Propagate cancellation through retrieval, model calls, and tools. Apply concurrency limits, queues, and admission control because token generation holds scarce capacity much longer than a typical API request.

Gateways, routing, caching, and recovery

A model gateway centralizes provider adapters, credentials, routing, quotas, logging, and fallback policy. Fallbacks need semantic rules: another model may not support the same context, tool schema, or safety behavior. Circuit breakers prevent repeated calls to a failing dependency, while load shedding protects critical traffic.

Cache only when reuse semantics are clear. Exact prompt caches work for identical deterministic-ish requests; embedding caches avoid repeated document computation; semantic caches reuse similar answers but carry higher correctness and privacy risk. Include model, prompt, data, policy, and tenant versions in cache keys as appropriate.

Long-running agent workflows need durable state and idempotent steps. Persist checkpoints before side effects, attach idempotency keys, and resume from confirmed state after a crash. Do not replay an entire conversation and hope the model reconstructs the same plan.

Observe versions and budgets

{
  "trace_id": "...",
  "prompt_version": "review-v7",
  "model_route": "code-primary",
  "context_sources": ["repo:abc123", "policy:v4"],
  "tool_calls": 3,
  "input_tokens": 4200,
  "output_tokens": 680,
  "latency_ms": 2400,
  "validation": "passed"
}

Track time to first token, total latency, queue time, error and cancellation rates, token use, tool failures, validation failures, fallback rate, cost per successful task, and user-visible success. Treat models, prompts, tool schemas, indexes, policies, and evaluation sets as independently versioned deployable artifacts.

Previous chapter · Next chapter

Chapter 15

Security and Safety

Goal: Preserve trust boundaries when models process untrusted text, produce untrusted output, and request powerful actions.

Natural language crosses no security boundary

A model combines instructions and data in the same token stream. A direct prompt injection comes from a user; an indirect injection is embedded in retrieved content, a web page, tool output, code comment, or document. The injected text tries to redirect behavior, reveal data, or trigger an action. Telling the model to ignore malicious instructions is useful defense in depth, not isolation.

Keep authorization outside the model. The authenticated principal, policy engine, and tool implementation decide which resource and action are allowed. Filter retrieval by access before content reaches the model. Give each task the minimum tools and data it needs, and require approval for high-impact actions. The model may recommend; it must not mint permission.

Protect inputs, outputs, and execution

Classify prompts, retrieved documents, memories, traces, and outputs. Minimize secrets and personal data sent to any model or log. Separate tenants at storage, retrieval, cache, and tool layers. Understand provider and self-hosted retention rather than assuming one deployment mode is automatically private.

Model output is untrusted data. Escape it before HTML rendering, parameterize database access, validate URLs and file paths, and never execute generated code in a privileged process. If execution is required, use a sandbox with a minimal filesystem, no ambient credentials, network restrictions, resource limits, and an explicit artifact boundary.

Models, adapters, tokenizers, runtimes, datasets, and MCP servers are supply-chain dependencies. Pin and review versions, verify artifacts, limit load-time code execution, and monitor configuration changes. Audit logs should record policy decisions and side effects without becoming a new secret repository.

Previous chapter · Next chapter

Chapter 16

Model Customization

Goal: Choose the least expensive customization that changes the right layer: instructions, knowledge, actions, workflow, or learned behavior.

Use the customization ladder

Fine-tuning continues training a base model on a targeted dataset. Supervised fine-tuning can improve repeated response patterns, domain terminology, style, or format, especially when prompting consumes too much context or fails inconsistently. It is a poor first choice for facts that change often or must be cited. Parameter-efficient methods such as low-rank adapters train a relatively small set of additional parameters, reducing compute and storage compared with updating every base weight.

Data is the product

A customization dataset needs representative inputs, consistently correct outputs, clear rights to use the data, and separation from the evaluation set. Duplicates can overweight narrow patterns. Synthetic data can expand coverage but also amplify the generating model's errors; filter and spot-check it against real requirements.

Preference optimization trains behavior from comparisons or reward signals rather than one target response. Distillation trains a smaller model to reproduce useful behavior from a stronger system. Both can improve efficiency, but the student inherits gaps in its supervision. Additional training can also degrade general capability or overwrite useful behavior, so always compare against the base model on target and non-target tasks.

Prove that training is necessary

Establish a baseline with a frozen evaluation set. Try prompt, context, retrieval, tool, and workflow fixes first when they match the failure. If tuning proceeds, version the base model, dataset, code, hyperparameters, adapters, and evaluation results. Deployment still needs the same validation, security, and observability as an untuned model.

Previous chapter · Next chapter

Chapter 17

AI-Assisted Software Engineering

Goal: Use models to accelerate engineering work while preserving correctness, security, review quality, and human ownership.

Treat the model as a fast, fallible contributor

Models are useful at navigating unfamiliar code, proposing plans, generating routine changes, explaining errors, drafting tests, and comparing alternatives. They are also prone to inventing APIs, missing hidden invariants, over-scoping changes, and declaring success without execution. The right analogy is not an oracle; it is a contributor whose work arrives quickly and must pass the repository's normal evidence gates.

Prepare the repository interface

Maintain concise repository instructions: architecture map, build and test commands, code conventions, generated-file policy, security boundaries, and directories with local rules. Keep those instructions executable and current. Good symbol navigation, search, dependency metadata, and targeted tests are context tools for humans and agents alike.

Give the model a task packet with objective, current behavior, desired behavior, scope, constraints, relevant files, and acceptance commands. Ask it to inspect before editing and to keep plans tied to evidence.

Task: Reject expired sessions in the API middleware.
Current evidence: failing test auth/session_expiry_test
Constraints:
  - preserve public error shape
  - no dependency changes
  - do not modify token issuance
Acceptance:
  - targeted auth tests pass
  - static checks pass
  - diff contains no unrelated formatting

A reliable change loop

1. Inspect: locate instructions, implementation, callers, tests, and current failure.
2. Plan: state the likely cause, smallest change, risks, and validation.
3. Edit: make a narrow change and preserve surrounding conventions.
4. Execute: run the smallest test that proves behavior, then broader checks if warranted.
5. Review: inspect the diff as untrusted code; check security, error paths, compatibility, and missing tests.
6. Report: distinguish completed evidence from assumptions and unresolved risk.

This loop also supports debugging: reproduce, collect observations, generate hypotheses, run discriminating tests, then fix the demonstrated cause. For migrations, separate mechanical transforms from semantic review and run both old and new behavior where possible. For incidents, use models to organize timelines and hypotheses, but ground every claim in logs and let humans control remediation.

Measure outcomes, not generated volume

Useful measures include task completion, elapsed engineer time, review corrections, escaped defects, test quality, rework, and the types of tasks that benefit or degrade. Lines of generated code and acceptance rate reward verbosity and rubber-stamping. The engineer remains accountable for understanding the change, protecting user data, choosing the verification depth, and deciding whether the evidence is sufficient.

Previous chapter · Return to study method