|
[Webinar] How to stop babysitting your agents (Sponsored)
 |
Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in the correction loops.
More MCPs, rules, and bigger context windows give agents access to information, but not understanding. The teams pulling ahead have a context layer to give agents exactly what they need for the task at hand.
Join us for a FREE webinar on Sep 2 to see:
Where teams get stuck on the AI maturity curve and why common fixes fall short
How a context layer solves for quality, efficiency, and cost
Live demo: the same coding task with and without a context layer
If you want to maximize the value you get from AI agents, this one is worth your time.
When you type a follow-up question into an AI chat and press Enter, nothing happens for a second or two. Then the answer appears in a quick succession of words. It appears much faster than what the initial pause indicated.
This pause is not dead time. In a typical LLM, a single message passes through roughly a dozen distinct stages before a reply starts to appear. Two very different kinds of computing work happen in the background to make this possible. Some key points about this journey are as follows:
The model never receives the message as it was typed.
It has no memory of the conversation. The history on screen is rebuilt from scratch every turn.
It shares a machine with strangers, and the group it lands in can affect the reply.
Once a word has been sent, the model cannot take it back.
In this article, we are going to look at this entire journey in detail. Here’s what we will cover:
How is the input to the model assembled?
Why is every input message to the model independent?
Performing safety checks on the input
How does the model understand the words?
How is the model shared across multiple conversations?
Prefill and decode steps
Caching the existing calculations
Streaming and guardrails
How does the model run various tools?
 |
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
How the Input to the Model is Assembled?
The first key point to understand is that the sentence typed into the box is not the exact thing that reaches the model. What reaches the model is a document assembled around that sentence before the request is sent.
This document contains several ingredients, which are as follows:
A system prompt, which is a block of instructions written by the LLM provider. These instructions tell the model how to behave during the conversation.
Definitions of any available tools, describing what each one does and what inputs it accepts.
Anything stored as memory from earlier sessions.
Documents pulled from a knowledge base in cases where retrieval is the key.
The full conversation so far.
Finally, the new message.
The process of deciding what goes into this document in what order, and what gets left out, is a discipline in its own right known as context engineering. It is not a simple matter of filling a container. Models have a finite attention budget, and every token added draws it down.
Accuracy degrades as input grows longer, even on tasks that are quite simple. The decline is gradual. A longer prompt does not break anything outright, but the underlying precision falls away.
The approach used towards this discipline of context engineering leads to scenarios where two products built on the same underlying model, given word-for-word the same question, return different answers. The model might be identical, but the document wrapped around the question is not.
Different providers also differ in when they gather the material for the document. Some retrieve everything up front. Others hand the model lightweight references, file paths, or stored queries, and let it pull in what it needs while working. The first is faster, and the second wastes fewer tokens on material that might be irrelevant.