What "agentic" actually means
A chatbot answers. An agent acts, looks at the result, and decides what to do next. That is the whole distinction, and it is implemented by a loop that is almost disappointingly simple.
Two details in that loop matter more than they look.
One reply can contain several tool calls. When it does, run them and return all of the results in a single message back. Splitting them across several messages teaches the model, within that conversation, that parallel calls do not work — so it stops making them and everything gets slower.
A failed tool is still a result. Return the failure, marked as an error, rather than dropping it. An agent told "that path does not exist" corrects itself on the next turn. An agent told nothing repeats the same call, because from where it sits nothing happened.
Tools are the product
The model is a commodity; everyone has access to roughly the same ones. What differs between a useful assistant and a frustrating one is the tool surface: what it can see, what it can change, and how precisely it can express an intention.
The set that pulls its weight is small:
- Search — find files by name and by content. This is how an agent orients itself in a codebase it has never seen.
- Read — a file, or a range of lines. Ranges matter: reading a 4,000-line file to change one method is how you burn a context window.
- Edit — a precise replacement of known text, not "write the whole file". Whole-file writes destroy anything the model did not think to reproduce.
- Run — tests, build, linter. This is what turns a plausible answer into a verified one.
Context is the real constraint
Context windows are large now, which has not made context management less important — it has made it less obvious when you are doing it badly. Everything in the window competes for attention, you pay for all of it on every turn, and a window full of noise produces worse answers than a smaller window full of signal.
In practice, a turn is assembled roughly like this:
The non-obvious part: the things you most want to inject — the current file, the time, a request id — are exactly the things that must go last, because anything placed before the history invalidates it.
Caching, and the silent invalidator
Prompt caching works on an exact prefix match. The provider hashes the request from the beginning; everything that matches a previous request can be served from cache, and the first byte that differs invalidates everything after it.
Which produces a failure mode worth knowing, because nothing errors:
// Looks harmless. Costs you the entire cache, every single turn.
system = $"You are a coding assistant. Current time: {DateTime.Now}";
The timestamp is in the prefix, so the prefix is different on every request, so nothing is ever a hit. The agent still works. It is just several times more expensive and slower to first token, and nothing in the response says so.
The same reasoning explains a rule that surprises people: do not rewrite history. Editing or re-summarising earlier turns changes the prefix, so the cache dies and the cost of the conversation jumps. Append; do not revise.
When the conversation gets too long
Two different remedies, often confused:
- Clearing removes old content outright — typically stale tool results, which are the bulkiest and least useful thing in a long session.
- Compaction replaces old turns with a summary, keeping the thread of the conversation at a fraction of the tokens.
Clearing is cheaper and safer; reach for it first. Compaction keeps more meaning, at the cost of a summarisation pass and a rewritten prefix.
Permissions: the UX problem
An agent that can edit files and run commands is, in security terms, a program executing instructions derived from text. The hard question is not "can it do damage" but "how often do we interrupt the human to ask".
Modes that proved genuinely useful, rather than theoretically complete:
- Plan first. The agent may read anything and write nothing until it has stated what it intends to do. This catches the "confidently solving the wrong problem" failure, which is the expensive one.
- Approve edits. Reads and searches run freely; every write is shown as a diff and waits. This is the default I keep coming back to.
- Approve commands. Edits flow, but anything touching a shell asks.
- Autonomous. For a sandbox or a throwaway branch, where the cost of a bad action
is a
git reset.
Make it check its own work
The single biggest difference between an assistant that saves time and one that costs it is whether it verifies. A model producing a change that looks right is unreliable. A model producing a change and then running the test suite is doing engineering.
So: give it the ability to run tests, build, and type-check — and ask for the change to be verified rather than merely written. The loop above is what makes this possible: it sees the failure output and gets another turn.
Where it helps, where it does not
Four questions decide whether an agent is the right shape for a task:
- Is it multi-step and hard to fully specify up front? If you could write the exact steps, write the script instead — it is cheaper and deterministic.
- Is the outcome worth the latency and cost? Agents are slower per task than a single call, because they take several turns.
- Is the model actually good at this? Be honest about the task type.
- Can a mistake be caught and undone? Version control and tests make errors cheap. Production data makes them expensive.
Where it genuinely earns its place, in my experience: exploring an unfamiliar codebase, mechanical refactors across many files, writing the tests nobody wants to write, turning a stack trace into a diagnosis, and first drafts of code that has a clear specification.
Where it does not: anything where the hard part is deciding what to build, anything load bearing that nobody will review, and anything whose correctness you cannot check. The agent is fast at producing plausible work. Deciding whether plausible is correct is still the job.