Spending less on AI tokens without getting worse answers

Most teams try to cut the bill by shortening prompts, which is the one lever that trades quality for money. The four that do not are caching, retrieval, routing and measurement — and the first one is usually broken without anybody noticing.

Where the tokens actually go

The intuition is that the model's answers cost the money. In a long-running agent the opposite is true, and it is not close.

Every turn resends the entire conversation. Turn one sends a system prompt, some context and a question. Turn twenty sends the system prompt, the context, and nineteen previous exchanges including every tool result the model has ever seen. Input grows linearly with turn count, and you pay for all of it, every time.

TOKENS BILLED PER TURN turn output input: the whole history, again
Output per turn barely moves. Input is a staircase, because the conversation is resent in full on every request. Optimise the staircase.

Which is good news, because input is also the cheapest thing to fix. You are not being asked to make the model terser. You are being asked to stop paying full price for text it has already read.

Prompt caching, and how it silently breaks

Prompt caching lets the provider keep the processed form of a prefix and charge a fraction of the normal input price when the next request starts with exactly the same bytes. The saving is large and the mechanism is brittle, in a specific way worth understanding.

It is a prefix cache. It matches from the beginning of the request and stops at the first byte that differs. Everything after that point is reprocessed at full price.

So a single dynamic value near the top — a timestamp, a user name, a "today is" line — invalidates the entire rest of the prompt on every request. The cache reports a hit of approximately zero and nobody looks, because nothing fails. The bill just never goes down.

The number to watch Cache-read tokens near zero from turn two onwards means something near the top of your prompt is changing every request. It is the single most valuable number in the usage response, and almost nobody logs it.

Order the prompt by how often it changes

Once you know it is a prefix cache, the design falls out: sort the prompt from most stable to least, and put the breakpoints where the stability changes.

ORDERED BY HOW OFTEN IT CHANGES core rules never changes domain pack per case type documents per matter history this message the only new bytes cached →→→ up to here, every turn ONE TIMESTAMP AT THE TOP "today is …" changes hourly core rules domain pack history this message nothing cached — the prefix diverges in the first block, so every block after it is full price
The cache matches from the start and stops at the first difference. One volatile line at the top costs you the entire prompt.

Three rules follow from the picture, and they are the whole technique:

  • Nothing dynamic goes in the system prompt. Not the date, not the user, not a counter. Anything that changes per request is prepended to the newest user message, which sits at the end of the prefix where it invalidates nothing.
  • Put a breakpoint on the last stable block. Static rules, then the domain pack, then retrieved documents, then history. Each boundary is where the cache can resume.
  • Anything that changes the request shape invalidates everything. Changing the model, or switching a reasoning-effort setting mid-conversation, starts a fresh cache. If effort is going to vary by phase, pin it per phase and change it once, not per turn.

Stop pasting the codebase

The most expensive habit in agentic coding is answering "where is X handled?" by reading six files into the prompt and hoping the model finds it. You pay for every line, you pay again on the next turn because the whole history is resent, and the answer gets worse as the repository grows — because what you pasted is a sample, not the system.

The alternative is to compute the structure once, offline, and query it. I use graphify, which parses the repository with tree-sitter and writes a knowledge graph: every symbol a node, every call, import and inheritance an edge. The code pass is local and deterministic — no model is involved in building it, so the graph itself costs nothing in tokens.

Setting it up

Two commands. The PyPI package is graphifyy with two y's; the command it installs is graphify.

# once, globally
uv tool install graphifyy      # or: pipx install graphifyy
graphify install               # registers the skill with your assistant

graphify install detects which assistant you are running — Claude Code, Cursor, Codex, Gemini CLI and a couple of dozen others — and registers a /graphify skill, so there is no per-assistant configuration. Then, in the repository:

# in your assistant
/graphify .

# three files, in graphify-out/
graphify-out/
├── graph.json        # the graph itself, queried by the CLI
├── graph.html        # clickable, filterable, searchable
└── GRAPH_REPORT.md   # god nodes, communities, key paths

For a shared repository, install the skill into the project rather than your user profile, so everyone on the team gets the same behaviour:

graphify install --project     # writes .claude/skills/graphify/ etc.

Querying it instead of reading files

Three commands cover almost everything I used to burn tokens on:

# what is this, what does it touch, who calls it
graphify explain "ImmigrationService"

# how do these two things actually connect
graphify path "ImmigrationController" "AnthropicService"

# a scoped subgraph for a plain-language question
graphify query "where are refresh tokens rotated"

The output is a few hundred tokens and it is derived from the whole repository rather than from whichever files you guessed at. Each edge is tagged EXTRACTED when it is explicit in the source or INFERRED when resolution worked it out — which matters more than it sounds, because it tells you which parts of the answer are facts and which are deductions.

READ THE FILES IN tens of thousands of tokens six files, resent every turn, still partial QUERY THE GRAPH hundreds one scoped answer tree-sitter AST local, no model graph.json nodes and edges derived answer whole repo, not a sample
The cheaper path is also the more accurate one, because it is computed from the whole repository rather than sampled from the part you happened to open.

Making the assistant prefer it

Building the graph is the easy half. The habit is the hard half, because an assistant left alone will grep. One more command writes the configuration that points it at the graph first:

graphify claude install        # or: graphify codex install, etc.

On assistants that support hooks, that fires before search-style tool calls and before reading source files one at a time, and nudges the assistant toward a scoped graphify query. On the others it writes persistent instructions — AGENTS.md, a Cursor rule — that say the same thing. Graphify Labs put the saving at around 80% of agent token spend; I have not measured that number myself, but the direction is not in doubt once you have watched the two approaches side by side on the same question.

The one operational gotcha A graph is a cache of the code, so it goes stale the moment you refactor. Rebuild it after a batch of changes, and force a full rebuild rather than an incremental one — deletions otherwise leave orphaned nodes behind, and a confident answer about a class you deleted last week is worse than no answer at all.

The principle generalises past code. Anything you can compute deterministically, compute deterministically: an index, a schema, a lookup table, a call graph. A model should be asked to reason, not to look things up.

Cheap work belongs on a cheap model

In most AI products, the majority of calls are not the hard ones. Extracting fields from a document, classifying an intent, summarising a page, transcribing a scan — this is high-volume, low-judgement work, and sending it to your best model is the easiest money you will ever save.

In a document pipeline I work on, extraction runs on a small fast model and the reasoning turns run on a large one. The split is roughly: the small model handles the majority of the calls and a small minority of the cost.

Two practical notes from doing this:

  • Models do not share a parameter surface. A setting the large model requires may be rejected outright by the small one — I have had a 400 saying in as many words that the model does not support the parameter. Make model, token limit and any tuning flags per-request rather than global, and leave the ones that do not apply unset.
  • Normalise cost before you compare anything. Convert every model's input, output, cache-read and cache-write usage into one unit — I use "input-token equivalents on the large model" — weighted by each one's real price. Otherwise "we used 40 million tokens" is a number with no meaning, since a cache-read token and an output token can differ by more than an order of magnitude.

Let the task decide the effort

Reasoning-effort controls are a genuine lever, and the mistake is treating them as a global setting. A matter being filled in from a form does not need deep reasoning; the same matter being drafted into a legal document does.

So pin effort to the phase, server-side, and let it change exactly once as the work moves from intake to drafting. Two things make this work: the decision is never the client's to make, and it flips at a point where the cache was going to be invalidated anyway.

Measure, or you are guessing

Every one of these techniques can be applied perfectly and silently stop working. A refactor moves a dynamic value into the system prompt. A new feature sends the big model where the small one would do. Nothing breaks; the bill goes up.

So record, per turn, one structured line:

var usage = new TurnUsage {
    InputTokens      = r.Usage.InputTokens,
    CacheReadTokens  = r.Usage.CacheReadInputTokens,   // the health metric
    CacheWriteTokens = r.Usage.CacheCreationInputTokens,
    OutputTokens     = r.Usage.OutputTokens,
    Model            = r.Model,
    Effort           = request.Effort,
    StopReason       = r.StopReason,                   // see below
    TtftMs           = firstByte.ElapsedMilliseconds,
    DurationMs       = total.ElapsedMilliseconds
};

Two of those fields earn their place immediately. Cache-read tokens tell you whether your prefix is still intact — graph it, and a refactor that breaks caching shows up as a cliff on the day it shipped. Stop reason tells you when a response was cut off by the token limit rather than finishing, which is how you find the most wasteful failure there is: a long, fully-paid-for response that got truncated, failed to parse, and was thrown away.

A real one A structured-output call had its output limit set to roughly what the answer needed. The model's own reasoning consumed the budget before it finished writing, every response was truncated, the parser rejected the lot, and the UI swallowed the error and simply stopped. Every one of those calls was paid for in full and produced nothing. The fix was a larger budget and surfacing the stop reason as a distinct error — but what made it findable was logging the stop reason at all.

The checklist

  • Nothing dynamic in the system prompt. Dates, names and counters go on the newest user message.
  • Order the prefix stable-first and put cache breakpoints on the boundaries.
  • Log cache-read tokens and watch them. Near zero from turn two means the prefix is broken.
  • Compute what can be computed. A code graph, an index, a lookup table — anything deterministic should not be a model's job.
  • Route by difficulty. Extraction and classification on a small model; judgement on a large one.
  • Normalise usage to one unit by cost before comparing anything or billing anyone.
  • Set the output limit above the answer you expect, and treat a length stop as an error rather than a result.
  • Deduplicate repeated inputs. The same document uploaded twice should be processed once; hash the content and reuse the result.
The summary Shortening prompts trades quality for money, which is why it is the lever people reach for and the one they should reach for last. Caching, retrieval and routing cost you nothing in answer quality — and in the case of retrieval, they improve it.

← All articles