Cost Control for LLM Products: The Cheapest Token Is the One You Never Send

Most advice on LLM cost starts with tokens, caching and cheaper models. Those matter, but they are the third and fourth levers, not the first. The first one is deciding whether a step needs a model at all.

I learned that on a production automation I built with Claude Code for a Dubai field services company. The loop that ran live in July read the dispatcher’s daily job feed, matched jobs to the booking portal, skipped cancellations and sent pre- and post-visit messages, all without a person touching it. On its best day it detected 42 jobs and delivered 39 messages with 0 errors. There was no LLM in the runtime. I had written down the reason on 30 June: an LLM there was overkill and costly. The model was kept for drafting and for handling replies, where language is the actual problem.

This article is the order of operations I use now.

TL;DR

• Cost control starts with a design question (does this step need a model?) before it becomes a tokens question.
• Fix the unit of cost first: cost per successful outcome, not cost per request.
• Caching, routing and prompt trimming each attack a different part of the bill; apply them in that order of effort versus payoff.
• Every cost cut needs an eval gate, or you are trading money for silent quality loss.
• Set a cost ceiling per feature before launch, the same way you would set a kill criterion.

What is the right unit of cost for an LLM product?

The right unit is cost per successful outcome: what you spend to get one booking confirmed, one ticket resolved or one document correctly extracted. Cost per request or per million tokens hides retries, failures and human review, which is where most of the real spend sits.

A cheap model that fails 30% of the time and triggers a human escalation can cost more per outcome than an expensive model that succeeds on the first pass. I track four numbers per feature: model spend, retry rate, human-review rate and successful outcomes. Divide total cost (including the human minutes) by successful outcomes and you have a figure a finance person and an engineer can both argue about.

This also keeps you honest about the pricing page. Provider prices change; your retry rate and review rate are properties of your design, and they are the part you control.

Does this step need a model at all?

Only steps where the input is unstructured language, or where the answer cannot be written down as a rule, need a model. Anything with a stable format, a lookup, a threshold or a schedule is cheaper, faster and more reliable as ordinary code.

The field services loop above is the clean example. The dispatcher’s feed had a predictable shape, so matching jobs to the booking portal was deterministic logic. Putting a model in that path would have added per-run cost, latency and a new failure mode (a plausible but wrong match) in exchange for flexibility the task did not need. Where language did matter, drafting messages and reading free-form replies, the model earned its place.

At Mashkor the same split existed in the other direction. The recommendation engine ran on Google Vertex AI in production and moved activations by 15%, because ranking what to show a user is a problem rules cannot solve well. A good test: if you can write the acceptance test as an exact-match assertion, try code first.

How much can prompt caching save?

Prompt caching lets the provider reuse a prompt prefix across requests instead of reprocessing it. On Anthropic’s API, cache reads are billed at 0.1 times the base input price on most models, while a 5-minute cache write costs 1.25 times and a 1-hour write costs 2 times (Anthropic prompt caching docs). Some newer models price reads even lower.

Those multipliers set the break-even. A write costs a quarter more than a normal read, and every hit after it costs a tenth. If a long, stable prefix (system prompt, tool definitions, reference documents) is reused even two or three times inside the cache window, you are ahead. If every request has a different prefix, caching gives you nothing.

The design rule that follows: put what never changes first, and what changes last. System instructions and tool definitions at the top, retrieved documents next, the user’s message at the end. One timestamp injected near the top of a prompt invalidates the whole prefix, and it is the most common way I see caching silently fail. Check the cache fields in the usage response rather than assuming the discount is arriving.

When should you route to a smaller model?

Route to a smaller model when the task is narrow, the output is easy to verify and a wrong answer is cheap to catch. Classification, extraction into a fixed schema and intent detection are typical. Keep the larger model for open-ended reasoning and for the steps where an error is expensive.

Routing is a product decision as much as an engineering one, because it decides where you accept lower quality. I decide it per step, not per product:

Step type	Typical choice	Why
Intent or category classification	Small model	Narrow output, easy to check against labels
Extraction into a fixed schema	Small model plus validation	Schema validation catches malformed output
Drafting customer-facing text	Mid or large model, human gate if sensitive	Tone and accuracy both matter
Multi-step planning or ambiguous reasoning	Large model	Errors compound across steps
Anything with a fixed rule	No model	Code is cheaper and deterministic

A fallback pattern works well: the small model answers, a cheap check scores confidence or validates the output, and only failures escalate to the larger model. This is the same shape as the confidence and sensitivity gate I use for human review, applied to cost instead of risk.

How do you cut tokens without hurting quality?

Cut tokens by removing what the model does not need: trim retrieved context to the passages that matter, cap conversation history with summaries, and ask for structured, short outputs. Output tokens usually cost more than input tokens, so a tighter response format often saves more than a shorter prompt.

Three habits that pay off consistently:

1. Retrieve less, rank better. Sending the top 3 relevant passages instead of the top 20 cuts input cost and often improves answers, because there is less noise to ignore.
2. Bound the output. Ask for a label, a JSON object or a two-sentence answer when that is all the downstream step uses.
3. Summarise long histories instead of replaying them in full on every turn.

Each of these is a change to the product’s behaviour, so none of them ships without a test. Which brings up the guardrail on the whole exercise.

Why does every cost cut need an eval gate?

Because cost cuts fail quietly. A smaller model, a shorter prompt or a trimmed context window can degrade quality in ways that no dashboard flags, and the saving shows up immediately while the damage shows up weeks later in complaints or churn.

My rule is the one from evals for agents that act: a change earns promotion by passing a defined set of cases, including adversarial ones, and then by matching its predicted behaviour in a small live test. A cost optimisation is just another change. Run the existing eval suite on the cheaper configuration, compare against the current one, and promote only if the quality gap is inside a tolerance you set before looking at the results.

Pre-committing the tolerance matters. If you pick the threshold after you see a 40% saving, you will find a reason to accept it.

How do you set a cost ceiling before launch?

Set a maximum cost per successful outcome and a monthly spend cap per feature before launch, and tie both to a decision: if the feature exceeds the ceiling for a defined period, it gets re-architected or switched off. This is the cost version of a kill criterion.

The same discipline runs through my own work: I have assessed 60+ candidate systems for production and promoted 4, because the gates were written before the results came in. Cost belongs in those gates. An agent feature with no spend ceiling is an open-ended liability, particularly once a retry loop or a runaway tool call can multiply usage overnight. Practical controls:

• A per-request token cap and a per-user daily cap.
• A circuit breaker on retries, so one bad input cannot loop.
• An alert on cost per outcome, not only on total spend, so a quality problem that triggers retries shows up as a cost signal.

The order of operations

1. Ask whether the step needs a model. Remove it if not.
2. Define cost per successful outcome and measure it.
3. Structure prompts so the stable prefix is cached.
4. Route narrow, verifiable steps to smaller models.
5. Trim tokens in context and output.
6. Gate every change behind the eval suite, and cap spend per feature.

Done in that order, most of the saving comes from the first and third steps, and the later ones are tuning. Done in reverse, you spend weeks shaving tokens off a step that should never have called a model.