Building an AI feature has never been easier. A working prototype takes an afternoon: pick a frontier model, write a prompt, wire up a few tools, ship it.

Then the invoice arrives.

The prototype that cost a few dollars a day in testing now serves thousands of users. Every request carries a 6,000-token system prompt. The chat history grows with every turn. An agent somewhere is stuck in a loop, calling the same tool forty times. And every one of those calls is going to the most expensive model you have access to, because that’s what you picked on day one.

LLM costs scale in ways traditional infrastructure doesn’t. You pay per token, in and out, on every call. Waste compounds quietly: an extra 2,000 tokens of irrelevant context is invisible on a single request and very visible across ten million of them.

The good news is that most of this waste is avoidable, and you rarely have to trade quality for savings. This guide covers 21 techniques, grouped into four strategies:

1. Reuse, don’t re-pay. Stop paying for the same tokens twice.
2. Right-size every call. Send fewer tokens, get fewer tokens back, and use only as much model as the task needs.
3. Route to the cheapest capable model. Make model choice a runtime decision instead of a hard-coded one.
4. Make cost visible and controlled. You can’t optimize what you can’t see, and you can’t trust what you haven’t capped.

A quick note on numbers: provider pricing changes often, so treat any percentages here as typical ranges and check your provider’s current price sheet before you build a business case.

---

Part 1: Reuse, Don’t Re-pay

The cheapest token is one you’ve already paid for. These techniques stop you from buying the same work again.

1. Prompt Caching

Most production prompts share a large, stable prefix: the system prompt, tool definitions, few-shot examples, and often a block of reference documentation. Only the last part, the user’s question, actually changes between calls.

Prompt caching lets the provider store the processed state of that prefix. When the next request starts with the same prefix, the cached part is billed at a steep discount (commonly 50–90% off normal input pricing, depending on the provider) and the response usually comes back faster too.

The rule that makes it work: put stable content first and variable content last. Anything that changes, such as a timestamp, a user name, or a request ID, will break the cache for everything after it if it sits near the top.

Here’s what explicit caching looks like with the Anthropic Messages API:

response = client.messages.create(
    model=MODEL,
    max_tokens=500,
    system=[
        {"type": "text", "text": SYSTEM_INSTRUCTIONS},
        {
            "type": "text",
            "text": PRODUCT_DOCUMENTATION,  # large and stable
            "cache_control": {"type": "ephemeral"},  # cache everything up to here
        },
    ],
    messages=[{"role": "user", "content": user_question}],  # changes every call
)

Other providers, such as OpenAI, apply prefix caching automatically once a prompt passes a minimum length, so the main job there is to keep your prefix byte-for-byte identical.

Where it pays off most: agents (which resend the same tool definitions on every step), RAG assistants with a fixed instruction block, documentation bots, and any multi-turn chat.

Watch out for: minimum cacheable lengths, short cache lifetimes (often minutes, not hours), and an extra charge on some providers for writing to the cache. Caching a prefix that’s rarely reused can cost more than it saves.

2. Response Caching

Prompt caching discounts the input. Response caching skips the call entirely.

If the exact same request comes in twice, such as “What are your business hours?”, there’s no reason to generate the answer again. Store the response keyed by a hash of the normalized prompt, model, and parameters, and return it on a match.

import hashlib, json

def cache_key(model: str, messages: list, params: dict) -> str:
    payload = json.dumps({"m": model, "msgs": messages, "p": params}, sort_keys=True)
    return "llm:" + hashlib.sha256(payload.encode()).hexdigest()

def cached_completion(model, messages, **params):
    key = cache_key(model, messages, params)
    if (hit := redis.get(key)) is not None:
        return json.loads(hit)
    result = call_llm(model, messages, **params)
    redis.setex(key, 60 * 60 * 24, json.dumps(result))  # 24h TTL
    return result

Where it pays off most: FAQs, product descriptions, deterministic transformations (translate this string, summarize this fixed document), and any endpoint with a long tail of repeated inputs.

Watch out for: stale answers. Give every entry a TTL and invalidate the cache when the source data changes. Never cache responses that contain personal or user-specific data under a shared key.

3. Semantic Caching

Exact-match caching misses the obvious: users don’t phrase things the same way. “What is OAuth?”, “Explain OAuth”, and “Tell me about OAuth” deserve the same answer, but they hash to three different keys.

Semantic caching compares meaning instead of strings. You embed each incoming query, search a vector store of previously answered queries, and return the stored response if the similarity clears a threshold.

def semantic_lookup(query: str, threshold: float = 0.93):
    vector = embed(query)  # embedding calls are far cheaper than generation
    match = vector_store.nearest(vector, k=1)
    if match and match.score >= threshold:
        return match.response
    return None

Where it pays off most: support bots, internal help desks, and “reset my password”-style traffic where a small set of intents accounts for most volume.

Watch out for: false hits. “How do I cancel my order?” and “How do I cancel my subscription?” are close in embedding space and have completely different answers. Tune the threshold on real traffic, scope the cache per tenant or product area, and log hits so you can audit them. Start strict and loosen gradually.

4. Batch Processing

Not every request needs an answer in 800 milliseconds. Evals, nightly backfills, data enrichment, content tagging, and report generation can all wait.

Batch processing comes in two forms, and you should use both.

Batch APIs. Major providers offer asynchronous batch endpoints that process large sets of requests within a window (often up to 24 hours) at roughly half the normal price. You upload a file of requests, poll for completion, and download the results. If a job doesn’t have a user waiting on it, it probably belongs here.

Packing multiple records per request. Instead of 100 calls to classify 100 support tickets, send one call with 20–50 tickets and ask for a JSON array back. You pay for the instructions once instead of 100 times.

prompt = f"""Classify each ticket as billing, bug, feature_request, or other.
Return a JSON array of {{"id": ..., "label": ...}} objects, one per ticket.

Tickets:
{json.dumps(tickets)}"""

Watch out for: quality drift as batch size grows. Models can skip items or get sloppier toward the end of a long list. Validate that every input ID comes back, retry only the missing ones, and find your batch size with an eval rather than a guess.

5. Context Trimming

Every token you send is a token you pay for, and in multi-turn apps the context grows with every exchange. A conversation that started at 500 tokens can easily reach 20,000, most of which has nothing to do with the current question.

Trimming means deliberately deciding what the model sees:

• Keep the system prompt and the last few turns.
• Drop messages that are off-topic, duplicated, or superseded.
• Strip stale tool outputs. A 5,000-token API response from ten steps ago rarely needs to be re-read in full.
• Remove boilerplate: signatures, HTML, repeated disclaimers, base64 blobs.

def build_context(history, max_recent=6):
    recent = history[-max_recent:]
    trimmed = []
    for msg in recent:
        if msg["role"] == "tool" and msg["age_in_steps"] > 3:
            msg = {**msg, "content": f"[tool result omitted: {msg['tool_name']}]"}
        trimmed.append(msg)
    return dedupe(trimmed)

Getting a 20k-token conversation down to the 2k tokens that actually matter cuts input cost on that call by about 90%. It often improves answers too, because the model has less noise to sift through.

Watch out for: trimming something the user refers back to (“like I said earlier…”). Pair trimming with summarization so the facts survive even when the raw messages don’t.

6. Conversation Summarization (Context Compaction)

Trimming throws history away. Summarization compresses it.

Once a conversation passes a threshold, replace the older turns with a short summary of what matters: key facts, user preferences, decisions made, and open questions. Keep the recent turns verbatim and the rest as compact memory. A 40-turn, 90k-token thread can often be represented in about 3k tokens without losing anything the model needs.

SUMMARY_PROMPT = """Summarize this conversation for an assistant continuing it.
Keep: user goals, stated preferences, decisions made, facts provided, open questions.
Drop: pleasantries, resolved tangents, raw tool output.
Max 300 words."""

if count_tokens(history) > 30_000:
    summary = call_llm(SMALL_MODEL, [{"role": "user", "content": SUMMARY_PROMPT + render(history[:-6])}])
    history = [{"role": "user", "content": f"Conversation so far: {summary}"}] + history[-6:]

Use a small, cheap model for the summary itself. It’s a well-defined task and doesn’t need frontier reasoning.

Watch out for: summary drift. Repeatedly summarizing summaries loses detail over time. Store important structured facts (user’s plan tier, account ID, chosen options) in explicit memory fields rather than relying on prose summaries to carry them.

---

Part 2: Right-size Every Call

Once you’ve stopped re-paying for work, the next lever is making each call as lean as it can be: the right model, the right inputs, and only the output you actually need.

7. Use the Smallest Model That Can Do the Job

This is the single biggest lever on most AI bills. The price gap between small and frontier models is often 10x or more per token, and a large share of production traffic doesn’t need frontier capability at all.

Small models are usually very good at:

• FAQ answering over provided context
• Classification and sentiment analysis
• Entity and field extraction
• Short summaries and reformatting
• Simple routing decisions

Reserve large models for tasks that need them: complex coding, multi-step planning, ambiguous reasoning, and long-running agent workflows.

The habit to break is picking a default model once and using it everywhere. Match the model and the context window to the task. Don’t use a frontier model to fetch milk.

Watch out for: assuming. Run your real tasks through a cheaper model on an eval set (see technique 21) before switching. Sometimes the small model is worse, and the retries erase the savings. Often it’s just as good.

8. Retrieve, Don’t Stuff (RAG)

The lazy way to give a model knowledge is to paste the whole manual into the prompt. It works, and it’s expensive: you pay for 100 pages of input to answer a question that lives in one paragraph.

Retrieval-Augmented Generation sends only what’s relevant:

1. Split your knowledge base into chunks.
2. Embed and index them in a vector store (often combined with keyword search).
3. At query time, retrieve the top matching chunks.
4. Send only those chunks to the model with the question.

chunks = retriever.search(user_question, k=20)
top_chunks = reranker.rerank(user_question, chunks)[:5]  # top-5, not the whole wiki
context = "\n\n".join(c.text for c in top_chunks)

A good reranker is worth its cost here. Retrieving 20 candidates and keeping the best 5 is usually more accurate than sending 15 mediocre matches, and it’s much cheaper.

Watch out for: chunks that are too small to make sense on their own, or so large they defeat the purpose. Measure retrieval quality separately from answer quality so you know which part to fix.

9. Cap the Output

Output tokens are typically several times more expensive than input tokens, so verbose responses are one of the most expensive habits a system can have.

Three tools for controlling output length:

• max_tokens: a hard ceiling on response length. Set it per endpoint based on what the response actually needs, not a global default of 4,096.
• Stop sequences: end generation as soon as the useful part is done, for example stop: ["###"].
• Concise instructions: “Answer in 3 bullet points.” “Reply in under 50 words.” “Return only the SQL, no explanation.”

Watch out for: silent truncation. When a response hits max_tokens, it’s cut off mid-sentence or mid-JSON. Always check the stop reason, and handle truncation explicitly instead of passing broken output downstream.

10. Use Structured Outputs

When the response feeds code rather than a human, ask for structure instead of prose.

{"sentiment": "negative", "category": "billing", "urgency": 3}

versus

“After carefully reviewing this message, it appears the customer is expressing frustration regarding a billing issue, which I would classify as moderately urgent…”

Structured outputs save money three ways:

1. Fewer tokens. No preamble, no explanation, no filler.
2. Easier parsing. No regex over free text.
3. Fewer retries. Using your provider’s JSON schema or structured output mode means you aren’t paying again for malformed responses.

Keep schemas tight: use enums instead of free-text fields where you can, and don’t ask for fields you won’t use.

11. Dial the Reasoning Effort

Reasoning models and “extended thinking” modes can produce a lot of hidden thinking tokens before the visible answer, and those tokens are generally billed as output. On a hard problem that’s money well spent. On “reformat this date,” it’s waste.

Most providers now expose controls for this, such as reasoning effort levels or thinking token budgets. Treat it as a per-task setting:

• Low or no reasoning: extraction, classification, formatting, simple Q&A.
• Medium: multi-step questions, moderate analysis.
• High: complex coding, math, planning, and anything where a wrong answer is expensive.

Watch out for: setting reasoning to maximum globally “just to be safe.” Check your token logs. If reasoning tokens dwarf answer tokens on routine traffic, you’re overpaying.

12. Tool-First Architecture

LLMs are a costly and nondeterministic way to do things that ordinary code does perfectly. Before invoking a model, ask whether a deterministic tool can answer directly:

Request	Use this, not the LLM
“What time is it in Tokyo?”	Clock / timezone library
“What’s the weather in Paris?”	Weather API
“Convert 200 USD to EUR”	Currency API + arithmetic
“How many orders did we ship yesterday?”	SQL query
“Is this a valid email address?”	Regex / validation library

The model’s job is reasoning: understanding intent, choosing which tool to call, and composing the final answer. The tools do the lookups and arithmetic. In many systems, a simple rules layer in front of the model can answer a meaningful share of requests without any LLM call at all.

Bonus: tools are also more accurate. Models are bad at arithmetic and know nothing about today’s exchange rate.

---

Part 3: Route to the Cheapest Capable Model

Technique 7 says to use the smallest model that works. This section is about making that decision automatically, per request, at runtime.

13. Query Classification and Multi-Model Routing

Put a lightweight classifier in front of everything. Its job is to decide what kind of request this is and send it down the right path:

• FAQ → semantic cache or a canned answer
• Knowledge question → RAG pipeline with a small or mid-sized model
• Data question → SQL generation with a mid-sized model
• Coding or complex task → large model
• Multi-step task → agent

ROUTES = {
    "faq": handle_faq,             # cache / small model
    "rag": handle_rag,             # small or medium model
    "sql": handle_sql,             # medium model
    "coding": handle_coding,       # frontier model
    "agent": handle_agent,         # frontier model + guardrails
}

intent = classify(user_query)      # rules, embeddings, or a tiny model
return ROUTES.get(intent, handle_rag)(user_query)

The classifier itself must be cheap. Keyword rules, an embedding nearest-neighbor lookup, or a small fine-tuned model all work. Using a frontier model to decide whether to use a frontier model defeats the point.

This is also where open-weight models fit. Your routing pool doesn’t have to come from a single provider. An open-weight model that handles your classification traffic for a fraction of the cost belongs in it.

The biggest win from routing is often what it prevents: an agent spinning up for a question a FAQ could have answered.

Watch out for: misrouting. Log every routing decision with its outcome, and review cases where the cheap path failed so you can tune the classifier.

14. Cascade on Confidence

Routing decides up front. Cascading decides after the fact: try the small model first and escalate to a stronger one only when the answer isn’t good enough.

answer = call_llm(SMALL_MODEL, prompt)
if not passes_checks(answer):        # schema valid? tests pass? confidence high enough?
    answer = call_llm(FRONTIER_MODEL, prompt)
return answer

The hard part is passes_checks. Good signals are verifiable ones:

• The output validates against a JSON schema.
• Generated code compiles and passes tests.
• Generated SQL runs and returns a sensible shape.
• Extracted values appear in the source document.
• Token log-probabilities, where your provider exposes them.

Be careful with asking a model to rate its own confidence. Self-reported confidence is often poorly calibrated. Use it as one signal among several, not as the gate.

If the small model handles 80% of traffic on its own, you pay frontier prices on only the remaining 20%, plus the cost of the failed small-model attempts. That last part matters: if the small model fails most of the time, cascading costs more than going straight to the large model. Measure the escalation rate.

15. Self-Host for Volume

At high volume, per-token pricing can lose to running your own hardware. If you have a well-defined task (classification, extraction, embeddings, moderation) running at millions of calls per day, an open-weight model on your own GPUs turns a variable cost into a mostly flat one.

Do the break-even math honestly before committing:

• GPU cost, whether reserved, on-demand, or owned
• Utilization: idle GPUs are expensive GPUs, and spiky traffic hurts
• Engineering time for serving, scaling, monitoring, and upgrades
• Quality gap between the open-weight model and the API model on your task
• Inference stack (vLLM, TGI, and similar) and its throughput on your workload

Self-hosting suits steady, high-volume, narrow workloads. It rarely makes sense for low-volume, spiky, or open-ended tasks where frontier quality matters. Many teams land on a hybrid: self-host the high-volume workhorses and call APIs for everything else.

16. Distill the Winners

If a frontier model is doing a repetitive task well, you can teach a smaller model to do the same thing.

Distillation works like this:

1. Log frontier model inputs and outputs on a specific task.
2. Filter for the high-quality outputs (human-reviewed, validated, or from successful outcomes).
3. Fine-tune a small model on that dataset.
4. Evaluate it against the frontier model on a held-out set.
5. Route that task to the small model if it holds up.

The result is a model that’s fast, cheap, and specialized for your exact use case. It often matches the frontier model on that narrow task while costing a fraction per call.

Watch out for: drift. When your inputs change (new product lines, new ticket types), the distilled model can quietly degrade. Keep evaluating it, and keep a path to retrain. Also check your provider’s terms on using model outputs for training.

---

Part 4: Make Cost Visible and Controlled

Every technique above is easier with good instrumentation, and some of them are impossible without it. This section is about seeing where money goes and stopping it from going where it shouldn’t.

17. Agent Guardrails

Agents are often the biggest single source of runaway AI spend. One user request can trigger dozens of model calls, each carrying a growing context full of tool results. Without limits, a confused agent can loop for hundreds of steps before anyone notices.

Every agent needs hard limits:

@dataclass
class AgentLimits:
    max_iterations: int = 15
    max_tool_calls: int = 25
    max_total_tokens: int = 200_000
    max_cost_usd: float = 0.50
    timeout_seconds: int = 120

def run_agent(task, limits: AgentLimits):
    usage = Usage()
    for step in range(limits.max_iterations):
        if usage.tokens > limits.max_total_tokens or usage.cost > limits.max_cost_usd:
            return graceful_stop(task, reason="budget_exceeded")
        if usage.elapsed > limits.timeout_seconds:
            return graceful_stop(task, reason="timeout")
        action = plan_next_step(task, usage)
        ...

Beyond hard caps:

• Detect loops. If the agent calls the same tool with the same arguments twice, intervene.
• Fail gracefully. When a limit trips, return partial results and a clear explanation, not a stack trace.
• Combine with earlier techniques. Agents benefit enormously from prompt caching (tool definitions repeat every step) and context trimming (old tool results pile up fast).

18. Real-Time Cost Visibility

You can’t optimize what you can’t see. The first step for any team serious about AI costs is tagging every LLM call with metadata and logging it:

log_llm_call(
    feature="support_bot",
    user_id=user.id,
    team="customer_success",
    agent="refund_agent",
    model=model,
    input_tokens=usage.input_tokens,
    cached_tokens=usage.cache_read_input_tokens,
    output_tokens=usage.output_tokens,
    cost_usd=compute_cost(model, usage),
    latency_ms=latency,
)

From that, build a dashboard that tracks:

• Cost per user, per request, per feature, and per agent
• Tokens per request (input, cached, output, and reasoning)
• Cache hit rate, for both prompt and response caches
• Model usage mix: what share of traffic goes to each model
• Daily spend and trend lines
• Top offenders: the most expensive prompts, users, and agent runs

Make it visible to every engineer, not just finance. When developers can see that their new feature costs $0.12 per request, they start optimizing without being asked.

19. Budgets and Alerts

Dashboards tell you what happened. Budgets and alerts tell you before it becomes a problem.

Set token or dollar budgets at several levels:

• Per team to keep shared costs accountable
• Per feature to catch a single endpoint gone wrong
• Per agent to catch runaway automation
• Per user (with rate limits) to protect against abuse or a single heavy account

Alert at meaningful thresholds, such as 50%, 80%, and 100% of budget, and alert on anomalies: a sudden spike in tokens per request often means someone changed a prompt or broke a cache. For non-critical workloads, consider hard caps that degrade gracefully (switch to a cheaper model, queue for batch) instead of failing.

The goal is to hear about a cost problem from your alerting system in an hour, not from your finance team at the end of the month.

20. Measure Cost per Outcome, Not Cost per Token

Cost per token is the wrong optimization target on its own. A cheap model that resolves 40% of support tickets can be more expensive per resolved ticket than a pricier model that resolves 85%, once you count retries, escalations, and the human time spent cleaning up.

Track the metric your business actually cares about:

• Cost per resolved support ticket
• Cost per successfully merged pull request
• Cost per document correctly processed
• Cost per qualified lead

This reframes every decision in this guide. A technique that cuts tokens but lowers success rate may raise your cost per outcome. A more expensive model that solves the task in one shot instead of three may lower it.

21. Evals Before Upgrades

A new, bigger model comes out. It tops the leaderboards. Should you switch?

Not before testing it on your own tasks. Public benchmarks measure general capability. Your eval set measures whether the model is better at your job.

Build a representative eval set from real traffic (anonymized as needed), with known-good answers or clear scoring criteria. Then for every proposed model change:

1. Run both models on the eval set.
2. Compare quality, cost, and latency together.
3. Upgrade only if the quality gain justifies the cost increase.

Sometimes the frontier model wins by a wide margin and the upgrade is obvious. Often it wins by 1% at three times the price, and the right call is to keep the smaller model. The same eval set lets you safely test downgrades, prompt changes, and every other technique in this article.

---

Where to Start

Twenty-one techniques is a lot to take on at once. Here’s a practical order:

Week 1: See the problem.
Add per-call logging with feature, user, and model tags (18). Find out which features, agents, and prompts account for most of your spend. In most systems a small number of them dominate.

Weeks 2–3: Quick wins.
These need little architecture change and pay back immediately:

• Restructure prompts for prompt caching (1)
• Set sensible max_tokens and concise instructions per endpoint (9)
• Move non-urgent jobs to batch APIs (4)
• Add agent guardrails (17)
• Set budgets and alerts (19)

Month 2: Structural changes.

• Build an eval set (21), then test smaller models on your top workloads (7)
• Add context trimming and summarization (5, 6)
• Add response and semantic caching where traffic is repetitive (2, 3)
• Put a classifier and router in front of your models (13)

Later: Advanced optimization.
Once you have evals, routing, and good data:

• Cascading (14)
• Distillation (16)
• Self-hosting high-volume workloads (15)

---

Cheat Sheet

#	Technique	What it does	Best for
1	Prompt caching	Discounts repeated prompt prefixes	Agents, RAG, long system prompts
2	Response caching	Skips calls for exact repeats	FAQs, deterministic tasks
3	Semantic caching	Skips calls for similar questions	Support bots, help desks
4	Batch processing	~50% off async work; amortizes instructions	Evals, backfills, enrichment
5	Context trimming	Drops irrelevant history and stale tool output	Multi-turn chat, agents
6	Summarization	Compresses old history into compact memory	Long conversations
7	Smaller models	Cheapest model that does the job	Classification, extraction, FAQ
8	RAG	Sends top chunks, not whole documents	Knowledge assistants
9	Output caps	Limits expensive output tokens	Every endpoint
10	Structured outputs	Fewer tokens, fewer retries	Machine-consumed responses
11	Reasoning effort	Reserves deep thinking for hard tasks	Reasoning models
12	Tool-first	Deterministic code before LLM calls	Lookups, math, data queries
13	Classification & routing	Picks the right path and model per request	Mixed-traffic products
14	Cascading	Small model first, escalate on failure	Verifiable tasks
15	Self-hosting	Flat cost at high volume	Narrow, steady, high-volume tasks
16	Distillation	Small model trained on frontier outputs	Repetitive specialized tasks
17	Agent guardrails	Caps iterations, tools, tokens, time	Any agent
18	Cost visibility	Per-call cost logging and dashboards	Everyone, first
19	Budgets & alerts	Catches overruns early	Teams, features, agents
20	Cost per outcome	Measures dollars per result	Business decisions
21	Evals before upgrades	Proves a model change is worth it	Model changes

---

Final Thoughts

Cutting tokens is not the goal by itself. A system that’s cheap and wrong is worse than one that’s expensive and right. The goal is reliable AI outcomes without paying for waste: tokens you’ve already bought, context nobody needed, reasoning nobody asked for, and frontier models doing a small model’s job.

Most teams find that the first round of optimization (visibility, caching, output limits, and right-sized models) cuts a large share of their bill with no loss in quality. The rest comes from treating model choice, context, and cost as engineering decisions you measure and tune, like latency or uptime, rather than settings you configure once and forget.

The four-part grouping in this article (reuse, right-size, route, make cost visible) draws on the “Tokenmaxxing” framework shared by Rathnakumar Udayakumar.

Which of these techniques has made the biggest difference to your AI infrastructure costs? Share your experience in the comments.