Most model bills are not high because the model is expensive. They are high because the same prompt prefix is paid for on every call, because output tokens cost several times input tokens, and because nobody measured whether a cheaper configuration would have passed. LLM cost optimization turns that from a guess into an ordered set of measured changes. Sigi Technologies works on this as LLM optimization.
What is LLM cost optimization?
LLM cost optimization is the practice of reducing the money and latency cost of serving a model at a fixed quality bar. The bar is the part teams skip. Without an evaluation set, a cheaper configuration is not a saving, it is an untested change. The unit worth optimizing is cost per completed task, because a cheap call retried twice is not cheap.
- Claude API cache reads are billed at 0.1 times base input and a five-minute cache write at 1.25 times, so Anthropic’s pricing documentation states that caching pays off after a single cache read.
- Asynchronous batch endpoints carry a 50% discount on input and output tokens at Anthropic, OpenAI and Google, in exchange for delivery within 24 hours.
- FrugalGPT reported matching GPT-4, the best individual model of its day, with up to 98% cost reduction from a cascade of cheaper models, on its own 2023 benchmark set.
- Quantization is close to free at 8 bits and needs testing at 4: over 500,000 evaluations on Llama-3.1 found FP8 effectively lossless and well-tuned INT8 within 1 to 3% degradation.
- Self-hosting is a utilization question: on identical H100 hardware, a 2026 study measured $0.21 to $15.25 per million output tokens.
- Token prices fell roughly 600-fold between 2020 and 2026, but flagship models barely moved, carrying a reasoning premium averaging 31.5 times non-reasoning prices.
What drives LLM inference cost?
LLM inference cost is the amount billed each time a model reads a prompt and produces a response. Five quantities show up on the invoice, and only the first is obvious.
- Input tokens: everything sent, including the system prompt, the tool definitions and the whole conversation so far. Anthropic publishes the tool use overhead per model, 286 tokens on Claude Opus 5, before any of your own text.
- Output tokens: roughly five times the input price on current frontier models, so answer length moves the bill more than question length.
- Cached input tokens: a separate, much cheaper line, covered next.
- Reasoning tokens: models that think before answering bill that thinking as output. A 2026 study named this the reasoning tax and found inference spend is often dominated by internal thinking rather than the visible answer.
- Server-side tool calls: web search on the Claude API costs $10 per 1,000 searches on top of tokens, and fetched pages become input tokens on every later turn.
Prices as of 11 September 2026, from vendor documentation and subject to change: Anthropic lists Claude Opus 5 at $5 per million input tokens and $25 per million output, and Claude Haiku 4.5 at $1 and $5. OpenAI lists GPT-5.6-Terra at $2 and $12. Google lists Gemini 3.8 Flash at $0.75 and $3.75. Read the current page before building a cost model on these.
How much does prompt caching actually save?
Prompt caching stores the processed form of a stable prompt prefix so later requests reuse it instead of paying to process it again. Anthropic’s pricing page gives the multipliers exactly: a five-minute cache write costs 1.25 times base input, a one-hour write 2 times, and a cache read 0.1 times. Its own summary is that caching pays off after one read on the five-minute duration and two reads on the one-hour duration.
The discount is not the same everywhere, which is where the widely repeated 90% figure breaks. OpenAI’s prompt caching guide says cached input is discounted up to 90% and that reads cost 0.1 times the uncached rate on GPT-5.6 and later, but its own price list still shows GPT-4o cached input at $1.25 against $2.50 uncached, which is 50%.
Google bills Gemini context caching twice, per cached token and per hour of storage: the pricing page lists Gemini 3.8 Flash cached input at $0.075 per million against $0.75 uncached, plus $0.50 per million tokens per hour to hold it. A cache you write and never read is a bill, not a saving.
- Minimum prefix length. Anthropic will not cache below 512 to 4,096 tokens depending on the model, and returns no error when it skips caching. OpenAI sets 1,024 tokens on GPT-5.6 and later; Google sets 4,096 on Gemini 3.5 Flash and newer.
- Prefix instability. Caching is a prefix match, so a timestamp, a request identifier or a reordered JSON key near the top invalidates everything after it.
- Short lifetimes. Anthropic’s default cache lives five minutes and is refreshed free on each use; OpenAI keeps a prefix eligible for 30 minutes after its most recent use on GPT-5.6 and later.
- No measurement. Anthropic and Google both return a cached token count on every response. If it stays zero across repeated calls with the same prefix, the problem is in the prompt, not the price list.
Does model routing or an LLM cascade cut costs?
Model routing sends each request to the cheapest model likely to answer it correctly. A cascade goes further: it calls a cheap model first and escalates only when a scoring step judges the answer inadequate. FrugalGPT, by Chen, Zaharia and Zou in 2023, is the reference result, reporting that it can match the best individual LLM with up to 98% cost reduction and improve accuracy over GPT-4 by 4% at the same cost. RouteLLM, by Ong and colleagues in 2024, trains a router on preference data and reports cutting costs by over two times in certain cases without compromising response quality.
Two conditions decide whether that transfers to your product. The first is query mix: the saving is capped by the share of traffic the cheap model can serve. The second is the router, a classifier that costs tokens to run and errs in both directions. A false negative sends easy work to the expensive model and wastes the saving. A false positive ships a wrong answer. Measure the router as its own component, on the evaluation set that defines the quality bar.
When does batching reduce LLM costs?
Batching means two things and both reduce cost. On the vendor side, an asynchronous batch endpoint trades latency for price: Anthropic, OpenAI and Google all publish a 50% discount on input and output tokens, and OpenAI states that each batch completes within 24 hours, with up to 50,000 requests per batch. Anything that does not face a user belongs here: back-filling classifications, overnight summaries, an evaluation suite.
On the serving side, batching is how a graphics processing unit (GPU) is kept busy. The vLLM paper by Kwon and colleagues in 2023 introduced PagedAttention, which manages the key-value cache the way an operating system manages virtual memory, and reports 2 to 4 times higher throughput than FasterTransformer and Orca at the same latency. DistServe, by Zhong and colleagues in 2024, separates prefill from decoding onto different GPUs and reports serving 7.4 times more requests, or meeting a 12.6 times tighter service level objective, within latency constraints for over 90% of requests. Neither figure reaches you unless you run the serving stack yourself.
Do quantization and distillation reduce cost without breaking the model?
Quantization
Quantization stores model weights, and sometimes activations, at lower numeric precision so the model needs less memory and less memory bandwidth. GPTQ, by Frantar and colleagues, quantizes 175-billion-parameter models to 3 or 4 bits per weight in about four GPU hours with what the paper calls negligible accuracy degradation, and reports end-to-end speedups over FP16 of around 3.25 times on an NVIDIA A100 and 4.5 times on an A6000. AWQ, by Lin and colleagues, protects the roughly 1% of weight channels activations mark as salient and reports more than 3 times speedup over the Hugging Face FP16 implementation.
Those are speed numbers on named hardware, not a promise about your model. Accuracy was measured separately by Kurtic and colleagues, who ran over 500,000 evaluations across the Llama-3.1 family and found FP8 effectively lossless at all scales, well-tuned INT8 at a surprisingly low 1 to 3% degradation, and 4-bit weight-only quantization stronger than expected. The honest summary: 8-bit is close to a free win, 4-bit needs your own evaluation set before it ships, and none of it applies to a hosted API where you never touch the weights.
Distillation
Distillation trains a small model to imitate a large one. The idea comes from Hinton, Vinyals and Dean in 2015, who compressed the knowledge in an ensemble into a single model much easier to deploy. The version that matters for language tasks is Distilling Step-by-Step, by Hsieh and colleagues in 2023, which trains the small model on the large model’s rationales as well as its labels and reports a fine-tuned 770-million-parameter T5 beating a few-shot prompted 540-billion-parameter PaLM on a benchmark using only 80% of the available data. Distillation earns its effort when one narrow task carries high volume and rarely changes.
Speculative decoding
Speculative decoding runs a small draft model to guess several tokens ahead and uses the large model to verify them in one pass. Leviathan, Kalman and Matias reported a 2 to 3 times acceleration on T5-XXL with identical outputs, no retraining and no architecture change. It cuts latency, and therefore hardware idling between tokens, so it helps a team running its own serving stack and does nothing for a per-token API bill.
Does a longer context window cost more?
Per token, often not. Anthropic states that Claude 4.6 and later models include the full one-million-token context window at standard pricing, so a 900,000-token request is billed at the same per-token rate as a 9,000-token one. In total, always. Context is billed on every call it appears in, which is why agent loops get expensive: each tool result is appended and re-sent on the next turn, so the loop pays for its history repeatedly.
Prompt compression is one answer, and LLMLingua, by Jiang and colleagues in 2023, reports up to 20 times compression with little performance loss across GSM8K, BBH, ShareGPT and Arxiv-March23. Deciding what not to carry forward is cheaper than compressing what you carry, and easier to debug.
Is self-hosting an LLM cheaper than an API?
It depends on utilization, not on the price list. A 2026 study by Patil, Beyond Per-Token Pricing, argues that per-token calculators treat GPU utilization as fixed when it is load-dependent, so a utilization-naive estimate understates true cost by exactly one over utilization. On identical H100 hardware the paper measures cost spanning $0.21 to $15.25 per million output tokens, an underutilization penalty of 2.5 to 24 times at one to ten requests per second and up to 36.3 times near idle.
Rented hardware bills by the hour whether or not anyone uses it, so self-hosting wins on steady, high load and loses on spiky traffic. The same arithmetic governs infrastructure generally, covered in Sigi’s guide to cloud cost optimization.
Why is evaluation a cost lever rather than a cost?
Every lever above is a quality risk. A cheaper model, a shorter prompt, a 4-bit weight, a cascade that escalates too late: each can degrade output quietly, and a bad output costs more than it saved once a person redoes the work. An evaluation set converts a guess into a measurement, and it is the only thing that lets a team downgrade a model on purpose.
Judge models make that affordable. Zheng and colleagues, in the paper that introduced MT-Bench and Chatbot Arena, found strong LLM judges such as GPT-4 reach over 80% agreement with human preferences, the same level humans reach with each other. They also named the failure modes: position bias, verbosity bias, self-enhancement bias and limited reasoning. Validate the judge against human labels once, then re-run it on every configuration change. Reliability belongs in the same harness: track refusal rate, malformed-output rate and retries, because a retry bills the whole prompt again at full price.
Are LLM prices falling fast enough that you can just wait?
For cheap models, largely yes. Epoch AI’s analysis, published in March 2025, tracked state-of-the-art models on six benchmarks over three years and found the price to reach GPT-4’s performance on PhD-level science questions fell by 40 times per year, with declines across benchmarks ranging from 9 to 900 times per year. It cautions that the fastest drops happened most recently, so they may not persist.
A 2026 economic analysis by Du documents an approximately 600-fold decline in token prices from 2020 to 2026, then splits the tiers. Economy models show a price half-life of 1.10 years and mid-tier 1.55 years. Flagship models show almost no exponential fit, at an R-squared of 0.031, because of a reasoning premium averaging 31.5 times non-reasoning prices. If your workload sits on the frontier, waiting is not a strategy.
In what order should you reduce LLM costs?
- Instrument first. Log input, output, cached and reasoning tokens, retries and cost per completed task, tagged by feature and by customer. You cannot route traffic you cannot attribute.
- Build the evaluation set before changing anything, with enough real failures in it to detect a regression.
- Take the free wins: a stable cached prefix, shorter system prompts and tool definitions, output length limits, and no context the next turn does not need.
- Move everything that does not face a user onto a batch endpoint at half price.
- Then trade quality for cost deliberately: lower reasoning effort, a smaller model on routes where the evaluation set holds, and a cascade only where the query mix is genuinely split between easy and hard.
- Leave self-hosting, quantization and distillation last, for load that is high, steady and predictable enough to keep the hardware busy.
Related reading
Retrieval adds input tokens to every call, so read what RAG is and when not to use it for the quality side, and how to build an AI chatbot for your business for the product side. For the infrastructure underneath, see cloud cost optimization. Sigi Technologies describes this work on its LLM optimization page, task-completing systems on AI agent development, and the wider practice on AI development services. To review a token bill and an evaluation set together, contact Sigi.