AI Inference Cost Is Architecture

An AI system’s architecture determines its inference bill. When a model decides every next step, rereads the job after every tool call, and reasons through every recovery, the invoice is describing the control flow. A system that spends tokens to decide whether it should keep spending tokens has placed its most expensive component in charge of the loop.

A free-ranging loop is a short path to a useful demo. The model picks tools, reads the results, and decides when the job is done. Then volume arrives, context grows, retries compound, and the shape of the loop becomes a line item.

The Bill Is a Trace

The vendors’ rate cards divide usage into input, cached input, and output. Generated output usually carries the highest rate. Cached input is cheaper because the provider can reuse work it has already done.¹

That price structure tells you where an agentic loop accumulates cost. Each turn sends context back in. Each decision generates more output. A failed tool call adds the call, the error, the diagnosis, and the revised attempt to the next turn. Five useful actions can require ten model decisions once planning and recovery are counted.

Prompt caching helps when the same prefix appears again. It can lower the price and latency of repeated input.² It does not remove a call, shorten generated output, or keep the model from reconsidering a branch that code already knows how to take. Caching a bloated loop makes its input cheaper. The loop remains bloated.

The total token count is less useful than the trace. Which step needed judgment? Which step merely chose the next item in a known sequence? Which retry recovered from a real surprise, and which one repaired a format the application could have enforced from the start?

Code Owns the Workflow

Conventional code should own the states, branches, retry budget, deadlines, and persistence. The model belongs at the points where the answer cannot be specified in advance.

Consider invoice reconciliation. Code can fetch the records, check that required fields exist, calculate totals, join purchase orders, and route exact matches. A model can inspect the residue: vague descriptions, inconsistent vendor names, and exceptions that need a judgment call. It returns a category, a reason, and a confidence value in a fixed schema. Code takes over again.

The free-ranging version hands an agent the invoices and says, “reconcile these.” The agent discovers the process by spending inference. It chooses queries, remembers which records it has checked, decides whether a mismatch matters, and judges when the work is complete. Every deterministic fact becomes context the model must carry.

The bounded version spends model tokens on ambiguity. The rest is ordinary software. It can resume after a failure without asking a model to reconstruct state from a transcript. It can retry one operation without replaying the surrounding thought process. It can be tested without grading a conversation.

This is the same discipline as treating prompts as specifications, applied one level higher. The prompt defines the uncertain step. The program defines the workflow.

Contracts Remove Repair Loops

Model prose is a poor interface between two pieces of code. If the next step needs three fields, the model should return those three fields through a tool call or a constrained schema. OpenAI’s Structured Outputs, for example, constrains a response to a supplied JSON Schema.³

A schema does not make the values true. It removes one class of failure: the application no longer has to parse a plausible paragraph, discover that a field is missing, explain the format again, and pay for another attempt. Semantic checks still belong downstream. The amount must balance. The identifier must exist. The proposed action must be allowed.

The contract also makes model choice easier. A narrow classification with a measurable answer can be evaluated across several models. An open-ended loop that changes its own plan is much harder to compare because each run performs a different job.

Boundaries Belong in Code

Production model steps need ceilings: maximum calls, maximum generated tokens, maximum tool retries, and a deadline. Those numbers are part of the architecture. They say how much uncertainty the system is willing to buy before it returns an exception to code or a person.

A prompt that says “be concise” is a useful prior. A token limit is a boundary. “Do not retry forever” is sensible guidance. A counter enforced by the orchestrator is a control. Rules are not controls, especially when the rule governs the meter that pays for interpreting it.

An undersized model or token allowance can create more retries, more review, and worse work. One strong call with the right context may cost less than a chain of weak calls trying to repair one another. The unit that matters is cost per accepted result.

That gives each model call a budget and a success condition. Extract these fields within this schema. Classify this exception against these policies. Draft one answer from these sources. When the result fails validation, the system knows which step failed and what one retry is allowed to change.

Inference Belongs at Uncertain Steps

Some jobs remain agentic by nature. Research, debugging, and unfamiliar code changes require plans to change as evidence arrives. A loop is appropriate there. It still benefits from a wall-clock deadline, a tool-call budget, explicit completion criteria, and checkpoints stored outside the transcript.

The useful telemetry follows the same boundary. Cost by workflow step. Cache hits. Retry counts. Accepted results. Human corrections. The final invoice says how much was spent; the trace says why. Without that split, a high-value reasoning step and a wasteful recovery loop look identical.

Footnotes

1. Current OpenAI API pricing and Gemini API pricing separate input, cached input, and output rates. ↩

2. OpenAI’s prompt caching documentation describes reuse of repeated prompt prefixes and the resulting input discount. ↩

3. OpenAI, “Introducing Structured Outputs in the API”, August 6, 2024. ↩