AI providers usually advertise inference with a simple number: price per million input or output tokens. That number is useful, but it is not the cost of a working product. A real request can include long context, several tool calls, retries, retrieval, storage, network transfer, and a user waiting for a response.

The short answer

Calculate cost per successful task, not only cost per token. Include input and output tokens, cache behavior, retries, tool calls, retrieval, idle capacity, and human correction. A cheaper model can become more expensive if it needs more attempts to produce a usable result.

The hidden multipliers

Long conversations increase input tokens on every turn. Agent loops add calls. A failed tool can cause a retry, and a timeout may produce a duplicate request. Retrieval can add documents that the user never sees but still pays for.

Start with a request ledger containing run ID, model, input tokens, output tokens, tool calls, retry count, duration, and final status. The ledger should let you compare a successful response with an abandoned one.

Make the unit useful

“Cost per request” is not always a useful unit. For a support bot, use cost per resolved conversation. For a coding agent, use cost per accepted change. For a batch classifier, use cost per correctly classified record. The unit should include quality, otherwise the cheapest failure looks like the best result.

I would run a representative 2-hour test with normal prompts, long prompts, failures, and cancellations. Record p50 and p95 cost as well as the average. A few runaway requests can matter more than the median.

Practical controls

Limit context length, maximum turns, tool calls, and output tokens. Summarize old history instead of sending it forever. Cache stable retrieval results. Use a smaller model for routing or classification when an evaluation shows that quality remains acceptable.

Keep an explicit budget per user, team, and workflow. Return a useful partial result when a run reaches its limit. Silent cutoffs create support work and make the product look unreliable.

Sources and limits

OpenAI’s Cerebras partnership shows why low-latency inference can matter for interactive products. The Amazon partnership shows how custom silicon can be part of a serving strategy. The OpenAI scaling update gives broader context on dedicated inference capacity. Provider announcements do not predict your application’s bill; your request traces do.

The durable rule is simple: optimize the completed task, then optimize the token price inside it.

A pricing worksheet

For each workflow, multiply average input tokens by input price and output tokens by output price. Add expected retries, retrieval, tool execution, storage, and support overhead. Then divide by completed tasks, not requests. Keep a separate worst-case budget for long contexts and failed actions.

Run the worksheet against real traces for 2 hours and review the largest ten runs. Those outliers often reveal a prompt that includes unnecessary history or a tool that returns far more data than the model needs. Fixing that behavior can save more than switching providers for every request.

Keep the budget visible

Show the user or team which workflow consumed the budget. A daily total is too vague to debug. Break it down by feature, model, tenant, and failure type. When a product team can see that retrieval or retries created the increase, it can fix the workflow instead of blindly reducing quality everywhere.

Prices also change. Store the effective price used for each run so historical reports remain meaningful after a provider updates its rates. This turns cost control into an engineering metric rather than a monthly surprise.

That record also makes vendor comparisons fair.

It also gives finance and engineering one shared number to improve.

That shared metric should be reviewed after every major prompt or model change.

Review it before rollout.

Then compare it after launch.

Keep both numbers.

They explain the change.

That is the point.

Always.