Every AI product eventually meets the same infrastructure choice: reserve capacity ahead of time or request it when traffic arrives. Reserved capacity can protect an important workload from queues. On-demand capacity can protect a young product from paying for servers it does not yet need.

The right answer is rarely “always reserve” or “always use on-demand.” It depends on the shape of traffic, the cost of downtime, and how quickly the product is changing.

The short answer

Use on-demand capacity while traffic is uncertain and workloads are easy to pause. Reserve capacity when the workload is predictable, latency matters, and the discount or reliability benefit outweighs the commitment. Keep a small fallback path so one provider cannot stop the product completely.

What you are really buying

Reservation is not just a lower price. It may buy a capacity guarantee, a region, a hardware type, a service level, or a delivery window. Read the contract carefully. Some commitments are take-or-pay; others are flexible reservations that can be released.

On-demand capacity is not free flexibility. It can have higher unit pricing, queueing during demand spikes, and less predictable availability. If a coding assistant suddenly becomes popular, a request that usually completes in seconds may wait much longer.

I separate the decision into three budgets: a reliability budget, a cash budget, and an experimentation budget. A mature product may reserve the reliability-critical path while leaving experiments on on-demand compute.

A simple break-even test

Record monthly on-demand cost, average utilization, peak utilization, queue time, and the cost of a failed request. Compare that with the reservation’s fixed fee, expected utilization, and cancellation risk. Do not use a best-case utilization number; use the last 8 weeks of real traffic where possible.

For example, a team might discover that the workload is busy for 6 hours each weekday but nearly idle overnight. A full-day reservation could waste money unless batch jobs can fill the quiet period. A mixed schedule may be better: reserve the predictable baseline and burst on demand.

How to design the application

Put the provider behind a small interface with timeout, retry, cancellation, and usage accounting. Mark requests as interactive, batch, or background. Interactive requests should have a maximum wait time and a user-visible fallback. Background requests can enter a queue and run when capacity is cheaper.

Measure p95 latency, error rate, queue time, and cost per useful result. A 2-hour load test is long enough to expose some queueing patterns, but production telemetry should make the final decision.

What infrastructure headlines leave out

Large data-center announcements often describe planned capacity rather than capacity available to your application. OpenAI’s compute infrastructure overview explains why power, permits, and construction affect delivery. Its multi-cloud update shows why providers combine several clouds and silicon paths. The NVIDIA filing on cloud commitments is a useful example of why a commitment should not be confused with immediate productive usage.

A practical default

Start on demand, instrument everything, and reserve only the baseline you can defend with measured traffic. Add a fallback before the first large launch. That approach keeps a product adaptable without pretending that future capacity announcements are available today.

When to change the decision

Revisit the choice when the workload becomes predictable, when queue time starts affecting retention, or when a provider offers a commitment that can be cancelled without damaging the cash budget. Capacity planning is a product decision as much as an infrastructure decision. The best reservation is the one the team can explain with observed traffic and a clear fallback.

Also account for migration effort. Moving between accelerator types can require new kernels, test fixtures, and operational runbooks. A small price difference may disappear if the team must maintain two separate serving paths without enough traffic to keep either one healthy.

The same principle applies to storage and networking: reserve the bottleneck you can measure, not every component you might need someday.