OpenRouter’s Batch API makes waiting an engineering decision

OpenRouter announced its Batch API on September 22, allowing requests to be completed asynchronously within a 24-hour window. The launch covered more than 70 models and described prices typically around half the normal per-token rate, sometimes lower. OpenRouter said its two-week beta processed over 230,000 batches, with a seven-minute median completion time and 90% finishing within an hour.

The beta figures describe observed traffic, not a deadline that applications should promise to their users. A batch service trades immediacy for scheduling flexibility. That exchange suits work with a generous deadline, while a customer waiting for a live answer usually needs a different execution path.

Give each item its own identity

Consider an organization classifying a backlog of support tickets. Each request should carry a stable identifier that maps the result back to one ticket and one version of the classification instructions. The application should not depend on results arriving in the original order.

A completion handler should distinguish successful output, a model refusal, invalid output, and an infrastructure error. Those states need different responses. Retrying a temporary transport failure can be reasonable; repeatedly submitting a request that cannot satisfy the output contract simply creates more cost.

Retain the original inputs and the exact model configuration until the job is reconciled. If only a small fraction of items fail, retry that fraction. Resubmitting the entire batch can duplicate work and make it unclear which result should become authoritative.

Work backward from the real deadline

An overnight reporting pipeline should leave room for validation and repair after inference completes. If the report must be ready at 8 a.m., a submission at 7 a.m. is not protected by an attractive median observed during someone else’s beta. Plan around the service’s stated completion window and your own fallback.

Batching also changes how results enter a database. Write outputs to a staging area, validate the full set, and only then publish the new version. That avoids showing readers a mixture of old and newly generated classifications while a job is still running.

The launch makes deferred AI work easier to price explicitly. Teams can place interactive requests on one budget and scheduled processing on another. The discount becomes useful when the surrounding application treats waiting, partial failure, and reconciliation as normal states rather than unusual exceptions.