OpenRouter announced a redesign of its default provider-routing algorithm, moving beyond a predominantly price-based ranking. The new approach considers realized cost, latency, throughput, and reliability when selecting an endpoint for a model request.
The company reports improvements in its own experiments, including lower time to first token and fewer capacity errors, alongside a modest increase in measured cost. Routing remains cache-aware, and developers can still use explicit provider selection or sorting. The change affects provider selection rather than the model a developer requested.
A failed cheap request is still expensive
The listed price of a token captures only part of the user experience. A request that fails after a long wait may require another attempt and delay the rest of a workflow. In an interactive application, that delay can matter more than a small difference in the underlying inference bill.
A useful comparison should therefore measure the cost of accepted responses. Include failed attempts, retries, cache behavior, and the amount of output actually generated. This can produce a different ranking from a table that compares only the cheapest input-token rate.
The appropriate balance depends on the product. A batch job that runs overnight may tolerate slower responses. A live voice interface or an editing assistant may need predictable latency. Developers should make those priorities explicit instead of assuming one routing objective is ideal for every request.
Observe the distribution, not just the average
Average latency can improve while a small group of users still experiences long delays. Track tail behavior and failure rates alongside the mean, particularly during traffic bursts. Segmenting by prompt size and request type can reveal which workloads benefit from a routing change.
Cache affinity deserves similar attention. Moving a request to a healthier endpoint may improve reliability but lose a useful cache. The right tradeoff can change during an incident, so observability should show enough detail to explain sudden shifts in cost or response time.
For teams already using automatic routing, the practical next step is to compare a stable sample before and after the change. Keep the application workload consistent and avoid attributing every improvement to routing if the model or prompt changed at the same time.
OpenRouter's redesign reflects a more complete definition of inference value. The aim is a dependable answer at an acceptable total cost, with enough control for applications whose priorities differ from the default.
Have a correction or a useful perspective?
Get in touch with the publication.
