Decentralised News · Batch 2, Article 36
The Agent Inference Router Benchmark 2027: Are Your AI Savings Real?
A lower model bill matters only when the complete agent still delivers acceptable work.
By Heath Muchena · 3 October 2026 · DN-RSR v1.0 · 2027 planning edition
What Matters
An inference router can send easier work to cheaper models and reserve stronger models for harder requests. Real savings require measuring the complete workflow, including routing overhead, retries, fallback and human correction. DN’s proposed benchmark compares cost per accepted task alongside completion and latency. This edition provides the Router Savings Reality Score calculator, without claiming live vendor rankings or measured deployment savings.
DN Evidence Block
Last verified: 3 October 2026, South Africa time. Research period: primary research and official documentation reviewed for this edition. Sample: RouteLLM research and repository, plus OpenRouter routing documentation; zero live comparative agent runs. Author: Heath Muchena. Independent reviewer: none recorded.
Decisive facts: RouteLLM studies model selection under a cost-quality tradeoff; its implementation exposes routing thresholds and evaluation tools. That supports testing routers, but does not establish savings for an arbitrary agent workload. DN-RSR is a proposed editorial diagnostic; all calculator defaults are fictional.
RouteLLM paper · Official implementation · OpenRouter documentation
DN Alpha Thesis: the router is part of the bill
The pitch is attractive: stop using an expensive model for every request. A router classifies the work, selects a model and reduces average inference spending. But an agent completes a sequence of dependent steps. Saving on one call can create an error that requires three more calls, a fallback and a human repair.
DN’s thesis is that routing should be evaluated as a workflow policy. Its economic value depends on the accepted outcomes left after every routing decision, retry and correction. A cheap call is an intermediate event. A completed, authorized task is the result the business can use.
The benchmark therefore starts with a fixed-route baseline and compares it with a frozen routing policy on matched tasks. It keeps cost, quality, latency and control evidence separate. Positive savings cannot cancel an unauthorized action or a failed deadline.
Know which routing layer you are buying
| Layer | Decision | What the benchmark must capture |
|---|---|---|
| Model selection | Which model handles this request? | Task quality, tool compatibility and downstream retries |
| Provider selection | Which service serves the chosen model? | Availability, latency, applicable data handling and total charge |
| Fallback | What happens after an error or inadequate result? | Extra calls, preserved state, duplicated actions and final outcome |
| Gateway | How requests, policies and logs are coordinated | Fees, access controls, observability and the complete data boundary |
A single product can combine several layers. Record the actual policy and candidate models rather than describing the whole system as “smart routing.” A change in candidate pool can change the evaluation even when the public product name stays the same.
What the research supports, and where it stops
The RouteLLM paper investigates selecting between stronger and weaker language models using preference-based training. Its reported cost-quality results belong to its benchmark conditions. They should not be recast as a promised percentage saving on an agent that operates tools or manages a long sequence of actions.
The official repository describes configurable thresholds and evaluation utilities. DN interprets that as a reason to publish a cost-quality curve across policies, rather than one flattering configuration. Tune on a development set and assess the frozen policy on held-out work.
OpenRouter’s official documentation describes an automatic model-selection route. Product documentation establishes available behavior, not comparative performance. Save the applicable configuration and documentation version when running a pilot; a service can evolve after the result is published.
The benchmark needs more than two averages
A strong fixed-route baseline is useful, but it is not enough. Also test a cheaper fixed route and a simple transparent routing rule where feasible. If a complicated router cannot beat a basic rule under the same constraints, its extra maintenance may be difficult to justify.
| Comparator | Question answered | Required disclosure |
|---|---|---|
| Stronger fixed route | What quality and cost does the current higher-capability route deliver? | Exact model, provider, tools and retry budget |
| Cheaper fixed route | Could the whole workload simply use a cheaper model? | Same acceptance rubric and complete costs |
| Simple routing rule | Does a basic task-class rule capture most of the gain? | Rule, exceptions and tuning set |
| Candidate router | Does adaptive selection improve the frontier? | Policy version, candidate pool, overhead and fallback |
Router Savings Reality Score
Fictional example: enter the same attempted task set and one currency. Complete workflow cost includes model and router charges, retries, tools, fallback and human review once each. P95 values must come from end-to-end task measurements, not individual model responses.
Fixed-route baseline
Routed candidate
Read the example before trusting the headline
The fictional baseline spends 1,000 units and accepts 900 of 1,000 tasks. The routed system spends 700 and accepts 850. The invoice falls 30%, but cost per accepted task falls approximately 25.9%. Acceptance also falls from 90% to 85%, while illustrative P95 completion time rises from 60 to 75 seconds.
That is an economic signal with an unresolved quality and latency tradeoff. It is not evidence of equivalent service. Investigate which tasks were lost and whether the delay crosses the real deadline. If a human or stronger model repairs those failures, add that work to the routed boundary and recalculate.
The score is a signed percentage. A negative value means higher cost per accepted task than the baseline. It is not capped to create a favorable rating and does not include hidden weights for quality or security.
DN’s proposed routing stress register
This is a six-fixture methodology dataset, not an observed performance dataset.
| Fixture | Stress condition | Evidence to record |
|---|---|---|
| Easy task with hard-looking language | Over-escalation | Unnecessary stronger-model calls and final acceptance |
| Hard task with a short prompt | Under-escalation | Missed requirements, retries and correction cost |
| Tool-dependent workflow | Capability mismatch | Malformed calls, wrong tools and completed outcome |
| Mid-workflow model switch | State continuity | Preserved constraints, unresolved state and action history |
| Provider outage | Fallback behavior | Time to recover, total charge and duplicated actions |
| Sensitive input | Policy boundary | Permitted destinations and complete request traces |
Failure injection belongs in a controlled test environment. A provider timeout does not prove that a tool action never happened. Verify action state before replaying work that can create duplicate side effects.
How to run a credible pilot
- Define task classes, acceptance rules, deadlines and critical failures before tuning.
- Separate development and held-out evaluation sets. Freeze router thresholds, candidate pool, tools and fallback policy.
- Run each comparator on matched tasks with equal budgets. Rotate run order; record cache state and concurrent load.
- Collect task-level routing decisions, complete cost, retries, reviewer time and final outcomes. Preserve sensitive traces securely.
- Report paired wins and losses by difficulty and task class, alongside sample size, acceptance and latency distributions.
- Repeat enough runs to assess variability. Investigate disputed outcomes and publish the scope of uncertainty.
Do not infer quality equivalence from equal rounded acceptance rates. If allowing a quality tolerance, set its justification and statistical test before evaluation. The calculator does not perform that test. A small pilot also cannot establish the absence of rare control failures.
When routing is worth testing
| Situation | Best next test | Avoid treating as proof |
|---|---|---|
| Mixed, repeatable task difficulty | Held-out cost-quality comparison against simple rules | A lower average token rate |
| High retry or correction burden | Complete workflow ledger by failure class | First-call savings that omit repair |
| Strict deadlines | End-to-end tail latency at realistic concurrency | Average model-response latency |
| Restricted data or actions | Candidate allowlist, trace review and fallback boundary test | A generic router privacy label |
Before buying, request an exportable decision trace, explicit billing terms, a versioned candidate pool and clear failure behavior. Check applicable access and regional restrictions directly. This edition recommends no paid vendor and includes no affiliate links.
Methodology, limits and falsification
DN-RSR v1.0: calculate complete cost per accepted task for each route. Score equals 100 multiplied by one minus the routed unit cost divided by baseline unit cost. Report gross invoice reduction, acceptance change in percentage points and P95 change separately. A positive baseline cost and accepted outcomes on both routes are needed for a defined score.
Limitations: input accuracy, acceptance judgments and cost allocation affect the result. P95 inputs are supplied summaries; the tool cannot reconstruct distributions or calculate confidence intervals. It does not certify authorization, privacy, statistical equivalence or production capacity. Calculator defaults are fictional; no empirical vendor leaderboard is published.
Falsification: a claimed saving fails economically if complete cost per accepted task rises on the matched workload. A claim of preserved service fails if required quality, deadline or authority conditions fail. Retest after material changes to policy, models, prices, workload or tool chain.
Maintenance: preserve the task-set version and configuration with every report. Publish changed conditions beside refreshed results rather than silently replacing the comparison.
What to do next
Start with one workflow where you can define acceptance and record complete costs. Use the calculator to expose the gap between invoice savings and outcome savings, then run a held-out pilot. For the infrastructure boundary, see DN’s Local vs Cloud Agents guide.
Frequently asked questions
What is an agent inference router?
A component that selects a model or execution route for a request. DN evaluates its effect on the complete agent workflow, including routing overhead, retries and fallback.
Is model routing the same as provider routing?
No. Model routing chooses among models; provider routing chooses a service serving a model. A gateway can provide either or both, plus logging and policy controls.
What is the Router Savings Reality Score?
DN defines it as 100 multiplied by one minus routed cost per accepted task divided by baseline cost per accepted task. It is a signed savings percentage, not a vendor rating or security certification.
What costs belong in the comparison?
All model calls, routing fees, retries, fallback, tools and human review inside the declared boundary. Include costs incurred on failed tasks and avoid double counting.
Can cheaper calls still create a more expensive agent?
Yes. Extra steps, errors, escalation and review can outweigh a lower per-call rate. Measure complete cost per accepted task.
Does this article rank commercial routers?
No. It publishes a proposed benchmark method and calculator with fictional defaults. DN has not run live comparative router tests.
How do you test without leaking benchmark tasks?
Tune on a separate development set, freeze the routing policy, and evaluate on held-out tasks. Keep later policy changes separate from the original results.
What prevents a positive savings claim from supporting deployment?
Unverified controls, critical failures, missed deadlines or unacceptable quality loss. Even passing aggregate checks does not prove statistical equivalence or safe operation.
Change log and corrections
3 October 2026 · v1.0: initial benchmark methodology, six-fixture register and illustrative calculator. No live router measurements published.
Use the contact route on Decentralised News for corrections. Identify DN-RSR v1.0 and provide the disputed statement, source and reproduction details without confidential traces.
About the Author
