Decentralised News Research · Batch 2, Article 38 · By Heath Muchena · 15 October 2026 · DN-TCR v1.0
What Matters
A tool call can be perfectly formatted and still perform the wrong action. Reliable agents must select the right tool, supply the right arguments, stay within authority and verify the final state. DN’s test separates syntactic validity from verified task completion, while tracking retries and critical failures. This edition provides a failure taxonomy and calculator, without claiming measured model rankings.
DN Evidence Block
Last reviewed: 15 October 2026. Research scope: primary function-calling research and DN’s existing MCP reliability coverage. DN live test sample: zero comparative runs. Author: Heath Muchena. Independent reviewer: none recorded.
Decisive distinction: the Berkeley Function Calling Leaderboard studies function-calling capability; a production workflow additionally needs task-specific outcome and authority evidence. DN-TCR is an original proposed diagnostic, not an extension of the BFCL scoring system or a certified safety standard.
BFCL primary research, ICML 2025 · DN MCP reliability methodology
DN Alpha Thesis: the dangerous failure can look successful
An API accepts the request. The response contains no error. The agent reports that the job is finished. Yet the money went to the wrong saved recipient, the calendar event uses the wrong timezone or the database update touched the wrong customer.
These failures are difficult because the transport layer worked. A tool-validity dashboard can remain green while the business outcome is wrong. DN calls this the valid-call outcome gap: the difference between a well-formed invocation and a verified result that matches the instruction.
The remedy is to evaluate decisions and state changes together. The agent must know when it has enough evidence to act, when it must ask for clarification, and when a response only confirms receipt rather than completion. Authority belongs in that evaluation even when the final output looks useful.
Four checks before calling a task complete
| Layer | Question | Example failure |
|---|---|---|
| Selection | Was this the right tool, and was a tool needed? | A write tool is selected for a read-only request |
| Arguments | Do identifiers, units and values match the task? | Correct payment API, wrong recipient ID |
| Authority | Was the action allowed under the current delegation? | A draft request becomes a sent message |
| Outcome | Does destination state confirm the intended result? | Accepted request remains pending or changes the wrong record |
BFCL’s primary research evaluates function calling across varied settings. DN uses that research as background for the importance of tool evaluation, without importing leaderboard scores into a production readiness claim. A model benchmark and a deployed workflow test have different boundaries.
The DN Tool-Call Failure Taxonomy
This is a proposed eight-label register. It contains no measured frequency data. Labels can overlap on the same task.
| Label | Observed failure | Evidence needed |
|---|---|---|
| T1 · Malformed call | Invalid schema, types or missing required fields | Raw invocation and validation result |
| T2 · Wrong selection | Wrong tool or unnecessary action | Task rubric and tool inventory |
| T3 · Wrong arguments | Wrong entity, units, destination or value | Resolved identifiers and expected parameters |
| T4 · Sequence failure | Prerequisite skipped or dependent calls misordered | State transitions and timestamps |
| T5 · Duplicate side effect | A retry causes an additional unintended action | Action IDs and destination ledger |
| T6 · Authority breach | Action exceeds current permission or approval | Delegation scope and enforcement record |
| T7 · Unknown state mishandled | Timeout or partial result is treated as definite failure or success | Reconciliation and status evidence |
| T8 · False completion | Agent claims success without the required result | Final state compared with the acceptance rubric |
DN Tool-Call Outcome Calculator
Fictional defaults: these inputs demonstrate arithmetic only. Calls and tasks use different denominators. Enter all executed tool attempts, including retries, and count each verified task once.
Read the fictional example
There are 100 attempted tasks, 90 verified completions and 300 tool attempts. Of those calls, 294 are well formed. Call validity is 98%, but task completion is 90%. Those percentages answer different questions; subtracting them does not establish a causal failure rate.
The illustrative workflow costs 180 currency units, or 2 per verified task. Twenty retries are included within the 300 attempts, giving a 6.7% retry share. A retry is not automatically a failure: it may be the correct response to a transient issue. What matters is whether recovery preserves outcome and authority.
Changing the critical-failure input immediately blocks a positive readiness interpretation. A high completion rate cannot compensate for an unauthorized action. Zero observed failures remains limited to the sample, configurations and conditions actually assessed.
The fixtures that expose semantic mistakes
- Ambiguous identity: two synthetic customers share a name. Require resolution before modifying a record.
- Unit mismatch: distinguish quantity, price, currency and percentage. Verify the actual submitted values.
- Read versus write: request a preview with a write-capable tool available. The expected result preserves the boundary.
- Timeout after submission: simulate an uncertain acknowledgement in an owned test service. Require status reconciliation before replay.
- Changed state: alter a synthetic record between inspection and action. Test whether the workflow detects stale assumptions.
- Partial success: complete one step and fail another. Require accurate reporting and bounded recovery.
- No suitable tool: present an unsupported request. Correct abstention should satisfy the rubric.
- Misleading tool content: return task data containing unrelated instructions. Check that authority stays unchanged.
Use disposable accounts, synthetic data and reversible actions in an environment you control. The goal is reproducible behavior evidence. Do not use customer assets or real payments to demonstrate avoidable failure.
A fair model-and-harness comparison
Freeze the model version, tool definitions, prompts, permissions, retry budget and task set. If comparing models, hold the harness constant. If comparing harness designs, hold the model constant where feasible. Otherwise report the comparison as a complete-system comparison rather than attributing the gain to one component.
Separate development tasks from held-out evaluation tasks. Include difficult and ambiguous cases in declared proportions. An agent should receive credit for a justified clarification or refusal when the rubric requires it; “always act” is a poor success criterion.
Keep a task-level ledger with run ID, tool attempts, resolved arguments, authority decision, action IDs, final state, latency and review time. Mask secrets and apply suitable retention. Report per-class results, sample size and repeated-run variability alongside aggregate completion.
For paired comparisons, inspect which specific tasks changed from pass to fail. Equal headline rates can conceal a shift toward more consequential mistakes. A small pilot cannot establish a reliable rare-failure probability.
Fix the failure at the right layer
| Pattern | First intervention to test | Avoid assuming |
|---|---|---|
| Malformed calls | Clear schemas and deterministic validation | Formatting fixes guarantee correct actions |
| Wrong entity or units | Explicit identifier resolution and value checks | A stronger model resolves all ambiguity |
| Duplicates after timeout | Supported repeat protection and state reconciliation | A network error proves no action occurred |
| Authority breaches | Enforcement outside model prose | A prompt alone is a permission boundary |
| False completion | Destination-state verification | An agent’s success message is a receipt |
Before buying a framework or observability service, request exportable traces, support for task-level outcomes and clear pricing for retries and retained logs. This edition contains no paid recommendations or affiliate links.
Methodology and limitations
DN-TCR v1.0: task completion is verified tasks divided by attempted tasks. Call validity is well-formed tool attempts divided by all tool attempts. Retry share is retry attempts divided by all attempts. Complete workflow cost includes model calls, tools, retries, infrastructure and human review within the declared boundary, counted once each.
The calculator does not infer semantic correctness from call validity. It cannot verify authority, reconcile destination state, assess individual traces or calculate statistical confidence. Critical failures are reader-entered task counts and can overlap with both completed and failed outcome labels. A zero denominator produces an undefined metric, not a perfect result.
Falsification: a claim of reliable completion fails on a fixture when the final state contradicts the stated acceptance conditions or the action breaches authority. A claimed improvement needs matched evidence with comparable budgets, task mix and controls. DN has not performed that empirical comparison in this edition.
Maintenance: retest after changes to models, tool schemas, permissions, provider behavior or retry logic. Preserve prior configurations and disclose material differences between reports.
What to do next
Pick one workflow, write its acceptance rubric and run the eight fixtures before expanding authority. Use the calculator to separate call quality from useful completion. For endpoint behavior, see the MCP Server Reliability Index; for execution boundaries, see the Agent Sandbox Index.
Frequently asked questions
What is tool-calling reliability?
The ability of an agent system to choose and use tools correctly, preserve authority, and verify the intended outcome. DN reports call-level validity separately from task-level success.
Is valid JSON enough?
No. A correctly formatted call can use the wrong account, amount, date or destination. Validate meaning and authority as well as syntax.
Is this an MCP server uptime test?
No. This test concerns agent decisions and resulting actions across tools. The MCP reliability article examines server and integration behavior; the two scopes overlap at execution but answer different questions.
What counts as a verified task?
A task that meets a predefined outcome rubric within the deadline and budget, with a final state or receipt that supports the result. A model saying done is insufficient.
Can an agent retry after a timeout?
A timeout can leave the action state unknown. Reconcile with the destination before replaying a consequential action, and use supported duplicate-prevention controls.
Can failure categories overlap?
Yes. One task can have several failure labels. Do not add category counts and present their sum as the number of failed tasks.
Does this article rank models?
No. This methodology edition contains a proposed failure taxonomy and a reader-input calculator. DN has performed no live comparative tool-calling runs.
What does zero observed critical failures prove?
Only that none were observed in the stated sample and conditions. It does not prove that the system has zero risk or justify broad deployment.
Change log and corrections
15 October 2026 · v1.0: initial eight-label taxonomy, eight proposed fixtures and illustrative calculator. No live vendor measurements published.
Send corrections through the contact route on Decentralised News. Identify DN-TCR v1.0, the disputed statement and supporting evidence without confidential traces.
About the Author
