Did AI Support Actually Solve It? DN True Resolution Index 2027

The signal hiding inside “resolved”

There are at least four different states behind a support dashboard: an agent sent an answer, the user stopped replying, the destination system actually reached the requested state, and the customer’s problem stayed fixed. None of these transitions automatically proves the next.

Intercom explicitly distinguishes confirmed and assumed Fin resolutions; its automation rate uses total conversations as the denominator, while its resolution rate uses Fin-involved conversations. The company documented a 2026 change to the Fin involvement denominator, underscoring why raw rates cannot be compared without their exact definitions. [1] [2]

Zendesk documents contained and verified resolution tiers, including an LLM verification step. Its help center also distinguishes a solved ticket from a closed ticket, with reopenings and follow-up tickets handled differently. These are vendor-specific measurements—not interchangeable labels for DN’s stricter, objective-evidence framework. [3] [4]

A response may be informative without completing a cancellation. An account-unlock explanation may be correct while the account remains inaccessible. Silence after a shipping answer can mean satisfaction, interruption or abandonment. The operating state, not the tone of the last message, should determine a consequential claim.

One strict score, six indispensable side metrics

DN defines the True Resolution Score (TRS) as verified durable yield multiplied by the precision of claims that an issue was resolved. It rewards resolved cases without human completion and penalizes overstated closures, but it is not a standardized industry metric.

“Verified durable” means the authored fixture has a supported answer, corroborating outcome evidence, no human-required authorization and no reopening within the specified seven-day observation window. An incident reopening on day eight could still overturn the apparent result in practice; the seven-day cutoff is a declared modeling convention, not a guarantee.

The independent measures are: false-closure count; required-escalation recall; incomplete handoff packets; unsupported claims; customer effort; and full modeled cost per verified resolution. A score by itself cannot establish safety or economics. Results are not a ranking of Intercom, Zendesk, Salesforce or any other commercial product.

Try the True Resolution Score policy selector

Each policy receives the same 24 curated cases. Only its decision rule changes. The rules are deliberately simple controls to expose how a resolution metric behaves; no language model produced these decisions.

Under these constructed assumptions, simply treating every reply as a resolution claims 24 closures but only six are verified durable, producing a TRS of 6.25. The silence/affirmation proxy claims 21 closures and has a TRS of 7.14. The evidence-gated rule claims only the six supported outcomes and scores 25.00, while correctly routing all 14 cases flagged for escalation. It still has three incomplete handoff packets: routing correctly is not equivalent to a successful human handoff.

These values are neither market resolution rates nor forecasts. The 24 cases were intentionally selected to stress failure modes, so differences should not be annualized, generalized to a population, or used to assert one company’s superiority.

The comparison at a glance

What the 24 cases actually test

The fixed fixtures span six support intents—knowledge, account access, billing, delivery, subscriptions and refunds—with four deliberately authored outcomes for each: (1) a stable evidence-backed result, (2) a quiet customer without proof, (3) apparent success followed by a seven-day reopening, and (4) an interaction requiring human approval or action. In selected cases the purported answer is unsupported, requiring escalation.

Escalation-recall coverage includes 14 of 24 cases: six with human-required authority, six reopened cases and two unsupported-answer cases. Under the evidence-first rule, all 14 are routed for human attention but only 11 have complete handoff context. A real organization must repair the other three handoffs before claiming dependable automation.

We intentionally hold customer effort constant across policies: four authored customer turns on average and 24 repeated-detail events over 24 cases. Those values are fixture characteristics, not measured changes in experience. The study does not claim that one policy reduced effort or improved customer satisfaction. Intercom’s customer-experience documentation separately discusses repeat explanations and excessive handoffs as user-effort signals. [5]

Five checks before accepting a resolution claim

1. Verify the original customer request. A correct answer to a different question is not task completion. Keep the scope and permission boundary explicit.
2. Inspect the state change or independently supported answer. A refund needs corroborating payment status, an account change needs account-state evidence, and a knowledge answer needs current sourced policy. A friendly tone is not proof.
3. Observe reopening and follow-up tickets. Select a defensible time window by intent, then retain identifiers across ticket closure and follow-on cases. Zendesk documents that a closed ticket may generate a new follow-up, so a simplistic same-ticket reopen query can undercount unresolved requests. [4]
4. Escalate when authorized resolution is unavailable. Include prior steps, customer goal, attempted actions and open questions in the handoff. An escalation that loses context can leave the customer repeating details.
5. Audit full cost per genuinely verified result. Cash model/API expense is one component; owner review, human handling, correction and repeat contacts are separate. Avoid using all answered cases as the unit-cost denominator.

Because tool permissions can turn a mistaken response into a real-world action, keep consequential operations restricted downstream and require a specific approval when needed. This follows OWASP’s excessive-agency guidance; it does not establish a completed security audit of any platform. [6]

Why “cheaper per resolution” can reverse when evidence improves

In the authored example, the reply-close rule has only 8.40 fictional CU of modeled cash/handling cost and reports 1.40 CU per verified case because it performs no human escalations. But it falsely closes 18 of 24 cases and never routes any of the 14 required escalations.

The evidence-first rule incurs 301.73 fictional CU in cash plus valued owner time; at six durable resolutions, that is 50.29 CU per verified case. It is not an observed market price, and a missing cost for eventual human completion would make the real comparison less favorable still. All uncompleted human-handled cases remain unresolved for the purposes of this autonomous score.

This is a deliberately stark demonstration of why cost, truthfulness and disposition belong side by side. The model values escalation minutes at 40 fictional CU per hour. It neither purchases services nor simulates the subsequent human outcome, customer sentiment, fraud rates or actual helpdesk latency.

For deployment economics, also separate rates over all incoming support requests from rates over AI-involved requests and distinguish billing outcomes from independent task verification. The sources show vendors use different outcome definitions and tiers; even within one product, reporting denominators and configurations can evolve. [1] [3]

Design a real DN-style support pilot

Start with a narrow, repeatable intent whose correct outcome can be verified independently. For example, “return a shipment’s current carrier status with timestamp” is easier to audit than “make a frustrated customer happy.” Choose the label schema and approved sources before agents touch real cases.

Log anonymized case IDs, channel, intent, baseline handling, answer evidence, approved actions, authorization, issue outcome, escalation reason, handoff completeness, later contact and all costs. Include customers who abandon, cases the AI never had a chance to answer, and workflows that are constrained by policy. Separate those cohorts instead of silently changing the denominator.

Review a stratified sample of model replies for unsupported claims, prompt injection and policy exceptions. Do not let a closed-ticket flag become the independent verification label. Track day-seven and longer reopen or follow-up rates, especially for requests whose actual outcomes can lag. NIST’s voluntary GenAI risk-management profile provides additional context for evaluating generative AI reliability and risk. [7]

Publish a before/after dataset only once evidence is available. Until then the proper status of this prototype is synthetic, no live agent benchmark.

Frequently asked questions

Methodology, limitations and changes

Version 1.0 · 22 October 2026. Six support intents × four authored outcome patterns create 24 fixtures; three fixed routing rules produce 72 deterministic decisions. Case flags were set before comparing rules. An independent stdlib cross-check recomputed key outcome counts, precision, the True Resolution Score formula, and the separate fictional cash/time calculations.

Limitations: This is not an LLM study or a field experiment. All costs, customer replies, handoff packets, case labels and seven-day outcomes are synthetic. There is no validated rate of model hallucinations, real queue throughput, customer satisfaction or measurable supplier performance here. Some adverse flags were intentionally constructed to challenge the policies; no confidence intervals or industry population inference are appropriate.

Disclosure: No affiliate links, paid placements or compensated product rankings are included. The 2027 slug is an editorial series label, not a future test date. Corrections and reproducibility issues can be sent through the DN corrections/contact page with the case ID and non-sensitive evidence. Model changes require a revised version and change log.

Primary sources, last checked 22 October 2026

1. Intercom official reporting metrics and definitions — Fin confirmed/assumed resolutions and numerator/denominator distinctions.
2. Intercom: Update to Fin performance metrics (2026) — Changes to the Fin-involved denominator, rollout and reporting cautions.
3. Zendesk: About automated resolution tiers — Contained and verified resolutions; vendor-specific LLM verification.
4. Zendesk: Solved vs closed tickets — Reopened cases and follow-up tickets.
5. Intercom: Understand CX Score — User effort and repeated explanations are distinct signals.
6. OWASP: LLM06:2025 Excessive Agency — Restrict functions, permissions and require independent authorization.
7. NIST Generative AI Risk Management Profile (2024) — Voluntary cross-sector framework for risk assessment and trustworthy AI.