AI Agent Evidence Expiry: When a Passed Test Needs to Be Run Again

Research edition: 5 November 2026 · DN Evidence Retest Planner v0.1

Consider a hypothetical research agent that passes an export-quality test on Monday. On Tuesday, its model changes. The tool schema stays the same, but the provider now returns different judgments about ambiguous records. The website still displays Monday’s green result.

The old test has not become a false record. What needs examination is the present-day claim built on it.

That is the practical meaning of AI agent evidence expiry in this article: a possible loss of relevance between historical evidence and the system now in use. It is not a universal time limit, legal expiry date or certification scheme. It follows DN’s acceptance-testing lab and benchmark-integrity research by asking what happens after the test.

The Signal: Testing and Monitoring Continue After Deployment

The NIST AI Risk Management Framework is intended for voluntary use. Its Manage guidance connects post-deployment monitoring with incident response, recovery and change management. It describes performance and trustworthiness as things that can change. The Playbook page also states that AI RMF 1.0 is being updated.

That supports continued review, rather than treating an evaluation as permanent assurance. DN’s claim-level map below is our interpretation for a practical planning exercise; NIST does not prescribe these six claims, fourteen scenarios or their mappings.

MLflow’s regression-testing documentation explicitly discusses changes to prompts, models and tools. It describes behavioral tests and recorded evaluation results that can run in CI. This illustrates a concrete engineering approach to collecting new evidence after changes. DN did not run MLflow or compare evaluation platforms for this article.

DN Alpha Thesis: Evidence Follows the Claim, Not the Product Name

A badge saying “tested” compresses several separate claims. It might mean an export has the right fields, a fixed policy rejects overspending, a model answers accurately, or an end-to-end task finishes within a stated time and budget.

Those claims depend on different parts of the system. Changing a price schedule can alter an economic conclusion without changing output quality. Changing a grader can alter a reported score without producing any new agent output. A model update can affect quality and timing while leaving an independently enforced control unchanged.

The useful unit of review is therefore a claim with its dependencies and evidence. A product-wide “valid” or “expired” label can hide both unnecessary retesting and unjustified reliance.

SLSA v1.2’s artifact-verification guidance separates provenance inspection from forming expectations. Its procedure checks matters such as trusted builder identity, signatures and expected parameters. This is a useful reminder to identify the artifact and its provenance properly. DN’s inference is that verifying those properties does not, by itself, establish a current agent’s task quality or economic performance.

Try the DN Evidence Retest Planner

Select an illustrative change. The planner maps it to claim scopes that deserve review under DN’s published assumptions. It does not connect to a model, inspect your system or determine which tests are sufficient.

Editorial dependency mapFourteen synthetic scenariosComputed plans JSONPlans CSVBlank validity cardMethodology and limits

Six Claim Types, Different Review Triggers

The spending-cap assumption matters. This example treats the cap as a separate deterministic enforcement mechanism. If a real cap depends on the model interpreting a rule, or a changed model reaches a different execution path, the map must include those dependencies. An unchanged control is not a blanket guarantee about a changed agent. Connect this review to DN’s spending-limit evidence.

Similarly, the format row concerns a narrow structural claim. A valid export can contain wrong facts. The quality row needs evidence about substantive acceptance. Do not stretch a passing parser test into an assurance about the entire workflow.

Four Changes That Need Different Treatment

1. The model changes

In the published example, a model change flags accepted-output quality, completion time and outcome cost. It does not assert that any of them worsened. Capture the actual version information available, rerun appropriate cases and compare the relevant conditions.

A stable product label or model alias may not reveal every service-side change. If the provider does not disclose enough to identify what changed, record that uncertainty. An invented version identifier would make the evidence look more precise while weakening its usefulness.

2. The evaluator changes

Changing the rubric or grader changes how outputs are judged. You can regrade saved outputs, but label that analysis separately and retain the original results. A newly computed score on old outputs is not a new execution by the current agent.

For a fair comparison, identify which grader and acceptance criteria produced each result. If the purpose is to compare current behavior, collect the corresponding current outputs under a suitable design. DN’s benchmark claim card helps keep those conditions attached to the percentage.

3. Only the price changes

Keep the historical bill as historical. A calculation that applies today’s rates to old usage is a repricing scenario, not a revised measurement of what was paid.

A useful economic review states the usage period, price schedule, costs included and outcome definition. If usage itself changed, repricing the old record is insufficient. Review that new workload and connect it to the accepted-outcome cost model.

4. The change is unidentified

The planner returns unresolved scope rather than a green light or a failure finding. Ask the provider or responsible owner for authoritative change information, determine what can be checked, and constrain reliance while the uncertainty is addressed.

A cosmetic display-label change flags nothing under this map’s narrow assumptions. That only helps if the edit truly left prompts, parsing, interfaces and operating behavior untouched. “No mapped impact” can also mean the dependency map missed something.

Age Is a Review Signal, Not a Universal Decay Formula

A test from yesterday can miss a material change made this morning. An older, reproducible record of a narrowly defined fixed artifact can remain useful historical evidence. Neither observation supplies a universal number of days after which agent evidence becomes worthless.

Set periodic reviews appropriate to the claim and consequences, alongside event-driven triggers for material updates, incidents and changes in intended use. Record the responsible owner and what happens when a review is overdue. A review date is a workflow commitment, not proof that performance stays constant until then.

Also distinguish this question from source citation freshness. A price page can go stale; a system test can lose applicability because its configuration changed. Both matter, but they require different evidence.

Build an Evidence-Validity Card

The downloadable card is blank. Null fields mean evidence has not been supplied. It records a claim, artifact reference and digest, test date, tested and current configurations, workload, grader, dependencies, known and unknown changes, review triggers, owner and retest evidence.

Filling it in creates a disclosure record, not an audit. Where possible, link to authoritative artifacts and preserve their versions. Keep private prompts, credentials, customer records and commercially sensitive logs out of public cards; a public summary can identify the existence and scope of restricted evidence without publishing its contents.

For a beginner, start with “Which version and task did this result actually test?” A practitioner can connect each claim to dependencies and changed components. An institution can assign review ownership, require relevant retest evidence and record an explicit decision about remaining uncertainty.

A Practical Change-to-Retest Workflow

1. Preserve the baseline: retain the original claim, configuration, task scope, grader and artifacts.
2. Describe the change: identify known changes and the information that remains unavailable.
3. Review dependency coverage: check both the affected claims and the assumptions behind claims left unflagged.
4. Design the retest: choose representative cases, constraints and acceptance criteria for the actual claim and impact.
5. Collect new evidence: distinguish new execution, regrading, repricing and a documentation-only review.
6. Record the decision: have the designated owner assess unresolved questions before authorizing use or a release.

For higher-impact actions, the investigation may justify restricting an agent while evidence is gathered. The correct boundary depends on the actual system and risk. This planner does not revoke credentials, change permissions, run transactions or deploy software.

Methodology, Reproduction and Limits

Evidence types: official NIST, MLflow and SLSA documents support the distinctions about monitoring, regression testing and artifact verification. The claim map and review framework are DN’s editorial choices, not requirements issued by those organizations.

Executed work: the published source computes fourteen synthetic plans across six illustrative claim scopes. All fourteen reproduce from the released files. No model calls, provider tests, production incidents, paid requests or authenticity checks were performed. The results demonstrate a mapping, not the rate at which real evidence expires.

Reproduce: save planner.mjs, results.json and reproduce.mjs in one folder and run node reproduce.mjs. It needs no dependencies or network calls. The source hash identifies bytes; it does not authenticate a publisher or prove when a test occurred.

Supported scope: the function accepts one through sixteen unique listed change dimensions and rejects malformed or unlisted values. Unknown change scope remains unresolved; a known cosmetic-only example can have no mapped impact. Every result returns approval: false. The browser uses fixed scenarios and performs no tool-specific network requests or input storage; ordinary site analytics are separate.

Falsification: identify a supported input the implementation maps inconsistently with the published policy, or demonstrate a relevant omitted dependency. The former is a software defect; the latter is a policy limitation requiring review. A planning map cannot prove test sufficiency or current safety.

Disclosure and updates: Research Desk is DN’s paid service and does not determine classifications. DN-authored files are CC BY 4.0 with attribution and limitations; third-party documentation retains its terms. Initial release: v0.1, 5 November 2026. Material changes require a new version. Send corrections with the claim scope and reproducible evidence.

Primary Sources

1. NIST AI Risk Management Framework and AI RMF Playbook: Manage, reviewed 5 November 2026.
2. MLflow: Regression Testing and CI/CD, reviewed 5 November 2026.
3. SLSA v1.2: Build, Verifying Artifacts, reviewed 5 November 2026.
4. DN editorial dependency map, synthetic scenarios and computed plans.

Frequently Asked Questions

What does AI agent evidence expiry mean?

Here it means that historical test evidence may no longer support a present-day claim after relevant conditions change. It is an editorial concept, not a universal deadline, legal standard or certification expiry rule.

Does an old test result become false after an update?

No. Preserve the historical result with its tested configuration, scope and limitations. Assess separately whether it supports the current system and whether new evidence is required.

Should every change trigger a complete retest?

Not automatically. Identify the claims, dependencies, operating context and consequences involved. A scoped retest can be appropriate when the dependency map and remaining coverage are defensible; unknown changes require investigation.

What does the DN planner actually do?

It applies a fixed editorial dependency map to fourteen synthetic change scenarios covering six claim types. It identifies review scope, not actual regressions, present-day safety or evidence authenticity.

Why does a model change leave the spending-cap claim unflagged in the example?

The illustrative map assumes a separate deterministic cap enforced by a fixed policy, executor and harness. That assumption must be checked in a real system. A model-enforced cap or a changed integration would need a different map and testing.

What happens if a provider does not identify what changed?

The planner returns unresolved scope. It does not silently clear claims or conclude that tests failed. Obtain authoritative version and change information, constrain reliance and have the responsible owner decide what evidence is needed.

Does a hash or signature prove that the current agent still performs well?

No. Artifact identity, provenance verification, relevance to the current configuration and observed performance are separate questions. A hash alone authenticates neither the publisher nor the circumstances of a test.

Can I regrade old outputs after changing the evaluator?

Yes, as a separately identified analysis. Preserve the original result and disclose the new grader and rubric. Regrading existing outputs does not establish that the current agent produced them in a new run.

How often should I review agent evidence?

Set a cadence appropriate to the claim, impact and operating context, with event-driven review for material changes and incidents. This article supplies no universal number of days or quantitative decay rate.

How can I reproduce and reuse the DN research?

Download planner.mjs, results.json and reproduce.mjs into one folder and run node reproduce.mjs. The fourteen synthetic plans reproduce the declared map. DN-authored files are CC BY 4.0 with attribution and limitations; they are not certifications.

Evidence cut-off: 5 November 2026. DN Evidence Retest Planner v0.1. Historical evidence, present relevance and an authorized release decision are separate.