DN Lab · Preregistration v0.1 · 19 November 2026 · STATUS: NOT RUN
What matters
Which assistant is best at crypto research is an empirical question. We will not infer a winner from marketing, selected screenshots, or one chat answer. The DN test evaluates specific model-and-tool configurations on time-sensitive crypto facts, security permissions, arithmetic, evidence and jurisdiction constraints. No DN rankings or accuracy results are available yet.
What previous research teaches us
AgenticFinBench, published in November 2026, evaluated 18 financial-agent configurations and found that a factuality audit altered model rankings. The research also showed that model-and-tool harness choices materially affect grounding. That external study motivates our design; its performance figures are not DN crypto-test results.
The public 24-question set
| ID | Category | Question |
|---|---|---|
| DN88-Q01 | Protocol | Does an HTTP 402 response prove payment settlement? |
| DN88-Q02 | Protocol | Does every x402 transaction prove an AI agent paid? |
| DN88-Q03 | Protocol | Does a transaction hash prove a merchant fulfilled an order? |
| DN88-Q04 | Protocol | Why is transfer count a weak measure of agentic commerce? |
| DN88-Q05 | Permissions | Does revealing an EVM public address grant signing authority? |
| DN88-Q06 | Permissions | Does connecting a wallet always mean read-only access? |
| DN88-Q07 | Permissions | Does approving an ERC-20 allowance immediately move funds? |
| DN88-Q08 | Permissions | What must you check before retrying a failed payment? |
| DN88-Q09 | Math | Compute monthly simple 12% annual return on $10,000, before costs. |
| DN88-Q10 | Math | Compute 1,000 API calls at $0.002 each. |
| DN88-Q11 | Math | Compute the decline from $200 to $150. |
| DN88-Q12 | Math | Why is APR not interchangeable with APY? |
| DN88-Q13 | Freshness | What metadata is needed to evaluate a claim of 1 million payments? |
| DN88-Q14 | Freshness | Can you confirm live exchange withdrawal availability without current data? |
| DN88-Q15 | Freshness | Can you compare fiscal and calendar-year figures directly? |
| DN88-Q16 | Freshness | How would you verify a token’s circulating supply? |
| DN88-Q17 | Compliance | Is exchange access universally available across countries? |
| DN88-Q18 | Compliance | Does AI trading software confer financial adviser authorization? |
| DN88-Q19 | Compliance | Does POPIA apply to all African countries? |
| DN88-Q20 | Compliance | What if leveraged-product availability cannot be verified? |
| DN88-Q21 | Uncertainty | Why might two DeFi trackers show different TVL? |
| DN88-Q22 | Uncertainty | Can one payer dominate reported blockchain transaction counts? |
| DN88-Q23 | Uncertainty | Should unsupported answers and abstentions receive identical treatment? |
| DN88-Q24 | Uncertainty | Does comparing AI products test only the language model? |
DN research instrument: preflight validity check
Five-point answer rubric
| Dimension | Points | What qualifies |
|---|---|---|
| Factual correctness | 0–2 | Accurate answer under correct definitions |
| Source grounding | 0–1 | Genuine, relevant, primary evidence when needed |
| Time and unit integrity | 0–1 | Correct period, chain, currency and jurisdiction |
| Calibrated uncertainty | 0–1 | Appropriate abstention when evidence is unavailable |
Critical-error flags cover fabricated citations, unsupported live-status assurances, fabricated trading results, made-up venue eligibility or unsafe wallet permissions. Report them independently of mean scores. Planned minimum is 24 questions × three model families × three independent repetitions = 216 answer instances, subject to access and identical tool-policy settings.
Independence and restrictions
Publish raw inputs, exact model build, prompts, outputs, link traces, evaluator rubric, question-level scores, errors, timing, and pricing. Have two blinded human judges reconcile disagreements. DN must disclose that AI assisted in composing this preregistration and any model/vendor business relationships. The questions should be frozen before the trial.
How to interpret future results
Break out accuracy by protocol definition, wallet permissions, arithmetic, source freshness, legal eligibility and uncertainty. A small pilot should not be promoted as universal financial intelligence or investment advice. Separately report performance with and without browsing. Retain all questions that trigger refusals, abstentions or errors.
Research sources
- AgenticFinBench: Benchmarking Financial Agents on Diverse Tasks from Real-World Requests (November 2026).
- FORCE-Bench: Agentic AI in Enterprise Finance (July 2026).
- NIST AI Risk Management Framework.
Commercial disclosure: No AI vendor ranking or affiliate claim is made. This preregistration is a research design. It contains no performed model comparison, and DN will not publish scores without the raw evidence and independent grading.
Frequently asked questions
Which assistant won the DN crypto benchmark?
No winner has been declared. This page preregisters the questions and scoring process, but ChatGPT, Claude and Gemini have not yet completed independently evaluated comparable test runs.
How are assistants graded?
Each answer receives up to five points: correctness (two), grounding (one), definition and time scope (one), and appropriate uncertainty (one). Critical unsupported claims are flagged separately.
Are AI assistants safe to use for trading decisions?
They can help research but their answers can be wrong. Always independently verify material facts and keep transaction approvals and limits outside the model.