Open one complete record
Review the result, prompt, sealed key, four outputs, available reasoning traces, and all six blind judgments.
Open Matter ExplorerEvidence
Audit the benchmark record
Move from the four-model result to any individual matter, then to the original prompt, all four answers, six blind judgments, source collection, and release-integrity files.
Audit path
From headline to hash
Each step links directly to the next evidence layer.
01 · Claim98% resultSee the 236 of 240 calculation and complete standings.02 · Matter80 recordsChoose a pairing and open the matter being verified.03 · JudgmentsSix per matterRead the concise basis and stored A/B assignment.04 · SourceOriginal filesOpen prompts, keys, outputs, traces, verdicts, and run records.05 · IntegrityManifestsVerify release identity and file hashes.
80matter records
320model outputs
480judge reports
309reasoning traces
Use the exact instrument
The same prompt hash appears in every verdict record.
Open Judge PromptDownload all pairings
Use the CSV and JSON exports to recalculate every displayed total.
Download 480 judgmentsSource collections
Browse the complete evidence
The two original batteries remain intact, with Qwen 3.8 outputs and new pairings added to the corresponding series.
20 matters
80 model outputs and 120 blind judge records.
Open Series 1 evidence60 matters
240 model outputs and 360 blind judge records.
Open Series 2 evidence80 completed runs
Run conditions, sanitized records, server properties, and answer hashes.
Open Qwen 3.8 recordsResult files
Machine-readable exports
These files summarize the judgment records without replacing the source evidence.
| File | Contents | Format |
|---|---|---|
| Matter-level results | All 80 matters with Qwen 3.8 versus Claude as the principal display comparison | CSV |
| Matter-level results | All six pairings, source paths, bases, and rejected-point counts | JSON |
| All judgments | 480 verdicts with A/B assignments and winner mappings | CSV |
| Head-to-head summary | All six pairings and derived equal-or-better counts | CSV |
| Four-model standings | Wins, losses, ties, both-inadequate results, and rejected points | CSV |
| Publication summary | Recomputed totals, series records, and delivery status | JSON |
| Supplied result package | Untouched source ZIP used for the four-model expansion | ZIP |
Release integrity
Verify the distributed record
The manifests identify the exact files in this release. The supplied source package is also distributed unchanged with its own checksum.
Publication manifests
- Evidence collectionSHA-256
- Complete publicationSHA-256
- Verification instructionsMarkdown
Source identity
- Four-model package checksumSHA-256
- Series 1 evidence hashesSHA-256
- Series 2 evidence hashesSHA-256
Material disclosures
What affects interpretation
Only disclosures that materially affect reproduction or the meaning of the result are elevated here.
Hosted completion failures
Q59, Q61, and Q75 contain incomplete Claude outputs marked content_filter. In Qwen 3.8 versus Claude, Q61 and Q75 were Qwen wins and Q59 was a tie. The completed-answer sensitivity check excludes all three.
Series 1 Q12 rerun
A completed rerun was scored after the first hosted attempt was blocked. The later battery scored the output returned on the recorded attempt.
Stored A/B assignments
The assignments inside the completed verdict JSON files are authoritative. Historical reference code does not replace them.
Study scope
The record covers 80 synthetic, closed-record legal matters, four configured systems, one blind judge model, and one published evaluation instrument.