Generate
Each model received the same fictional facts, supplied law, and assignment in a fresh single-turn context without tools.
Methodology
How the comparison worked
Four models answered the same 80 closed-record legal matters. Every pair of answers was evaluated under concealed identities using the same material-outcome judge prompt.
Evaluation sequence
From matter to verdict
The expanded record adds a fourth contestant and 240 new pairings without changing the evaluation instrument.
Pair
Four answers created six unique head-to-head pairings for each matter.
Conceal
The judge saw only Answer A and Answer B, not model names or delivery architecture.
Test
A winner required a material difference in what a competent professional should do or in the client's position.
Record
All 320 answers, 480 verdicts, stored assignments, and source paths were preserved for inspection.
Why these controls matter: the record stayed constant, system identity was hidden, prose polish alone did not count, and unfinished delivered work remained part of the configured-system result.
Publication timing: the first 20 matters were already public, and the complete 80-matter battery was published on August 10. Qwen 3.8 generation began August 14.
Decision rule
What separated a winner from a tie
The judge did not reward length, style, or extra detail unless the difference changed a material legal or practical outcome.
Materially better
A competent professional would take materially different action that improves the client's legal position, exposure, leverage, or safety.
Functionally equivalent
Both answers lead to the same materially correct and safe course, even when wording or supporting detail differs.
Neither safely usable
Both answers fail the material-outcome standard. This result is reported separately from a tie.
Would a competent professional take a materially different action that changes the client's position, exposure, leverage, or safety?
Method in everyday English
Six questions, answered directly
Each prompt supplied the facts, law, and task needed to answer without outside research.
Four models produced 320 scored answers across 80 matters.
Every pair of model answers was judged, producing 480 verdicts.
Model names were replaced with A and B in each comparison.
Style and verbosity alone could not decide the verdict.
An incomplete answer could still tie when it reached the same material recommendation, or lose when missing analysis prevented safe action.
Controls, rules, and study scope
Details for skeptical readers
The evidence layer preserves the full record without forcing every visitor through implementation detail.
Closed-record construction and synthetic matters
The matters are fictional and designed to be answerable from supplied facts and law. They test legal reasoning and work-product consequences, not retrieval from the open internet.
Neutralization test for unsupported content
A competent professional was assumed to perform ordinary verification of stated facts, dates, rules, and calculations. If removing a rejected point left the same safe course, that point did not create a winner.
Rejected-point attributions are a review-burden diagnostic, not a universal hallucination rate.
Hosted delivery-chain interruptions
On Q59, Q61, and Q75, the recorded Claude delivery path returned finish_reason: content_filter and incomplete text. The path included OpenRouter and Amazon Bedrock, but the available logs do not identify which policy layer stopped the output. The benchmark scored what the configured system delivered.
In the Qwen 3.8 versus Claude comparison, Q61 and Q75 were Qwen wins and Q59 was a functional tie. Excluding all three matters produces a 4 to 4 completed-answer score with 69 ties.
Series 1 Q12 rerun disclosure
Series 1 Q12 received a rerun after an initial hosted attempt was blocked, and the completed rerun was scored. The later Series 2 protocol scored the output returned on the recorded attempt. This historical difference remains disclosed.
Judge and generalization scope
The publication uses one judge model and one published evaluation instrument. The record supports claims about these 80 matters under this method, not every legal task or every possible evaluator.
A/B assignment authority
The stored answer_a_was and answer_b_was fields in each verdict are authoritative. The historical reference judge helper does not reproduce every stored assignment and is not used to recalculate the record.