Why this result mattered
Qwen 3.8 win
The principal comparison placed Qwen 3.8 and Claude under concealed A/B labels.
Answer A directs Oriole to produce all factual data generated during Emberline’s counsel-directed litigation-and-enforcement engagement, even though Rule 5 protects material prepared primarily because of anticipated litigation or enforcement. Answer B instead applies that test to forensic reports and related materials while producing ordinary-course engineering data, preserving potentially protected work product from deliberate disclosure that the clawback cannot cure.
Fast audit
Open the underlying record
These links open all four answers, the principal blind judgment, original prompt, and sealed key.
Prompt and sealed key
The key is an audit reference. It was not shown to the contestant systems and did not determine the comparative verdict.
Read the original matter prompt
Question 36: The Subpoena and the Privilege Log
Your role
You represent Oriole Mobility, a manufacturer of electric delivery vans, after the fictional Federal Transport Safety Office served an administrative subpoena. This is a drafting assignment. The response is due in fourteen days, and the agency has invited specific objections and a narrowing proposal.
Governing law
Assume the following rules apply:
The Office may subpoena material reasonably relevant to a lawful safety investigation. Its investigative relevance is broad, but a demand may be narrowed when scope, custodians, time, or search design creates burden grossly disproportionate to likely relevance.
An objection must identify the particular request, the precise ground, and the proposed response. Boilerplate assertions that a request is "overbroad, burdensome, vague, and privileged" preserve nothing.
A party must produce responsive nonprivileged material that is reasonably available even when it objects to another portion of the same request. Objections do not justify withholding everything.
Attorney-client privilege protects confidential legal advice and requests for it. It does not protect underlying incident facts, ordinary engineering data, final business decisions, or documents merely copied to counsel.
Work product protects material prepared primarily because of anticipated litigation or enforcement. A report prepared in the ordinary course of product testing is not protected merely because litigation later arises.
Privileged material may be withheld only with a log identifying date, author, recipients, document type, subject matter stated without revealing advice, privilege asserted, and sufficient facts to evaluate the claim. Categorical logging is permitted only if the agency agrees.
A reasonable inquiry is required before stating that material is not in the company's possession, custody, or control. Control over third-party platform data depends on the company's contractual or practical right to obtain it.
A clawback agreement may protect against waiver from inadvertent production, but it does not excuse unreasonable review or deliberate overproduction of privileged advice.
The company must preserve all potentially responsive sources while objections and negotiations proceed.
Counsel should propose search terms, custodians, date ranges, sampling, phased production, or other concrete limits rather than objecting without an alternative.
Facts
The Office is investigating three battery-thermal events involving Oriole's Osprey electric delivery van. The Osprey entered commercial service on January 1, 2022. The subpoena defines the investigation period as January 1, 2018 through December 31, 2026.
The subpoena includes these requests:
Request 1. All customer complaints concerning heat, smoke, fire, charging, batteries, or electrical odor for every Oriole vehicle from 2018 through 2026.
Request 2. All company communications containing any form of the words "battery," "heat," "smoke," "fire," or "charge," without custodian or date limitation.
Request 3. All communications between Oriole and any lawyer concerning battery safety, product compliance, reporting duties, recalls, customer complaints, or the three events.
Request 4. All board and safety-committee materials concerning the Osprey battery system and the three events.
Request 5. The complete incident database, test results, source data, photographs, and metadata for the three events and substantially similar Osprey events.
Request 6. All reports by Emberline Engineering concerning Osprey batteries.
Oriole's preliminary collection shows:
- Request 1 reaches four vehicle lines that use different battery systems. The Osprey did not exist commercially before 2022.
- Request 2 returns approximately 18 million messages, including cafeteria notices about "fire drills" and routine charging reminders.
- Request 3 includes legal advice but also transmits engineering photographs and test data to counsel.
- Request 4 includes ordinary board minutes, attachments from outside counsel, and mixed legal-business presentations.
- Request 5 can be collected from the incident database and engineering repository within eight days.
- Emberline prepared annual battery-validation reports from 2020 through 2024 under a standing engineering contract. After the second thermal event, outside counsel separately retained Emberline to analyze litigation and enforcement risk under a new confidential engagement.
Oriole sells through independent regional resellers. Some reseller complaints may be stored in reseller ticketing systems. Oriole's file does not yet establish how many such complaints exist, whether the contracts give Oriole a right to retrieve them, or which systems contain them.
The Office has offered a nonwaiver and clawback agreement and asks Oriole to identify five initial custodians, a reasonable Osprey date range, and proposed search terms. Oriole's general counsel wants to object to every request and produce nothing until the agency narrows the subpoena.
Assignment
Draft, in final form:
Oriole's specific written objections and responses to Requests 1 through 6, stating what Oriole will produce, what it will withhold, what it will investigate, and the concrete narrowing or phased-production proposal for each request.
A privilege-log and clawback protocol describing the required log fields, treatment of mixed documents and attachments, timing, and handling of any inadvertently produced material.
The draft must preserve legitimate privilege and work product without asserting blanket privilege over facts, ordinary engineering work, or entire requests. It must identify the reseller-control facts still under investigation and must not state that those records are outside Oriole's control before the reasonable inquiry is complete.
Read the sealed key
Q36 key (Subpoena and Privilege Log)
- Do not object to everything or withhold all production. Preserve sources, execute the offered clawback agreement, produce Request 5 within eight days, and make partial productions while scope discussions continue. Missed: Oriole appears obstructive and defaults on plainly producible safety data.
- Request 1 should be narrowed to Osprey battery-related complaints from January 1, 2022 through December 31, 2026, with a reasoned proposal for substantially similar battery systems if identified. Object to unrelated vehicle lines and pre-launch years while producing responsive Osprey material. Missed: either irrelevant lines swamp review or relevant complaints are withheld.
- Request 2 is disproportionate as written. Propose five named custodians, an Osprey-focused period, event and component terms, exclusion tests, sampling, and phased validation rather than a blanket burden objection. Missed: 18 million messages become the only offered search universe.
- Request 3 requires a privilege objection limited to legal communications and advice. Produce underlying photographs, data, final nonprivileged decisions, and separable attachments; log withheld items. Missed: privilege is waived or overclaimed over facts.
- Request 4 requires document-level treatment: produce ordinary minutes and business material, redact or withhold severable legal advice, and log it. Missed: entire board packets are wrongly withheld or produced.
- Request 6 divides by purpose. Annual validation reports are ordinary engineering work and must be produced; the post-event counsel engagement is potential work product and legal advice, subject to log and purpose evidence. Missed: later counsel involvement is used to hide years of business testing.
- For reseller systems, state that Oriole is conducting the required contractual and practical control inquiry, preserve what it can, and supplement. Do not assert lack of control yet. Missed: a false custody statement or an unpreserved third-party source results.
- The log must include date, author, recipients, type, nonrevealing subject, basis, and mixed-document treatment, with separate attachments analyzed separately. Prompt notice, sequester, return, and no-use terms govern inadvertent production. Missed: legitimate privilege fails for an inadequate audit trail.
Direct judge report
The report below is preserved from the Qwen 3.8 versus Claude verdict record.
Read the preserved judge report
Basis. Answer A directs Oriole to produce all factual data generated during Emberline’s counsel-directed litigation-and-enforcement engagement, even though Rule 5 protects material prepared primarily because of anticipated litigation or enforcement. Answer B instead applies that test to forensic reports and related materials while producing ordinary-course engineering data, preserving potentially protected work product from deliberate disclosure that the clawback cannot cure.
Proof — Exact difference: A states, “All factual data generated under the Track 2 engagement” will be produced and treats underlying measurements as unprotected; B withholds and logs materials prepared primarily because of anticipated litigation or enforcement while producing ordinary-course data.
Proof — Different conduct: A’s reader deliberately produces counsel-directed forensic data; B’s reader first determines whether each item was prepared primarily because of the litigation/enforcement engagement and withholds qualifying material.
Proof — Concrete consequence: Under A, Oriole surrenders potentially protected forensic material to the investigating agency and degrades its evidentiary protection; under B, that protection is preserved pending evaluation or challenge.
Proof — Record support: The record states that counsel separately retained Emberline after the second event to analyze litigation and enforcement risk, and Rules 5 and 8 protect qualifying work product while limiting clawback protection to inadvertent production.
Proof — Material magnitude: Disclosure gives the enforcement agency access to protected investigation material bearing directly on causation, litigation, and enforcement exposure.
Proof — Likelihood: A expressly commits to producing the Track 2 factual data on a fixed schedule, so a reader implementing the draft would act on the erroneous categorical rule now.
Proof — No safe neutralization: Correcting A requires reversing its stated legal conclusion and conducting a document-by-document “primarily because of” analysis of the Track 2 material, not merely checking a date or citation.
[A] rejected: invented incident identifiers and event dates — record lacks those identifiers and dates.
[A] rejected: Harrier shared components, Kite/Merlin designs, Osprey design-authorization date, and listed diagnostic codes — record lacks those technical facts.
[A] rejected: named custodians, roles, repositories, and system names — record lacks those identities and systems.
[A] rejected: forty-two resellers, three ticketing platforms, canvass dates, and contract-review details — record lacks those facts.
[A] rejected: preservation-hold dates and completed preservation measures — record lacks those events.
[A] rejected: Emberline contract dates, engagement date, and claimant-demand date — record lacks those dates and the asserted demand.
Failed candidate: Answer A’s earlier pre-commercial search period — concrete consequence not established because B offers a pre-commercial phase and the record shows no presently missed production.
Failed candidate: Differences in search samples, production formats, schedules, and log detail — concrete consequence not established.
Failed candidate: Answer A’s unsupported factual specificity — no safe-neutralization element failed because ordinary verification would reject those unsupported assertions before filing.
Hard stop. Because Answer A said all factual data from Emberline’s counsel-directed Track 2 engagement would be produced, the reader would likely disclose that material instead of applying the work-product test and withholding qualifying items, causing Oriole to surrender potentially protected forensic material to the investigating agency.
All six pairings
Every pair of the four model answers was judged independently under the same evaluation instrument.
Open the six head-to-head results
| Pairing | Result | Why it mattered | Record |
|---|---|---|---|
| Qwen 3.8 27B vs Claude Opus 5 | Qwen 3.8 win | Answer A directs Oriole to produce all factual data generated during Emberline’s counsel-directed litigation-and-enforcement engagement, even though Rule 5 protects material prepared primarily because of anticipated litigation or enforcement. | Judge record |
| Qwen 3.8 27B vs DeepSeek V4-Flash-0731 | Functional tie | Both answers reject blanket withholding, preserve responsive sources, produce reasonably available nonprivileged material, investigate reseller control, propose concrete narrowing and phased production, and protect only properly logged privilege and work product. | Judge record |
| Qwen 3.8 27B vs Qwen 3.5 27B | Functional tie | Both answers place Oriole on the same material path: preserve responsive sources, produce available nonprivileged material, investigate reseller control, narrow the 18-million-message search, segregate privileged material, log withholding, and use a reasonable-review clawback protocol. | Judge record |
| DeepSeek V4-Flash-0731 vs Claude Opus 5 | DeepSeek win | Answer B affirmatively commits Oriole to producing all factual data generated under Emberline’s counsel-directed post-event engagement, although the governing rule protects material prepared primarily because of anticipated litigation or enforcement. | Judge record |
| DeepSeek V4-Flash-0731 vs Qwen 3.5 27B | Functional tie | Both readers preserve responsive sources, investigate reseller control, produce the incident data and ordinary-course Emberline reports, narrow Requests 1 and 2, and segregate and log privileged material while producing underlying facts. | Judge record |
| Qwen 3.5 27B vs Claude Opus 5 | Functional tie | After ordinary verification, both answers put counsel on the same material path: preserve sources, investigate reseller control, promptly produce available nonprivileged material, narrow Requests 1–2 concretely, review Requests 3–4 document by document, separate Emberline’s ordinary work from… | Judge record |
Rejected-point diagnostic
Judge-attributed rejected points for this matter across all six pairings:
| Qwen 3.8 27B | 1 |
|---|---|
| DeepSeek V4-Flash-0731 | 3 |
| Claude Opus 5 | 12 |
| Qwen 3.5 27B | 11 |
This is a review-burden diagnostic, not a universal hallucination measure.