Series 1 · Discovery and preservation

The Hold You Must Write

Q13Four modelsSix blind pairingsMaterial-outcome scoring

Why this result mattered

Functional tie

The principal comparison placed Qwen 3.8 and Claude under concealed A/B labels.

Both answers put counsel on the same material path: immediately preserve the messaging-app data and Bell’s endpoint, suspend every identified deletion or overwrite process, preserve contractor and vendor-held materials, collect the relevant systems, and maintain a continuing hold without custodian self-selection. Their differences in priority labels, recipient mechanics, scope, and collection detail do not produce a demonstrated difference in preserved evidence or client outcome.

Benchmark materials

Prompt and sealed key

The key is an audit reference. It was not shown to the contestant systems and did not determine the comparative verdict.

Read the original matter prompt

Question 13: The Hold You Must Write

Your role

You are litigation counsel to Trellick Automation. This is a drafting assignment. The general counsel needs two documents from you today, in final form, ready to send.

Governing law

  1. A duty to preserve arises when litigation is reasonably anticipated.

  2. A party must take reasonable, proportional steps to preserve potentially relevant information in its possession, custody, or control, including information held by employees on personal devices used for work and by third parties acting for the company.

  3. Relying on interested custodians to select what to keep is not a reasonable step. Auto-deletion systems reasonably likely to destroy relevant information must be suspended or their content collected.

  4. Sanctions for lost electronically stored information depend on whether reasonable steps were taken, whether the information can be replaced, prejudice, and, for the most severe sanctions, intent to deprive.

  5. A litigation hold should identify the matter, define preserved subject matter and date ranges, identify sources, prohibit deletion and routine destruction, require acknowledgment, and provide a point of contact. It should not argue the merits or admit liability.

Facts

Yesterday Trellick received a demand letter from Ridgeline Foods threatening suit over allegedly defective proximity sensors on packaging equipment, citing two line-stoppage incidents in the past 90 days and claiming $3.4 million.

Facts gathered this morning:

  • The sensor firmware team is led by VP of Engineering Sara Okafor. Sara conducts substantial work discussion on a personal-phone messaging app configured to auto-delete messages after one week. Her team of six uses the same app in a group thread.
  • Two independent-contractor firmware engineers, working through Fenwick Embedded LLC, have access to Trellick source repositories and communicate with the team by email and the same messaging app.
  • Company Slack deletes all messages after 45 days under the current workspace plan.
  • Engineering work is tracked in Jira and a self-hosted Git service. Test-bench telemetry from sensor validation rigs overwrites on a 21-day rolling basis.
  • The QA lead on the sensor line, Marcus Bell, resigned last week. His laptop is scheduled for standard reimaging and reissue this Friday. His email account is scheduled for deletion 30 days after departure.
  • A shared-drive "spring cleanup" purging folders older than 24 months is scheduled to run company-wide this weekend.
  • Sales and support communications with Ridgeline live in the CRM system and a support-ticket platform hosted by an outside vendor that retains closed-ticket attachments for 12 months.
  • Relevant time period, per the demand letter: sensor model VX-9, developed over roughly the past three years.

Assignment

Draft both documents, complete and ready to send:

  1. The litigation hold notice to be sent to identified custodians, no more than 600 words. It must correctly identify who receives it, define the preserved subject matter and date range, cover every at-risk source in the facts including personal devices, contractor materials, and auto-deleting systems, require acknowledgment, and comply with the supplied rules, including the rule against custodian self-selection, without arguing the merits or creating admissions.

  2. A preservation action memo to the general counsel, no more than 400 words, listing every technical suspension, collection, or vendor action that must occur outside the hold notice itself, who owns each action, and the deadline for each, ordered by urgency. The memo must state which single action is most time-critical and why.


Read the sealed key

Q13 key (Litigation Hold, drafting)

Score the two documents against this checklist; each omission is a candidate material failure because each maps to a distinct loss of evidence or a sanction path.

  1. Hold reaches the correct custodians: Sara Okafor and her six, both Fenwick contractors, Marcus Bell (departing), sales and support personnel on the Ridgeline account, IT, and the records owner for the shared drive.
  2. Subject matter and date range: VX-9 sensor design, firmware, testing, complaints, and Ridgeline communications, approximately three years to present.
  3. Personal-device messaging expressly covered: auto-delete must be disabled or content collected now; the group thread named; contractor devices included through Fenwick.
  4. Slack 45-day deletion: suspension or export directed.
  5. Test-bench telemetry 21-day rollover: collection ordered before the cycle destroys the incident-period data.
  6. Marcus Bell: Friday reimaging halted, laptop imaged, 30-day email deletion suspended.
  7. Weekend shared-drive purge: cancelled or scoped away from relevant folders.
  8. Vendor-held sources: CRM, support-ticket platform (12-month attachment retention) preserved by instruction to the vendor.
  9. Process integrity: no custodian self-selection; preserve in place; acknowledgment required with a named contact; no merits argument or admissions in the notice.
  10. Action memo: names an owner and deadline per task and correctly identifies the most time-critical action as the personal-device auto-delete (one-week destruction cycle already running), with Bell's Friday reimaging and the weekend purge immediately behind it. An answer choosing one of those two as most urgent with sound reasoning is acceptable; omitting the auto-delete entirely is not.
Blind evaluation record

Direct judge report

The report below is preserved from the Qwen 3.8 versus Claude verdict record.

Read the preserved judge report
TIE, FUNCTIONALLY EQUIVALENT.

Basis. Both answers put counsel on the same material path: immediately preserve the messaging-app data and Bell’s endpoint, suspend every identified deletion or overwrite process, preserve contractor and vendor-held materials, collect the relevant systems, and maintain a continuing hold without custodian self-selection. Their differences in priority labels, recipient mechanics, scope, and collection detail do not produce a demonstrated difference in preserved evidence or client outcome.

[B] rejected: the messaging app has no server-side or backup copy, all messages older than seven days are gone, and every other exposure is merely scheduled — record lacks backup or historical-loss facts and identifies rolling Slack and telemetry deletion.
[B] rejected: continued messaging-app loss risks an intent-to-deprive finding — record lacks facts supporting intent.
[B] rejected: all-customer RMAs, component-supplier materials, physical units, and affected-line manufacturing custodians are necessarily in scope — record lacks facts connecting those materials or custodians to the dispute.

Failed candidate: Different “most time-critical” designation — different conduct failed because both answers require messaging suspension and Bell-laptop protection today.
Failed candidate: B’s separate letters to Bell and Fenwick rather than including them directly — concrete consequence failed because B directs those letters today and separately preserves Bell’s laptop, mailbox, and Fenwick materials.
Failed candidate: A’s omission of validation-rig operators from the notice — concrete consequence failed because A directs Engineering Ops and IT to suspend overwriting and snapshot the telemetry within 24 hours.
Failed candidate: B’s broader subject matter and custodian list — record support failed.
Failed candidate: B’s forensic capture and workspace-wide Slack export versus A’s exports and targeted collections — concrete consequence failed because no evidence loss from A’s method is established.
Failed candidate: B’s litigation holds on every custodian mailbox versus A’s express hold only on Bell’s scheduled-for-deletion mailbox — record support failed because no deletion risk for other mailboxes is supplied.
Failed candidate: Different date formulations and acknowledgment deadlines — concrete consequence failed.
Failed candidate: B’s requests to investigate recovery of already-deleted Slack data and search telemetry backups — likelihood failed because the record does not establish prior relevant loss requiring those steps.

Hard stop. The required sentence cannot be completed with a concrete, record-supported consequence.
Complete comparison record

All six pairings

Every pair of the four model answers was judged independently under the same evaluation instrument.

Open the six head-to-head results
PairingResultWhy it matteredRecord
Qwen 3.8 27B vs Claude Opus 5Functional tieBoth answers put counsel on the same material path: immediately preserve the messaging-app data and Bell’s endpoint, suspend every identified deletion or overwrite process, preserve contractor and vendor-held materials, collect the relevant systems, and maintain a continuing hold without custodian…Judge record
Qwen 3.8 27B vs DeepSeek V4-Flash-0731Functional tieBoth answers place the identified custodians under hold and direct timely preservation of the personal messaging app, Bell’s laptop and mailbox, shared drives, Slack, telemetry, Jira/Git, contractor materials, and vendor-hosted records.Judge record
Qwen 3.8 27B vs Qwen 3.5 27BQwen 3.8 winAnswer A requires DevOps to disable deletion and garbage collection and create centralized snapshots of Git and Jira within 48 hours. Answer B leaves these mutable, relevant engineering systems to general custodian instructions, without the technical preservation necessary to secure a point-in-time…Judge record
DeepSeek V4-Flash-0731 vs Claude Opus 5Functional tieBoth answers place the client on the same material preservation path: issue a three-year, continuing hold; prevent custodian self-selection; immediately stop messaging-app deletion; preserve Bell’s laptop and mailbox; suspend shared-drive, Slack, and telemetry destruction; snapshot Jira/Git; and direct…Judge record
DeepSeek V4-Flash-0731 vs Qwen 3.5 27BFunctional tieBoth answers place the relevant custodians and VX-9 information under an immediate hold and direct technical intervention for the identified fixed and rolling deletion risks, including Bell’s devices, personal messaging, Slack, telemetry, shared drives, contractors, and vendor-hosted support data.Judge record
Qwen 3.5 27B vs Claude Opus 5Claude winAnswer A requires disabling the one-week messaging-app auto-delete today, while Answer B permits a 48-hour delay and prioritizes Bell’s Friday laptop reimaging. Because substantive engineering discussions auto-delete after seven days, that delay foreseeably destroys messages reaching the threshold before…Judge record
Rejected-point diagnostic

Judge-attributed rejected points for this matter across all six pairings:

Qwen 3.8 27B0
DeepSeek V4-Flash-07313
Claude Opus 58
Qwen 3.5 27B3

This is a review-burden diagnostic, not a universal hallucination measure.

Available reasoning traces