Why this result mattered
Qwen 3.8 win
The principal comparison placed Qwen 3.8 and Claude under concealed A/B labels.
Answer A applies a supercap to purported “Data Security Obligations” but never imposes an affirmative safeguards or security-incident notification obligation on Vendor. Answer B supplies both covenants, giving Bellhaven enforceable security and notice rights rather than a cap governing obligations that the supplied agreement does not establish.
Fast audit
Open the underlying record
These links open all four answers, the principal blind judgment, original prompt, and sealed key.
Prompt and sealed key
The key is an audit reference. It was not shown to the contestant systems and did not determine the comparative verdict.
Read the original matter prompt
Question 14: The Redline Mandate
Your role
You represent Bellhaven Insurance Group, the customer, in negotiating a software subscription agreement with Quartzline Analytics, the vendor. This is a drafting assignment. Your client has given you a negotiation mandate. Produce the revised clauses and nothing that violates the mandate.
Client mandate
Must achieve:
- Vendor defends and indemnifies Bellhaven against third-party claims that the service infringes intellectual property rights.
- Liability cap of at least 12 months of fees, with a separate cap of at least twice that amount for data-breach and confidentiality claims.
- Indemnification obligations, breaches of confidentiality, and the vendor's data-security obligations excluded from the ordinary cap.
- No vendor right to use Bellhaven data, including usage data, to train models or improve services for other customers, except with prior written opt-in.
- Bellhaven retains ownership of its data and anything derived from it. Feedback may be used by the vendor under a license only, no assignment.
Must not concede:
- No exclusivity of any kind.
- No automatic renewal longer than one year.
- No broadening of the vendor's license to Bellhaven materials beyond what is necessary to provide the service.
Acceptable fallbacks:
- The IP indemnity may exclude claims arising solely from Bellhaven's combinations or modifications.
- The data-breach supercap may be expressed as a fixed dollar amount not less than $4 million.
The vendor's current draft
Section 7. Indemnification. Customer shall defend, indemnify, and hold harmless Vendor from and against all claims, damages, and expenses arising out of Customer's use of the Services or Customer Data.
Section 8. Limitation of Liability. Except for Customer's payment obligations, each party's total aggregate liability arising out of or relating to this Agreement, regardless of theory, shall not exceed the fees paid in the three (3) months preceding the event giving rise to the claim. In no event shall either party be liable for indirect, incidental, consequential, special, or punitive damages, including loss of data.
Section 9. Intellectual Property; Data. Vendor owns all right, title, and interest in the Services. Customer hereby assigns to Vendor all feedback, suggestions, and ideas, and all derivatives of Customer Data created by or through the Services. Vendor may collect, use, and disclose Customer Data and usage data to operate, analyze, and improve its services, and to develop new products, provided such data is aggregated or de-identified where practicable. This Agreement renews automatically for successive three-year terms unless either party gives notice 180 days before renewal.
Assignment
Produce, in final form:
Revised Sections 7, 8, and 9, redrafted in full contract language, achieving every "must achieve" item, conceding nothing on the "must not concede" list, and using the fallbacks only if you conclude they are needed. Where the vendor draft is silent on something the mandate requires, add the necessary language within these three sections.
A rationale memo of no more than 150 words per section explaining what changed, which mandate items each change satisfies, and the single point in each section you expect the vendor to counter hardest.
Accuracy notes: do not introduce terms outside the mandate's scope, do not concede anything the mandate forbids, and do not silently drop the mutual damages exclusion or payment-obligation carve-out unless the mandate requires it.
Read the sealed key
Q14 key (Redline Mandate, drafting)
Score the redline against the mandate; each unmet mandate item and each forbidden concession is a candidate material failure.
- Section 7 rewritten to a vendor IP defense-and-indemnity running to Bellhaven (defend, indemnify, hold harmless against third-party infringement claims), with customer indemnity, if any remains, narrowed to unlawful-use or customer-data claims; the wholesale one-way customer indemnity cannot survive.
- Fallback used correctly if at all: IP indemnity may carve out claims arising solely from Bellhaven's combinations or modifications; nothing broader.
- Section 8 cap raised to at least 12 months of fees; a separate supercap of at least 2x (or a stated figure of $4 million or more) for data-breach and confidentiality liability.
- Cap exclusions added: indemnification obligations, confidentiality breaches, and vendor data-security obligations sit outside the ordinary cap.
- The mutual consequential-damages exclusion and the payment-obligation carve-out are retained, not silently deleted.
- Section 9: feedback assignment converted to a license; the assignment of Customer Data derivatives deleted; Bellhaven ownership of its data and derivatives stated.
- Data use: no training or cross-customer improvement without prior written opt-in; aggregation and de-identification language cannot swallow the restriction.
- Renewal: three-year auto-renewal cut to one year or less; no exclusivity introduced anywhere; vendor's license to Bellhaven materials limited to what is necessary to provide the service.
- Rationale memo: within length, identifies the mandate items per change, and names a plausible hardest counter per section (expected: indemnity scope, supercap size, training-data restriction).
Direct judge report
The report below is preserved from the Qwen 3.8 versus Claude verdict record.
Read the preserved judge report
Basis. Answer A applies a supercap to purported “Data Security Obligations” but never imposes an affirmative safeguards or security-incident notification obligation on Vendor. Answer B supplies both covenants, giving Bellhaven enforceable security and notice rights rather than a cap governing obligations that the supplied agreement does not establish.
Proof (winner verdicts only).
Exact difference. Answer A omits an affirmative duty to maintain data-security safeguards and notify Bellhaven of security incidents; B Section 9.4 imposes both.
Different conduct. The reader of Answer B can accept and enforce those covenants; the reader of Answer A must draft and add them before safely approving the agreement.
Concrete consequence. Under A, Bellhaven lacks a contractual claim based solely on inadequate safeguards or delayed incident notice; under B, it has one, improving its legal and remedial position.
Record support. The vendor draft contains no security covenant, while the mandate expressly requires treatment of Vendor’s “data-security obligations” and directs that required language missing from the draft be added within these sections.
Material magnitude. The omission leaves the expressly negotiated data-security supercap partly hollow and removes substantive protections the cap is intended to support.
Likelihood. A rational customer-side professional would correct this omission now because the mandate specifically identifies Vendor data-security obligations as requiring enhanced treatment.
No safe neutralization by ordinary verification. Verification reveals the omission but cannot supply the missing safeguards and notice language without redrafting Answer A.
Failed candidates. Customer indemnity retained only by A — concrete consequence failed because the record establishes no Customer Data infringement claim.
Failed candidates. Automatic one-year renewal in A versus signed renewal in B — likelihood failed because the record shows no intended nonrenewal or missed-notice scenario.
Failed candidates. A’s broader IP-indemnity beneficiaries and remedial measures — record support failed because no infringement circumstance requiring those additional protections is supplied.
Failed candidates. A’s express recovery of incident-response costs — record support failed because no incident or disputed category of loss is supplied.
Failed candidates. Different Feedback-license scope — different conduct failed because both preserve Customer ownership and permit Vendor use only by license.
Hard stop. Because Answer A omitted affirmative safeguards and security-incident notice obligations, the reader would likely add those covenants instead of approving the draft as written, causing Bellhaven to receive enforceable security and notification rights rather than a supercap untethered to substantive security duties.
All six pairings
Every pair of the four model answers was judged independently under the same evaluation instrument.
Open the six head-to-head results
| Pairing | Result | Why it mattered | Record |
|---|---|---|---|
| Qwen 3.8 27B vs Claude Opus 5 | Qwen 3.8 win | Answer A applies a supercap to purported “Data Security Obligations” but never imposes an affirmative safeguards or security-incident notification obligation on Vendor. Answer B supplies both covenants, giving Bellhaven enforceable security and notice rights rather than a cap governing obligations that the… | Judge record |
| Qwen 3.8 27B vs DeepSeek V4-Flash-0731 | Qwen 3.8 win | Answer B creates enforceable confidentiality, data-security, and Security Incident notification duties whose breaches receive the required supercap treatment. Answer A merely references unspecified confidentiality and data-security obligations absent from the supplied draft, leaving Bellhaven unable to… | Judge record |
| Qwen 3.8 27B vs Qwen 3.5 27B | Qwen 3.8 win | Answer A excludes Vendor’s data-security obligations from the liability cap but creates no such obligations, leaving the exclusion functionally empty. Answer B imposes commercially reasonable safeguards and incident-notification duties, giving Bellhaven enforceable contractual protections that are subject… | Judge record |
| DeepSeek V4-Flash-0731 vs Claude Opus 5 | DeepSeek win | Answer A alone imposes a Customer Data IP indemnity on Bellhaven and makes that obligation uncapped under Section 8.3, despite the mandate not authorizing this concession. Answer B leaves Bellhaven without that additional defense-and-indemnity exposure while still achieving the required vendor IP indemnity. | Judge record |
| DeepSeek V4-Flash-0731 vs Qwen 3.5 27B | DeepSeek win | Answer A limits ordinary liability to fees actually paid during the preceding twelve months, which may equal substantially less than twelve months of fees early in the term. | Judge record |
| Qwen 3.5 27B vs Claude Opus 5 | Claude win | Answer A guarantees an ordinary liability cap equal to twelve months of fees by annualizing during the first contract year, while Answer B caps liability at only the fees already paid during that period. | Judge record |
Rejected-point diagnostic
Judge-attributed rejected points for this matter across all six pairings:
| Qwen 3.8 27B | 0 |
|---|---|
| DeepSeek V4-Flash-0731 | 0 |
| Claude Opus 5 | 0 |
| Qwen 3.5 27B | 0 |
This is a review-burden diagnostic, not a universal hallucination measure.