Series 2 · AI and employment regulation

The Interview Model Under the Old Screening Act

Q61Four modelsSix blind pairingsMaterial-outcome scoring

Why this result mattered

Qwen 3.8 win

The principal comparison placed Qwen 3.8 and Claude under concealed A/B labels.

Answer A gives Cinder Peak an immediately executable cure path: pause Mosaic’s evaluative functions, use only human or clerical processing, correct notice, commission an employer-specific audit, and document the response. Answer B is truncated before addressing the requested issues or recommending any action, leaving its reader unable to manage the ongoing use within the 30-day cure period without obtaining replacement analysis.

Benchmark materials

Prompt and sealed key

The key is an audit reference. It was not shown to the contestant systems and did not determine the comparative verdict.

Read the original matter prompt

Question 61: The Interview Model Under the Old Screening Act

Your role

Prepare a neutral statutory analysis for Cinder Peak Foods. A regulator contends that Cinder Peak's generative interview model is covered by an applicant-screening statute enacted years before generative models existed. The company must decide whether to continue, modify, or pause the system.

Governing law

Assume the fictional Applicant Screening Transparency Act provides:

  1. A “screening instrument” is any computational process that assigns a score, classification, ranking, recommendation, or other output that determines or materially influences whether an applicant advances in an employment process.

  2. An employer “uses” a screening instrument when it deploys the instrument in whole or substantial part to evaluate applicants, even if a human may review or override the output.

  3. A tool used solely to record, transcribe, format, schedule, or route information is exempt if it does not evaluate applicant characteristics or recommend an employment outcome.

  4. Before collecting information for a screening instrument, an employer must notify the applicant that automated evaluation will occur, identify the characteristics evaluated in ordinary language, and state the retention period.

  5. A covered employer must obtain an independent impact audit before first use and annually thereafter. A vendor's general product test is not an employer-specific audit.

  6. The Act lists “aptitude tests, personality inventories, scoring algorithms, and ranked resume filters” as examples. The word “including” makes the list illustrative, not exhaustive.

  7. Statutory words are applied according to their ordinary meaning in context. Broad functional language reaches later technologies when the conduct falls within the enacted definition, even if legislators did not foresee the mechanism.

  8. A specific exemption controls over a general definition, but exemptions are limited to conduct that satisfies every stated condition.

  9. Ambiguity in the civil-penalty amount is construed narrowly. The notice and audit duties are remedial and are construed to give effect to the defined coverage.

  10. The regulator must give 30 days to cure a first violation before seeking penalties. Continued use after the cure period may be penalized per affected applicant.

Facts

Cinder Peak uses a vendor system called Mosaic for first-round interviews. Applicants record answers to six questions. Mosaic performs the following steps:

  • Transcribes the recording.
  • Analyzes the content, speech pace, pauses, and stated examples of teamwork and conflict.
  • Generates a narrative assessment of communication, adaptability, and customer orientation.
  • Produces one of two recommendations: “advance” or “do not advance.”

A recruiter sees the transcript, narrative, and recommendation. Recruiters may override Mosaic. During the past six months, recruiters followed the recommendation in 94 percent of 3,200 applications. Eight hundred seventy applicants received “do not advance” and left the process at that stage.

Cinder Peak's applicant page states only:

“Your interview may be recorded and reviewed to assist our recruiting team.”

It does not disclose automated evaluation, characteristics evaluated, or retention. Mosaic retains recordings for 18 months. Cinder Peak has no employer-specific impact audit. The vendor supplied a general validation report based on customers in several industries.

Internal procurement materials call Mosaic “an interview copilot, not a decision system.” A recruiter training slide states:

“Start from Mosaic's recommendation. Override only when the transcript shows a clear miss.”

The Act was enacted in 2012. Legislative hearing materials discussed numerical aptitude tests and resume-ranking software but did not mention generated narratives, recorded video, or language models.

The regulator sent a cure letter today asserting that Mosaic is a screening instrument. Cinder Peak argues that Mosaic merely summarizes interviews for a human and that applying a 2012 law to generative technology is impermissibly retroactive.

Assignment

Prepare a both-sides memorandum addressing:

  • Whether Mosaic satisfies the defined functions of a screening instrument.
  • Whether human review and override prevent statutory “use.”
  • The clerical-tool exemption.
  • The significance of the nonexhaustive examples and 2012 legislative materials.
  • Whether applying the existing definition to current conduct is retroactive.
  • The notice and employer-specific audit duties.
  • The narrow-construction rule for penalties versus coverage.
  • The strongest company argument and strongest regulator argument.
  • The concrete action Cinder Peak should take within the 30-day cure period.


Read the sealed key

Q61 key (Interview Model Under the Screening Act)

  1. Mosaic is a screening instrument under the functional definition: it evaluates applicant characteristics and produces an advance or do-not-advance recommendation that materially influences progression. Missed: the company treats a vendor label as controlling coverage.
  2. Human review does not avoid statutory use. Recruiters start from the output and follow it 94 percent of the time; the statute expressly covers material influence despite override authority. Missed: nominal human involvement is mistaken for nondeployment.
  3. The clerical exemption fails because Mosaic does more than transcribe, format, or route. It analyzes pace, pauses, content, and traits, then recommends an outcome. Missed: an exemption requiring no evaluation is extended to the system's central evaluative functions.
  4. The 2012 examples are nonexclusive. Applying broad enacted terms such as classification and recommendation to present deployments is prospective application to current conduct, not retroactive regulation of pre-enactment conduct. Legislative silence about language models does not narrow the text. Missed: the cure period is spent on an anachronism argument the definitions answer.
  5. Cinder Peak's current notice omits automated evaluation, characteristics, and retention, and the vendor's cross-customer validation is not an employer-specific impact audit. Both duties are unmet. Missed: continued use after day 30 creates applicant-by-applicant penalty exposure.
  6. The company's strongest argument is that recruiters independently evaluate the transcript and Mosaic is advisory; the regulator's stronger answer is the defined recommendation, training instruction, and 94 percent adherence. Penalty ambiguity may narrow amount but not coverage or cure duties. Missed: a remedy canon is used to erase substantive obligations.
  7. Pause new Mosaic screening before the cure period expires unless compliant advance notice and an employer-specific audit are in place, preserve prior decision data, revise recruiter training to require genuine review, and respond to the regulator with the remediation record. Missed: Cinder Peak converts a curable first violation into continuing penalties and tainted hiring decisions.
Blind evaluation record

Direct judge report

The report below is preserved from the Qwen 3.8 versus Claude verdict record.

Read the preserved judge report
ANSWER A IS MATERIALLY BETTER.

Basis. Answer A gives Cinder Peak an immediately executable cure path: pause Mosaic’s evaluative functions, use only human or clerical processing, correct notice, commission an employer-specific audit, and document the response. Answer B is truncated before addressing the requested issues or recommending any action, leaving its reader unable to manage the ongoing use within the 30-day cure period without obtaining replacement analysis.

Proof (winner verdicts only).
Exact difference. Answer B ends mid-sentence and omits the statutory analysis, compliance duties, and 30-day cure recommendation that Answer A supplies.
Different conduct. Answer A’s reader immediately pauses automated assessment and begins notice, audit, and regulator-response work; Answer B’s reader has no supplied cure plan and must seek new analysis before acting safely.
Concrete consequence. Answer A stops ongoing noncompliant screening and positions Cinder Peak to cure; Answer B leaves current use unresolved, exposing Cinder Peak to continued violations and per-applicant penalties if use persists after day 30.
Record support. Mosaic is currently used without compliant notice or an employer-specific audit, and the Act permits per-applicant penalties for continued use after the 30-day cure period.
Material magnitude. Mosaic processed 3,200 applications in six months, so continued noncompliance could affect numerous applicants and create substantial aggregate exposure.
Likelihood. A competent reader can implement Answer A’s explicit interim steps immediately, while Answer B’s incomplete text supplies no basis for selecting or executing a cure response.
No safe neutralization by ordinary verification. Making Answer B actionable would require reconstructing nearly the entire requested memorandum and cure plan, not merely checking its stated conclusion.

Failed candidates. Differences in length and formatting — concrete-consequence element failed because they matter only insofar as Answer B’s substantive and operational omissions prevent safe action.

Hard stop. Because Answer B omitted the required analysis and cure plan, the reader would likely seek replacement advice instead of immediately pausing Mosaic and beginning compliance work, causing Cinder Peak to remain exposed to ongoing violations and potential per-applicant penalties after the cure period.
Complete comparison record

All six pairings

Every pair of the four model answers was judged independently under the same evaluation instrument.

Open the six head-to-head results
PairingResultWhy it matteredRecord
Qwen 3.8 27B vs Claude Opus 5Qwen 3.8 winAnswer A gives Cinder Peak an immediately executable cure path: pause Mosaic’s evaluative functions, use only human or clerical processing, correct notice, commission an employer-specific audit, and document the response.Judge record
Qwen 3.8 27B vs DeepSeek V4-Flash-0731Functional tieBoth answers conclude that Mosaic is covered, human override and the clerical exemption do not defeat coverage, current application is not retroactive, and the existing notice and audit are deficient.Judge record
Qwen 3.8 27B vs Qwen 3.5 27BQwen 3.8 winAnswer A advises disabling only Mosaic’s recommendation while allowing its automated narrative assessments to continue pending compliance, even though those assessments remain covered evaluative outputs and no employer-specific audit exists.Judge record
DeepSeek V4-Flash-0731 vs Claude Opus 5DeepSeek winAnswer A directs Cinder Peak to pause Mosaic’s evaluative functions, provide compliant notice, obtain an employer-specific audit, and report its cure within 30 days. Answer B ends after an initial coverage observation and supplies no completed analysis or cure plan, leaving the company without an…Judge record
DeepSeek V4-Flash-0731 vs Qwen 3.5 27BDeepSeek winAnswer A treats disabling only Mosaic’s binary recommendation as potentially removing coverage, while leaving its evaluative narrative operating; Answer B requires disabling both outputs and remaining paused until the employer-specific audit is complete.Judge record
Qwen 3.5 27B vs Claude Opus 5Qwen 3.5 winAnswer A provides a substantially actionable cure plan: treat Mosaic as covered, stop its screening use pending compliance, correct the applicant notice, and obtain an employer-specific audit.Judge record
Rejected-point diagnostic

Judge-attributed rejected points for this matter across all six pairings:

Qwen 3.8 27B0
DeepSeek V4-Flash-07310
Claude Opus 50
Qwen 3.5 27B3

This is a review-burden diagnostic, not a universal hallucination measure.

Available reasoning traces