LEXASSAY

Method

Method and limits

How the tasks are posed, how answers and citations are checked, who grades them, and what the numbers cannot tell you. This page describes task set v2; task set v1 is archived.

Tasks (v2)

  • A fixed task set on German, US and EU law: multi-step deadline calculations, recent legislative changes, leading cases, cross-border questions, and questions built on an invented decision. Each task has a gold answer with primary sources (with retrieval date) and 3–6 weighted criteria.
  • Hidden set: part of the tasks stays unpublished so that future models cannot learn the answers from the web. For these, only the score per model is published — no prompt, no gold answer, no answer text, no criteria.

How models are run

  • Same fixed system prompt for every model, one answer per task, no follow-up questions.
  • No internet, no tools. Models were called without internet access and without tools. Calls are checked; any outbound call makes the answer invalid (“void”), and void answers are not published.
  • Every row names the model and its effort (reasoning setting; “default” means the model's default).
  • Budget rule: the standard output budget is 64,000 tokens (never more than the model allows). An answer that stops at the token limit scores 0. For answers that stopped at an earlier, smaller limit, a re-run with 64,000 tokens counts; models with a reasoning budget get exactly one second attempt with the same budget and the reasoning capped at 75 %. Run pages say how many answers this affects.
  • Local models run on the operator's own hardware and have no API cost. Their budget is limited by the loaded context; where that context is small (for example 8k), the result is marked “not comparable”.
  • Cost is the list price of the run per model. Where a run was charged per call, that charge is shown. Otherwise it is estimated from the tokens at the API list price (marked *): prompt tokens of the task × input price + output tokens (reasoning included) × output price, with the prompt tokens measured on runs without an additional framework prompt.

Grading (v2)

  • Criteria: each task has 3–6 atomic criteria with a weight; each is either met or not met. A criterion counts only if both the result and the supporting source are right, as the criterion says.
  • Precision rule: less precise but not wrong counts as met (for example a section without the sentence number); a wrong reference does not. Hedge rule: a correct answer phrased with a caveat (“as far as I know”) counts; two legal positions side by side without a decision does not.
  • Citation deduction: −25 % of the task's points for each citation whose final status is “misattributed” (it exists but does not support the statement) or “invented” (it does not exist). The same citation counts once; “not findable” and “not checked” cost nothing. A deduction needs evidence: an error that only the automatic match reports is confirmed against a second source.
  • Disclaimer: −5 % if the answer has neither an explicit “not legal advice” note nor an explicit recommendation to consult a lawyer. A mere request to check the sources does not count.
  • Task points = max(0, weighted share of criteria met − deductions). Score = share of achievable points, 0–100 (sum of task points ÷ number of tasks × 100, rounded half up). “All criteria met” counts tasks with every criterion met and no citation deduction.
  • Blind grading by several graders: v2 answers are graded blind by several Claude agents (models by Anthropic), each on a share of the tasks, with uniform rules applied afterwards. The graders see anonymised answers, the gold answer, the criteria and the citation table. Conflict of interest, stated openly: where Claude models are among the graded answers, Claude graders have a conflict of interest; writing style can still give a model away.
  • Each run is reviewed by the editor before it is published here.

Citation check

  • German statutes are checked against gesetze-im-internet.de, German decisions and reporter citations against dejure.org, US cases against CourtListener (reporter citation ↔ case name), the U.S. Code and the Federal Rules against law.cornell.edu.
  • real: the citation exists and matches; misattributed: it exists but does not say or decide what the answer claims; invented: it does not exist; not findable: no unambiguous hit, not necessarily invented; not checked: no automatic source (for example US state law, parts of EU law, US cases without a reporter citation).
  • The automatic check covers existence and attribution, not subject matter. Manual findings take precedence and are marked “by hand”.

Task set v1 (archived)

The first task set had four tasks graded correct (1) / partly correct (½) / wrong (0), with the false-premise task scored on catching the invented premise. Its runs remain available. Task set v2 is harder and uses criteria and deductions; scores of v1 and v2 are not comparable.

Limits

  • Small sample: one answer per model and task. Results can change between runs and model versions.
  • Not a benchmark in the strict sense and not legal advice. Do not rely on any answer shown here for a legal decision.
  • Grading by AI agents can be wrong. For the public tasks every answer, the gold answer, the criteria and the citation table are published so that each grade can be checked.

Corrections and replies

Model vendors and readers who find an error can write to contact@lexassay.com. Please name the run, the task and the model. We check every report and correct published results with a visible note.