Models
Tested configurations
A configuration is model + effort. The table lists every published run of each configuration with its score.
Task set v2
| Model · effort | Runs and score | Citations (all runs) |
|---|---|---|
| Claude Fable 5.1 · default | 3 November 2026 · Run 6: 59 (8.8 of 15 tasks) (32,000 tokens) | 8690730 |
| Claude Fable 5.1 · high | 3 November 2026 · Run 7: 56 (8.3 of 15 tasks) (64,000 tokens) | 98150823 |
| Claude Haiku 5.5 · default | 3 November 2026 · Run 6: 50 (7.5 of 15 tasks) (32,000 tokens) | 7480634 |
| Claude Haiku 5.5 · high | 3 November 2026 · Run 7: 41 (6.1 of 15 tasks) (64,000 tokens) | 83140438 |
| Claude Opus 5.5 · high | 3 November 2026 · Run 5: 78 (11.7 of 15 tasks) (32,000 tokens) | 9230742 |
| Claude Opus 5.5 · medium | 3 November 2026 · Run 7: 79 (11.8 of 15 tasks) (64,000 tokens) | 8420737 |
| Claude Opus 5.5 · xhigh | 3 November 2026 · Run 7: 77 (11.5 of 15 tasks) (64,000 tokens) | 10030535 |
| Claude Sonnet 5.5 · high | 3 November 2026 · Run 6: 69 (10.4 of 15 tasks) (32,000 tokens) | 6250937 |
| Claude Sonnet 5.5 · low | 3 November 2026 · Run 7: 67 (10.1 of 15 tasks) (64,000 tokens) | 7150632 |
| Claude Sonnet 5.5 · medium | 3 November 2026 · Run 6: 68 (10.2 of 15 tasks) (32,000 tokens) | 6240645 |
| Command A Plus · default | 3 November 2026 · Run 6: 0 (0 of 15 tasks) (32,000 tokens) | 1641868 |
| DeepSeek V4 Flash · defaultlocal model (no API cost) · 16k context, not comparable | 4 November 2026 · Run 4: 12 (1.7 of 15 tasks) (14,500 tokens) | 2814129 |
| DeepSeek V4 Flash 0731 · default | 3 November 2026 · Run 8: 18 (2.8 of 15 tasks) (64,000 tokens) | 2381717 |
| DeepSeek V4 Pro 0813 · default | 3 November 2026 · Run 6: 20 (3.1 of 15 tasks) (32,000 tokens) | 31112517 |
| DeepSeek V4.1 Flash · default | 3 November 2026 · Run 6: 31 (4.7 of 15 tasks) (32,000 tokens) | 32150629 |
| DeepSeek V4.1 Flash 1017 · default | 3 November 2026 · Run 8: 35 (5.3 of 15 tasks) (64,000 tokens) | 34130629 |
| Gemini 3.8 Flash · default | 3 November 2026 · Run 8: 45 (6.7 of 15 tasks) (64,000 tokens) | 4380225 |
| Gemini 4 Argon · default | 3 November 2026 · Run 5: 55 (8.2 of 15 tasks) (32,000 tokens) | 4730726 |
| Gemma 4 26B A4B (QAT) · defaultlocal model (no API cost) | 4 November 2026 · Run 2: 3 (0.5 of 15 tasks) (62,000 tokens) | 1923337 |
| GLM 5.3 · default | 3 November 2026 · Run 8: 9 (1.3 of 15 tasks) (64,000 tokens) | 41391518 |
| GLM 5.3 Flash · default | 3 November 2026 · Run 8: 0 (0.0 of 15 tasks) (64,000 tokens) | 31510223 |
| GLM-5.3 FlashX · default | 3 November 2026 · Run 6: 3 (0.5 of 15 tasks) (32,000 tokens) | 416311231 |
| GLM-5.3 FlashX Legal · default | 3 November 2026 · Run 6: 8 (1.1 of 15 tasks) (32,000 tokens) | 21251332 |
| GLM-5.3 Prime · default | 3 November 2026 · Run 6: 4 (0.7 of 15 tasks) (32,000 tokens) | 16142211 |
| GPT-6 Astra · high | 3 November 2026 · Run 6: 72 (10.8 of 15 tasks) | 3901513 |
| GPT-6 Astra · xhigh | 3 November 2026 · Run 7: 68 (10.1 of 15 tasks) | 4120413 |
| GPT-6 Luna · high | 3 November 2026 · Run 7: 56 (8.4 of 15 tasks) | 2731513 |
| GPT-6 Luna · low | 3 November 2026 · Run 7: 25 (3.7 of 15 tasks) | 22140411 |
| GPT-6 Luna · medium | 3 November 2026 · Run 6: 34 (5.1 of 15 tasks) | 18110510 |
| GPT-6.1 Sol · high | 3 November 2026 · Run 5: 71 (10.6 of 15 tasks) | 4002512 |
| GPT-6.1 Sol · medium | 3 November 2026 · Run 7: 67 (10.0 of 15 tasks) | 3922613 |
| GPT-6.1 Sol · xhigh | 3 November 2026 · Run 6: 68 (10.2 of 15 tasks) | 3912513 |
| GPT-6.1 Terra · high | 3 November 2026 · Run 6: 55 (8.3 of 15 tasks) | 3071617 |
| GPT-6.1 Terra · xhigh | 3 November 2026 · Run 7: 45 (6.8 of 15 tasks) | 29101414 |
| Grok 4.7 · default | 3 November 2026 · Run 5: 49 (7.4 of 15 tasks) (32,000 tokens) | 53120325 |
| Kimi K3.1 Pro · default | 3 November 2026 · Run 6: 34 (5.1 of 15 tasks) (32,000 tokens) | 38173517 |
| Minimax M3 · default | 3 November 2026 · Run 8: 1 (0.2 of 15 tasks) (64,000 tokens) | 30340512 |
| Minimax M3.1 Flash · default | 3 November 2026 · Run 8: 0 (0 of 15 tasks) (64,000 tokens) | 36761833 |
| Mistral Large 4.0 · default | 3 November 2026 · Run 6: 10 (1.5 of 15 tasks) (32,000 tokens) | 50320528 |
| Ornith 1.5 35B A3B · defaultlocal model (no API cost) | 4 November 2026 · Run 3: 1 (0.1 of 15 tasks) (62,000 tokens) | 1542929 |
| Qwen3.8 Flash Next · defaultlocal model (no API cost) · 12k context, not comparable | 4 November 2026 · Run 1: 11 (1.6 of 15 tasks) (10,500 tokens) | 30240315 |
| Qwen3.8 Max Prime · default | 3 November 2026 · Run 6: 13 (1.9 of 15 tasks) (32,000 tokens) | 42262634 |
Swipe sideways to see all columns.
Task set v1 (archived)
Task set v1 (archived). Task set v2 is harder and uses criteria; its scores are not comparable with v1.
| Model · effort | Runs and score | Citations (all runs) |
|---|---|---|
| Claude Fable 5.1 · default tool-free not proven | 3 November 2026 · Run 2: 100 (4 of 4 tasks) | 563003 |
| Claude Haiku 5.5 · default tool-free not proven | 3 November 2026 · Run 2: 100 (4 of 4 tasks) | 420002 |
| Claude Opus 5.5 · high tool-free not proven | 3 November 2026 · Run 2: 100 (4 of 4 tasks) | 561014 |
| Claude Sonnet 5.5 · high | 3 November 2026 · Run 3: 100 (4 of 4 tasks) (16,000 tokens) | 440000 |
| Claude Sonnet 5.5 · medium tool-free not proven | 3 November 2026 · Run 2: 100 (4 of 4 tasks) | 421003 |
| Command A Plus · default | 3 November 2026 · Run 4: 63 (2.5 of 4 tasks) (32,000 tokens) | 83000 |
| DeepSeek V4 Pro 0813 · default | 3 November 2026 · Run 4: 88 (3.5 of 4 tasks) (32,000 tokens) | 230000 |
| DeepSeek V4.1 Flash · default | 3 November 2026 · Run 4: 88 (3.5 of 4 tasks) (32,000 tokens) | 344010 |
| Gemini 4 Argon · default | 3 November 2026 · Run 4: 100 (4 of 4 tasks) (32,000 tokens) 3 November 2026 · Run 1: 100 (4 of 4 tasks) (16,000 tokens) | 9350210 |
| GLM-5.3 FlashX · default | 3 November 2026 · Run 1: 63 (2.5 of 4 tasks) (16,000 tokens) | 328503 |
| GLM-5.3 FlashX Legal · default | 3 November 2026 · Run 4: 63 (2.5 of 4 tasks) (32,000 tokens) 3 November 2026 · Run 3: 63 (2.5 of 4 tasks) (40,000 tokens) 3 November 2026 · Run 1: 50 (2 of 4 tasks) (16,000 tokens) 3 November 2026 · Run 1: 75 (1.5 of 2 tasks) (40,000 tokens) | 576230 |
| GLM-5.3 Prime · default | 3 November 2026 · Run 4: 63 (2.5 of 4 tasks) (32,000 tokens) | 182000 |
| GPT-6 Astra · high tool-free not proven | 3 November 2026 · Run 2: 100 (4 of 4 tasks) | 250002 |
| GPT-6 Luna · medium | 3 November 2026 · Run 2: 100 (4 of 4 tasks) | 220001 |
| GPT-6.1 Sol · high | 3 November 2026 · Run 3: 100 (4 of 4 tasks) 3 November 2026 · Run 2: 100 (4 of 4 tasks) | 520002 |
| GPT-6.1 Sol · xhigh 1 void (tool use) tool-free not proven | 3 November 2026 · Run 2: 75 (3 of 4 tasks) | 270002 |
| GPT-6.1 Terra · high | 3 November 2026 · Run 2: 100 (4 of 4 tasks) | 210001 |
| Grok 4.7 · default | 3 November 2026 · Run 4: 100 (4 of 4 tasks) (32,000 tokens) | 300000 |
| Kimi K3.1 Pro · default | 3 November 2026 · Run 4: 88 (3.5 of 4 tasks) (32,000 tokens) | 282000 |
| Mistral Large 4.0 · default | 3 November 2026 · Run 4: 75 (3 of 4 tasks) (32,000 tokens) | 324100 |
| Qwen3.8 Max Prime · default | 3 November 2026 · Run 4: 88 (3.5 of 4 tasks) (32,000 tokens) | 493020 |
Swipe sideways to see all columns.
Citations: nrealnmisattributedninventednnot findablennot checked automatically