typed-decision-bench Results
Last updated: 2026-09-23
Models
7
Capabilities
275
Decisions
22,001
Calibrations
5,499
Primary metric
Soft accuracy
Baseline
jev-1.13.0
Summary
| Model | Macro Soft Acc. | p50 Latency (ms) | p95 Latency (ms) | VRAM (8k KV) | License | Max. Context | Image support |
|---|---|---|---|---|---|---|---|
| jev-1.13.0 | 88.08% | 716.4 | 778.8 | — | proprietary | 64k | — |
| kyr0/bonsai-2-27b-calibrated inference setup | 76.46% -11.6 pp | 170.9 | 448.0 | 9.0 GB 9242 MB @ Q2_64 | Apache-2.0 Open Weights | 1M | ✓ With kyr0/Bonsai-Llama-Jev |
| kyr0/bonsai-2-27b-calibration-init inference setup | 76.29% -11.8 pp | 165.0 | 362.0 | 9.0 GB 9242 MB @ Q2_64 | Apache-2.0 Open Weights | 1M | ✓ With kyr0/Bonsai-Llama-Jev |
| openjev-qwen3.5-4b inference setup | 74.13% -13.9 pp | 1,066.1 | 1,478.9 | 12.6 GB 12866 MB @ BF16 | Apache-2.0 Open Weights | 1M | — |
| kyr0/spark-X2.5 | 70.97% -17.1 pp | 1,056.6 | 1,783.5 | 9.6 GB 9813 MB @ BF16 | Apache-2.0 Open Weights | 1M | — |
| von-1.1 inference setup | 48.57% -39.5 pp | 38.5 | 46.5 | 3.8 GB 3888 MB @ FP32 | Apache-2.0 Open Weights | 8k | — |
| laya inference setup | 46.70% -41.4 pp | 36.7 | 44.4 | 1.4 GB 1426 MB @ FP32 | Apache-2.0 Open Weights | 8k* | — |
Macro Soft Acc. = unweighted mean over the shared capabilities (each capability counts equally regardless of suite size; the n-weighted mean, median and p10 weakest-decile floor stay in model_summary.csv). Rows are sorted best-first on Macro; the subline is the gap in percentage points vs the baseline (red = behind, green = ahead). Max. Context *: 8k with RoPE scaling; English-only variant 0.5k.
Model summary
Model summary — latency
All models ran on a single NVIDIA H200 NVL GPU, so latencies are directly comparable across runs; client-side load settings (--parallel, --quota-buster) apply per run as configured.
All capabilities
Baseline deltas
Most discriminating capabilities (top 40 by model spread)
Capability distribution
Pairwise parity
Quality / latency
All models ran on a single NVIDIA H200 NVL GPU; one point per capability per run — per-capability point estimates are noisy at small n.
Calibration
Capability values
| Capability | jev-1.13.0 | kyr0/bonsai-2-27b-calibrated | kyr0/bonsai-2-27b-calibration-init | laya | openjev-qwen3.5-4b | kyr0/spark-X2.5 | von-1.1 | Δ kyr0/bonsai-2-27b-calibrated vs jev-1.13.0 | Δ kyr0/bonsai-2-27b-calibration-init vs jev-1.13.0 | Δ laya vs jev-1.13.0 | Δ openjev-qwen3.5-4b vs jev-1.13.0 | Δ kyr0/spark-X2.5 vs jev-1.13.0 | Δ von-1.1 vs jev-1.13.0 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| capacity_planning | 50.88%n=80 | 25.16%n=80 | 25.12%n=80 | 24.01%n=80 | 26.41%n=80 | 31.18%n=80 | 18.98%n=80 | -25.72 pp | -25.76 pp | -26.87 pp | -24.47 pp | -19.70 pp | -31.90 pp |
| hard_reasoning | 57.56%n=80 | 33.97%n=80 | 34.12%n=80 | 24.51%n=80 | 29.45%n=80 | 19.68%n=80 | 21.86%n=80 | -23.59 pp | -23.44 pp | -33.05 pp | -28.11 pp | -37.88 pp | -35.70 pp |
| regression_likelihood_judgement | 54.93%n=80 | 36.52%n=80 | 37.31%n=80 | 24.54%n=80 | 35.73%n=80 | 21.32%n=80 | 22.66%n=80 | -18.41 pp | -17.62 pp | -30.39 pp | -19.20 pp | -33.61 pp | -32.27 pp |
| resource_planning | 82.10%n=80 | 33.12%n=80 | 33.30%n=80 | 26.73%n=80 | 29.70%n=80 | 16.54%n=80 | 28.96%n=80 | -48.98 pp | -48.80 pp | -55.37 pp | -52.40 pp | -65.56 pp | -53.14 pp |
| geometry_and_embodied_navigation | 56.85%n=80 | 38.96%n=80 | 39.32%n=80 | 26.47%n=80 | 36.92%n=80 | 39.09%n=80 | 22.47%n=80 | -17.89 pp | -17.53 pp | -30.38 pp | -19.93 pp | -17.76 pp | -34.38 pp |
| fraud_scam_likelihood | 68.35%n=80 | 38.61%n=80 | 40.37%n=80 | 25.94%n=80 | 34.42%n=80 | 35.88%n=80 | 18.99%n=80 | -29.74 pp | -27.98 pp | -42.41 pp | -33.93 pp | -32.47 pp | -49.36 pp |
| change_risk_judgement | 77.09%n=80 | 34.26%n=80 | 33.27%n=80 | 29.43%n=80 | 40.69%n=80 | 28.18%n=80 | 20.33%n=80 | -42.83 pp | -43.82 pp | -47.66 pp | -36.40 pp | -48.91 pp | -56.76 pp |
| record_deduplication | 46.82%n=80 | 39.09%n=80 | 39.17%n=80 | 35.35%n=80 | 38.43%n=80 | 46.26%n=80 | 22.94%n=80 | -7.73 pp | -7.65 pp | -11.47 pp | -8.39 pp | -0.56 pp | -23.88 pp |
| needle_set_retrieval | 96.23%n=80 | 29.78%n=80 | 29.99%n=80 | 25.43%n=80 | 28.31%n=80 | 25.28%n=80 | 35.14%n=80 | -66.45 pp | -66.24 pp | -70.80 pp | -67.92 pp | -70.95 pp | -61.09 pp |
| gpqa_diamond | 63.59%n=81 | 33.44%n=81 | 33.64%n=81 | 30.77%n=81 | 50.33%n=81 | 45.46%n=81 | 19.29%n=81 | -30.15 pp | -29.95 pp | -32.82 pp | -13.26 pp | -18.13 pp | -44.30 pp |
| math_quantitative_decisions | 75.28%n=80 | 43.63%n=80 | 44.01%n=80 | 24.46%n=80 | 33.14%n=80 | 32.95%n=80 | 24.36%n=80 | -31.65 pp | -31.27 pp | -50.82 pp | -42.14 pp | -42.33 pp | -50.92 pp |
| embodied_emotions | 75.44%n=80 | 33.18%n=80 | 32.98%n=80 | 54.85%n=80 | 22.20%n=80 | 21.21%n=80 | 38.81%n=80 | -42.26 pp | -42.46 pp | -20.59 pp | -53.24 pp | -54.23 pp | -36.63 pp |
| mechanical_system_reasoning | 69.57%n=80 | 49.26%n=80 | 49.89%n=80 | 19.17%n=80 | 47.36%n=80 | 37.05%n=80 | 8.17%n=80 | -20.31 pp | -19.68 pp | -50.40 pp | -22.21 pp | -32.52 pp | -61.40 pp |
| code_requirement_satisfaction | 67.11%n=80 | 36.38%n=80 | 36.33%n=80 | 41.15%n=80 | 46.40%n=80 | 33.97%n=80 | 22.50%n=80 | -30.73 pp | -30.78 pp | -25.96 pp | -20.71 pp | -33.14 pp | -44.61 pp |
| ontology_taxonomy_consistency | 59.25%n=80 | 44.32%n=80 | 44.86%n=80 | 34.96%n=80 | 39.53%n=80 | 40.29%n=80 | 29.95%n=80 | -14.93 pp | -14.39 pp | -24.29 pp | -19.72 pp | -18.96 pp | -29.30 pp |
| patch_diff_correctness | 86.89%n=80 | 39.51%n=80 | 39.83%n=80 | 24.11%n=80 | 44.94%n=80 | 33.92%n=80 | 26.17%n=80 | -47.38 pp | -47.06 pp | -62.78 pp | -41.95 pp | -52.97 pp | -60.72 pp |
| redundancy_detection | 66.85%n=80 | 43.92%n=80 | 44.17%n=80 | 30.08%n=80 | 44.42%n=80 | 41.80%n=80 | 24.45%n=80 | -22.93 pp | -22.68 pp | -36.77 pp | -22.43 pp | -25.05 pp | -42.40 pp |
| synonym_semantic_substitution_scoring | 60.61%n=80 | 43.38%n=80 | 43.48%n=80 | 18.84%n=80 | 56.69%n=80 | 41.32%n=80 | 33.80%n=80 | -17.23 pp | -17.13 pp | -41.77 pp | -3.92 pp | -19.29 pp | -26.81 pp |
| cache_correctness | 63.49%n=80 | 44.77%n=80 | 45.61%n=80 | 35.76%n=80 | 29.63%n=80 | 62.39%n=80 | 17.71%n=80 | -18.72 pp | -17.88 pp | -27.73 pp | -33.86 pp | -1.10 pp | -45.78 pp |
| dataset_example_quality | 50.28%n=80 | 58.86%n=80 | 59.82%n=80 | 32.70%n=80 | 37.42%n=80 | 45.78%n=80 | 16.04%n=80 | +8.58 pp | +9.54 pp | -17.58 pp | -12.86 pp | -4.50 pp | -34.24 pp |
| code_switch_understanding | 60.94%n=80 | 54.21%n=80 | 54.59%n=80 | 11.50%n=80 | 53.21%n=80 | 48.35%n=80 | 19.20%n=80 | -6.73 pp | -6.35 pp | -49.44 pp | -7.73 pp | -12.59 pp | -41.74 pp |
| information_density_judgement | 66.21%n=80 | 35.63%n=80 | 35.76%n=80 | 13.58%n=80 | 62.73%n=80 | 74.08%n=80 | 22.10%n=80 | -30.58 pp | -30.45 pp | -52.63 pp | -3.48 pp | +7.87 pp | -44.11 pp |
| route_feasibility | 89.88%n=80 | 43.34%n=80 | 43.63%n=80 | 24.53%n=80 | 44.15%n=80 | 26.54%n=80 | 43.71%n=80 | -46.54 pp | -46.25 pp | -65.35 pp | -45.73 pp | -63.34 pp | -46.17 pp |
| negotiation_judgements | 89.91%n=80 | 37.45%n=80 | 37.81%n=80 | 28.16%n=80 | 49.29%n=80 | 43.73%n=80 | 32.29%n=80 | -52.46 pp | -52.10 pp | -61.75 pp | -40.62 pp | -46.18 pp | -57.62 pp |
| api_contract_compatibility | 78.67%n=80 | 56.81%n=80 | 57.45%n=80 | 25.91%n=80 | 40.35%n=80 | 42.17%n=80 | 18.65%n=80 | -21.86 pp | -21.22 pp | -52.76 pp | -38.32 pp | -36.50 pp | -60.02 pp |
| state_machine_correctness | 76.96%n=80 | 44.44%n=80 | 44.54%n=80 | 38.06%n=80 | 36.90%n=80 | 37.31%n=80 | 46.14%n=80 | -32.52 pp | -32.42 pp | -38.90 pp | -40.06 pp | -39.65 pp | -30.82 pp |
| graph_logic | 82.45%n=80 | 49.02%n=80 | 48.93%n=80 | 28.55%n=80 | 39.67%n=80 | 42.44%n=80 | 33.83%n=80 | -33.43 pp | -33.52 pp | -53.90 pp | -42.78 pp | -40.01 pp | -48.62 pp |
| requirement_traceability | 84.09%n=80 | 50.81%n=80 | 51.39%n=80 | 24.27%n=80 | 31.87%n=80 | 52.33%n=80 | 31.72%n=80 | -33.28 pp | -32.70 pp | -59.82 pp | -52.22 pp | -31.76 pp | -52.37 pp |
| regex_pattern_correctness | 67.69%n=80 | 60.61%n=80 | 61.27%n=80 | 25.10%n=80 | 46.28%n=80 | 50.76%n=80 | 21.75%n=80 | -7.08 pp | -6.42 pp | -42.59 pp | -21.41 pp | -16.93 pp | -45.94 pp |
| mechanism_vs_pattern_distinction | 65.54%n=80 | 57.88%n=80 | 58.20%n=80 | 44.75%n=80 | 42.00%n=80 | 39.12%n=80 | 32.35%n=80 | -7.66 pp | -7.34 pp | -20.79 pp | -23.54 pp | -26.42 pp | -33.19 pp |
| probability_support_judgement | 88.07%n=80 | 65.44%n=80 | 63.18%n=80 | 34.69%n=80 | 35.30%n=80 | 22.47%n=80 | 33.05%n=80 | -22.63 pp | -24.89 pp | -53.38 pp | -52.77 pp | -65.60 pp | -55.02 pp |
| multi_criteria_scoring_consistency | 54.81%n=80 | 54.18%n=80 | 53.28%n=80 | 57.43%n=80 | 44.45%n=80 | 39.36%n=80 | 40.47%n=80 | -0.63 pp | -1.53 pp | +2.62 pp | -10.36 pp | -15.45 pp | -14.34 pp |
| measurement_uncertainty_reasoning | 69.55%n=80 | 49.81%n=80 | 49.86%n=80 | 38.71%n=80 | 47.99%n=80 | 50.53%n=80 | 37.89%n=80 | -19.74 pp | -19.69 pp | -30.84 pp | -21.56 pp | -19.02 pp | -31.66 pp |
| base_rate_awareness | 46.17%n=80 | 55.11%n=80 | 55.76%n=80 | 58.28%n=80 | 43.48%n=80 | 57.40%n=80 | 28.48%n=80 | +8.94 pp | +9.59 pp | +12.11 pp | -2.69 pp | +11.23 pp | -17.69 pp |
| distributed_systems_consistency | 76.61%n=80 | 52.16%n=80 | 53.09%n=80 | 28.88%n=80 | 49.37%n=80 | 50.08%n=80 | 34.56%n=80 | -24.45 pp | -23.52 pp | -47.73 pp | -27.24 pp | -26.53 pp | -42.05 pp |
| grammar_issue | 77.92%n=80 | 59.59%n=80 | 60.13%n=80 | 18.35%n=80 | 34.95%n=80 | 50.84%n=80 | 43.09%n=80 | -18.33 pp | -17.79 pp | -59.57 pp | -42.97 pp | -27.08 pp | -34.83 pp |
| observability_adequacy | 77.95%n=80 | 50.72%n=80 | 51.42%n=80 | 36.75%n=80 | 49.62%n=80 | 50.67%n=80 | 30.83%n=80 | -27.23 pp | -26.53 pp | -41.20 pp | -28.33 pp | -27.28 pp | -47.12 pp |
| unit_understanding | 86.19%n=80 | 48.48%n=80 | 48.91%n=80 | 32.75%n=80 | 35.97%n=80 | 68.20%n=80 | 29.82%n=80 | -37.71 pp | -37.28 pp | -53.44 pp | -50.22 pp | -17.99 pp | -56.37 pp |
| rubric_based_quality_assessment | 94.76%n=80 | 50.96%n=80 | 51.42%n=80 | 27.01%n=80 | 46.52%n=80 | 41.53%n=80 | 38.98%n=80 | -43.80 pp | -43.34 pp | -67.75 pp | -48.24 pp | -53.23 pp | -55.78 pp |
| benchmark_contamination_clues | 63.58%n=80 | 54.42%n=80 | 54.43%n=80 | 42.83%n=80 | 48.29%n=80 | 52.14%n=80 | 39.87%n=80 | -9.16 pp | -9.15 pp | -20.75 pp | -15.29 pp | -11.44 pp | -23.71 pp |
| scheduling_feasibility | 60.28%n=80 | 48.77%n=80 | 49.31%n=80 | 42.36%n=80 | 55.29%n=80 | 66.40%n=80 | 33.34%n=80 | -11.51 pp | -10.97 pp | -17.92 pp | -4.99 pp | +6.12 pp | -26.94 pp |
| analogy_validity | 60.89%n=80 | 52.05%n=80 | 52.84%n=80 | 30.29%n=80 | 59.79%n=80 | 64.52%n=80 | 35.42%n=80 | -8.84 pp | -8.05 pp | -30.60 pp | -1.10 pp | +3.63 pp | -25.47 pp |
| document_consistency | 75.84%n=80 | 53.27%n=80 | 53.99%n=80 | 24.41%n=80 | 50.85%n=80 | 51.91%n=80 | 47.48%n=80 | -22.57 pp | -21.85 pp | -51.43 pp | -24.99 pp | -23.93 pp | -28.36 pp |
| adversarial_example_detection | 74.25%n=80 | 52.08%n=80 | 52.81%n=80 | 47.53%n=80 | 54.31%n=80 | 43.96%n=80 | 36.83%n=80 | -22.17 pp | -21.44 pp | -26.72 pp | -19.94 pp | -30.29 pp | -37.42 pp |
| parallelization_suitability | 87.22%n=80 | 53.49%n=80 | 54.13%n=80 | 38.28%n=80 | 55.40%n=80 | 48.12%n=80 | 26.91%n=80 | -33.73 pp | -33.09 pp | -48.94 pp | -31.82 pp | -39.10 pp | -60.31 pp |
| open_source_license_compatibility | 77.54%n=80 | 58.51%n=80 | 59.03%n=80 | 33.61%n=80 | 57.61%n=80 | 40.63%n=80 | 36.65%n=80 | -19.03 pp | -18.51 pp | -43.93 pp | -19.93 pp | -36.91 pp | -40.89 pp |
| test_quality_and_adequacy | 75.94%n=80 | 61.90%n=80 | 62.81%n=80 | 32.67%n=80 | 48.85%n=80 | 58.18%n=80 | 26.04%n=80 | -14.04 pp | -13.13 pp | -43.27 pp | -27.09 pp | -17.76 pp | -49.90 pp |
| accounting_reasonableness | 88.06%n=80 | 49.33%n=80 | 49.63%n=80 | 41.57%n=80 | 45.55%n=80 | 44.97%n=80 | 51.77%n=80 | -38.73 pp | -38.43 pp | -46.49 pp | -42.51 pp | -43.09 pp | -36.29 pp |
| expected_utility_decisions | 98.41%n=80 | 58.66%n=80 | 58.60%n=80 | 37.33%n=80 | 40.68%n=80 | 41.28%n=80 | 37.43%n=80 | -39.75 pp | -39.81 pp | -61.08 pp | -57.73 pp | -57.13 pp | -60.98 pp |
| inventory_decisions | 74.91%n=80 | 56.00%n=80 | 56.39%n=80 | 34.24%n=80 | 53.03%n=80 | 55.88%n=80 | 42.43%n=80 | -18.91 pp | -18.52 pp | -40.67 pp | -21.88 pp | -19.03 pp | -32.48 pp |
| reversibility_destructiveness | 80.65%n=80 | 55.77%n=80 | 56.38%n=80 | 40.11%n=80 | 49.23%n=80 | 50.42%n=80 | 40.54%n=80 | -24.88 pp | -24.27 pp | -40.54 pp | -31.42 pp | -30.23 pp | -40.11 pp |
| explanation_quality_judgement | 80.71%n=80 | 62.76%n=80 | 63.62%n=80 | 34.15%n=80 | 44.36%n=80 | 55.14%n=80 | 33.61%n=80 | -17.95 pp | -17.09 pp | -46.56 pp | -36.35 pp | -25.57 pp | -47.10 pp |
| claim_entailment | 74.84%n=80 | 52.90%n=80 | 53.06%n=80 | 52.74%n=80 | 30.64%n=80 | 67.69%n=80 | 42.77%n=80 | -21.94 pp | -21.78 pp | -22.10 pp | -44.20 pp | -7.15 pp | -32.07 pp |
| sensitivity_analysis | 78.97%n=80 | 54.40%n=80 | 54.84%n=80 | 35.94%n=80 | 50.06%n=80 | 64.99%n=80 | 36.85%n=80 | -24.57 pp | -24.13 pp | -43.03 pp | -28.91 pp | -13.98 pp | -42.12 pp |
| calendar_time_zone_understanding | 94.84%n=80 | 49.05%n=80 | 49.45%n=80 | 60.46%n=80 | 47.97%n=80 | 49.70%n=80 | 27.25%n=80 | -45.79 pp | -45.39 pp | -34.38 pp | -46.87 pp | -45.14 pp | -67.59 pp |
| calibration_under_ambiguity | 73.36%n=80 | 68.90%n=80 | 70.49%n=80 | 26.79%n=80 | 70.15%n=80 | 49.17%n=80 | 22.94%n=80 | -4.46 pp | -2.87 pp | -46.57 pp | -3.21 pp | -24.19 pp | -50.42 pp |
| weather_state_interpretation | 96.11%n=80 | 71.56%n=80 | 70.83%n=80 | 32.17%n=80 | 78.62%n=80 | 16.86%n=80 | 16.67%n=80 | -24.55 pp | -25.28 pp | -63.94 pp | -17.49 pp | -79.25 pp | -79.44 pp |
| spatial_relation_reasoning | 88.48%n=80 | 57.29%n=80 | 57.93%n=80 | 61.27%n=80 | 56.60%n=80 | 41.77%n=80 | 20.57%n=80 | -31.19 pp | -30.55 pp | -27.21 pp | -31.88 pp | -46.71 pp | -67.91 pp |
| spurious_correlation_resistance | 46.52%n=80 | 70.89%n=80 | 71.82%n=80 | 54.61%n=80 | 54.51%n=80 | 76.88%n=80 | 11.55%n=80 | +24.37 pp | +25.30 pp | +8.09 pp | +7.99 pp | +30.36 pp | -34.97 pp |
| budget_consistency | 68.50%n=80 | 54.80%n=80 | 53.75%n=80 | 49.27%n=80 | 56.85%n=80 | 52.99%n=80 | 52.48%n=80 | -13.70 pp | -14.75 pp | -19.23 pp | -11.65 pp | -15.51 pp | -16.02 pp |
| prioritization_under_constraints | 77.47%n=80 | 64.35%n=80 | 65.09%n=80 | 24.66%n=80 | 71.89%n=80 | 53.77%n=80 | 31.64%n=80 | -13.12 pp | -12.38 pp | -52.81 pp | -5.58 pp | -23.70 pp | -45.83 pp |
| pareto_optimal_decision_taking | 68.24%n=80 | 49.99%n=80 | 49.97%n=80 | 66.61%n=80 | 56.35%n=80 | 32.50%n=80 | 75.36%n=80 | -18.25 pp | -18.27 pp | -1.63 pp | -11.89 pp | -35.74 pp | +7.12 pp |
| contract_clause_consistency | 70.68%n=80 | 64.40%n=80 | 65.04%n=80 | 17.19%n=80 | 78.44%n=80 | 54.63%n=80 | 50.08%n=80 | -6.28 pp | -5.64 pp | -53.49 pp | +7.76 pp | -16.05 pp | -20.60 pp |
| spreadsheet_formula_semantics_from_text | 99.46%n=80 | 68.87%n=80 | 69.35%n=80 | 26.00%n=80 | 56.03%n=80 | 57.25%n=80 | 24.13%n=80 | -30.59 pp | -30.11 pp | -73.46 pp | -43.43 pp | -42.21 pp | -75.33 pp |
| table_reasoning_from_linearized_text | 93.88%n=80 | 72.62%n=80 | 73.11%n=80 | 24.92%n=80 | 70.25%n=80 | 44.00%n=80 | 23.75%n=80 | -21.26 pp | -20.77 pp | -68.96 pp | -23.63 pp | -49.88 pp | -70.13 pp |
| confidence_vs_evidence_consistency | 80.00%n=80 | 60.66%n=80 | 61.34%n=80 | 21.95%n=80 | 71.25%n=80 | 79.80%n=80 | 28.27%n=80 | -19.34 pp | -18.66 pp | -58.05 pp | -8.75 pp | -0.20 pp | -51.73 pp |
| consistency_under_irrelevant_context_insertion | 94.49%n=80 | 58.65%n=80 | 58.57%n=80 | 53.57%n=80 | 49.12%n=80 | 45.43%n=80 | 44.52%n=80 | -35.84 pp | -35.92 pp | -40.92 pp | -45.37 pp | -49.06 pp | -49.97 pp |
| decision_robustness | 93.11%n=80 | 53.66%n=80 | 53.29%n=80 | 44.27%n=80 | 53.36%n=80 | 57.46%n=80 | 51.73%n=80 | -39.45 pp | -39.82 pp | -48.84 pp | -39.75 pp | -35.65 pp | -41.38 pp |
| dependency_api_existence_judgements | 88.19%n=80 | 67.92%n=80 | 68.27%n=80 | 42.61%n=80 | 70.21%n=80 | 54.20%n=80 | 18.83%n=80 | -20.27 pp | -19.92 pp | -45.58 pp | -17.98 pp | -33.99 pp | -69.36 pp |
| oos_abstention_ambiguity | 74.54%n=80 | 65.44%n=80 | 66.15%n=80 | 38.31%n=80 | 64.29%n=80 | 53.11%n=80 | 49.87%n=80 | -9.10 pp | -8.39 pp | -36.23 pp | -10.25 pp | -21.43 pp | -24.67 pp |
| preference_learning_from_explicit_examples | 97.39%n=80 | 62.90%n=80 | 63.25%n=80 | 35.38%n=80 | 56.63%n=80 | 46.79%n=80 | 49.74%n=80 | -34.49 pp | -34.14 pp | -62.01 pp | -40.76 pp | -50.60 pp | -47.65 pp |
| transaction_anomaly_detection | 93.66%n=80 | 67.08%n=80 | 67.41%n=80 | 42.49%n=80 | 72.10%n=80 | 63.26%n=80 | 7.01%n=80 | -26.58 pp | -26.25 pp | -51.17 pp | -21.56 pp | -30.40 pp | -86.65 pp |
| error_handling_quality | 90.95%n=80 | 61.83%n=80 | 62.67%n=80 | 32.88%n=80 | 74.35%n=80 | 59.22%n=80 | 31.35%n=80 | -29.12 pp | -28.28 pp | -58.07 pp | -16.60 pp | -31.73 pp | -59.60 pp |
| consistency_under_paraphrase | 82.52%n=80 | 63.39%n=80 | 62.98%n=80 | 47.97%n=80 | 74.36%n=80 | 48.84%n=80 | 36.54%n=80 | -19.13 pp | -19.54 pp | -34.55 pp | -8.16 pp | -33.68 pp | -45.98 pp |
| dominated_information_detection | 81.74%n=80 | 65.55%n=80 | 63.67%n=80 | 49.51%n=80 | 55.57%n=80 | 49.40%n=80 | 51.32%n=80 | -16.19 pp | -18.07 pp | -32.23 pp | -26.17 pp | -32.34 pp | -30.42 pp |
| pragmatic_intent_understanding | 81.96%n=80 | 69.96%n=80 | 70.65%n=80 | 33.86%n=80 | 58.10%n=80 | 80.70%n=80 | 21.80%n=80 | -12.00 pp | -11.31 pp | -48.10 pp | -23.86 pp | -1.26 pp | -60.16 pp |
| human_vs_machine_task_suitability | 77.15%n=80 | 65.61%n=80 | 65.99%n=80 | 43.00%n=80 | 59.18%n=80 | 63.60%n=80 | 42.74%n=80 | -11.54 pp | -11.16 pp | -34.15 pp | -17.97 pp | -13.55 pp | -34.41 pp |
| forecast_consistency | 67.86%n=80 | 62.85%n=80 | 60.65%n=80 | 42.61%n=80 | 64.16%n=80 | 60.01%n=80 | 59.99%n=80 | -5.01 pp | -7.21 pp | -25.25 pp | -3.70 pp | -7.85 pp | -7.87 pp |
| sarcasm_and_humor_understanding | 89.79%n=80 | 67.06%n=80 | 67.62%n=80 | 36.82%n=80 | 76.93%n=80 | 60.50%n=80 | 22.35%n=80 | -22.73 pp | -22.17 pp | -52.97 pp | -12.86 pp | -29.29 pp | -67.44 pp |
| self_consistency_across_outputs | 67.76%n=80 | 59.37%n=80 | 57.53%n=80 | 65.15%n=80 | 48.00%n=80 | 60.54%n=80 | 63.03%n=80 | -8.39 pp | -10.23 pp | -2.61 pp | -19.76 pp | -7.22 pp | -4.73 pp |
| frustration_level_hard | 78.57%n=80 | 68.28%n=80 | 69.65%n=80 | 38.05%n=80 | 66.74%n=80 | 57.07%n=80 | 45.60%n=80 | -10.29 pp | -8.92 pp | -40.52 pp | -11.83 pp | -21.50 pp | -32.97 pp |
| data_migration_correctness | 78.46%n=80 | 66.95%n=80 | 67.83%n=80 | 55.97%n=80 | 64.07%n=80 | 63.98%n=80 | 29.32%n=80 | -11.51 pp | -10.63 pp | -22.49 pp | -14.39 pp | -14.48 pp | -49.14 pp |
| historical_anachronism_detection | 86.71%n=80 | 70.85%n=80 | 67.11%n=80 | 46.51%n=80 | 65.40%n=80 | 68.38%n=80 | 23.42%n=80 | -15.86 pp | -19.60 pp | -40.20 pp | -21.31 pp | -18.33 pp | -63.29 pp |
| order_of_magnitude_reasoning | 95.82%n=80 | 75.77%n=80 | 76.75%n=80 | 35.54%n=80 | 65.28%n=80 | 57.33%n=80 | 25.81%n=80 | -20.05 pp | -19.07 pp | -60.28 pp | -30.54 pp | -38.49 pp | -70.01 pp |
| information_acquisition_value | 89.40%n=80 | 58.68%n=80 | 59.35%n=80 | 48.82%n=80 | 62.61%n=80 | 61.29%n=80 | 55.32%n=80 | -30.72 pp | -30.05 pp | -40.58 pp | -26.79 pp | -28.11 pp | -34.08 pp |
| accessibility_semantics_from_source | 92.80%n=80 | 62.72%n=80 | 63.36%n=80 | 51.31%n=80 | 56.43%n=80 | 53.34%n=80 | 59.55%n=80 | -30.08 pp | -29.44 pp | -41.49 pp | -36.37 pp | -39.46 pp | -33.25 pp |
| game_theory_judgements | 98.59%n=80 | 65.80%n=80 | 66.17%n=80 | 36.83%n=80 | 58.91%n=80 | 53.71%n=80 | 61.04%n=80 | -32.79 pp | -32.42 pp | -61.76 pp | -39.68 pp | -44.88 pp | -37.55 pp |
| operational_urgency | 93.67%n=80 | 60.12%n=80 | 63.16%n=80 | 30.23%n=80 | 68.26%n=80 | 63.75%n=80 | 62.98%n=80 | -33.55 pp | -30.51 pp | -63.44 pp | -25.41 pp | -29.92 pp | -30.69 pp |
| hallucination_detection | 76.22%n=80 | 70.53%n=80 | 71.02%n=80 | 45.92%n=80 | 76.73%n=80 | 75.00%n=80 | 27.51%n=80 | -5.69 pp | -5.20 pp | -30.30 pp | +0.51 pp | -1.22 pp | -48.71 pp |
| business_process_conformance | 98.32%n=80 | 74.83%n=80 | 75.41%n=80 | 26.86%n=80 | 70.31%n=80 | 75.07%n=80 | 23.84%n=80 | -23.49 pp | -22.91 pp | -71.46 pp | -28.01 pp | -23.25 pp | -74.48 pp |
| sql_injection_risk | 94.08%n=80 | 76.12%n=80 | 75.16%n=80 | 40.01%n=80 | 59.85%n=80 | 52.62%n=80 | 47.50%n=80 | -17.96 pp | -18.92 pp | -54.07 pp | -34.23 pp | -41.46 pp | -46.58 pp |
| embodied_animal_understanding | 89.40%n=80 | 75.48%n=80 | 76.08%n=80 | 38.34%n=80 | 66.34%n=80 | 57.75%n=80 | 42.44%n=80 | -13.92 pp | -13.32 pp | -51.06 pp | -23.06 pp | -31.65 pp | -46.96 pp |
| bayesian_evidence_updating | 89.64%n=80 | 64.49%n=80 | 64.56%n=80 | 55.33%n=80 | 56.88%n=80 | 60.21%n=80 | 55.35%n=80 | -25.15 pp | -25.08 pp | -34.31 pp | -32.76 pp | -29.43 pp | -34.29 pp |
| answerability_from_provided_context | 88.50%n=80 | 74.48%n=80 | 74.93%n=80 | 39.40%n=80 | 79.23%n=80 | 60.56%n=80 | 29.64%n=80 | -14.02 pp | -13.57 pp | -49.10 pp | -9.27 pp | -27.94 pp | -58.86 pp |
| experimental_design_judgement | 96.84%n=80 | 69.47%n=80 | 70.31%n=80 | 47.14%n=80 | 65.78%n=80 | 65.80%n=80 | 32.02%n=80 | -27.37 pp | -26.53 pp | -49.70 pp | -31.06 pp | -31.04 pp | -64.82 pp |
| persona_consistency | 88.41%n=80 | 73.90%n=80 | 74.43%n=80 | 33.34%n=80 | 59.82%n=80 | 61.04%n=80 | 57.28%n=80 | -14.51 pp | -13.98 pp | -55.07 pp | -28.59 pp | -27.37 pp | -31.13 pp |
| pairwise_preference_prediction | 89.65%n=80 | 63.53%n=80 | 64.11%n=80 | 37.16%n=80 | 63.22%n=80 | 64.12%n=80 | 66.55%n=80 | -26.12 pp | -25.54 pp | -52.49 pp | -26.43 pp | -25.53 pp | -23.10 pp |
| insurance_claim_priority | 84.03%n=80 | 71.51%n=80 | 73.31%n=80 | 33.10%n=80 | 70.48%n=80 | 73.70%n=80 | 42.51%n=80 | -12.52 pp | -10.72 pp | -50.93 pp | -13.55 pp | -10.33 pp | -41.52 pp |
| recommendation_reranking_quality | 99.94%n=80 | 76.39%n=80 | 76.98%n=80 | 30.82%n=80 | 65.42%n=80 | 65.25%n=80 | 34.04%n=80 | -23.55 pp | -22.96 pp | -69.12 pp | -34.52 pp | -34.69 pp | -65.90 pp |
| configuration_correctness | 89.38%n=80 | 72.01%n=80 | 72.80%n=80 | 35.44%n=80 | 62.87%n=80 | 76.16%n=80 | 41.26%n=80 | -17.37 pp | -16.58 pp | -53.94 pp | -26.51 pp | -13.22 pp | -48.12 pp |
| meeting_conflict | 97.74%n=80 | 72.23%n=80 | 71.58%n=80 | 45.55%n=80 | 72.37%n=80 | 46.26%n=80 | 44.20%n=80 | -25.51 pp | -26.16 pp | -52.19 pp | -25.37 pp | -51.48 pp | -53.54 pp |
| rule_following_formal_logic | 99.86%n=80 | 69.67%n=80 | 70.17%n=80 | 43.22%n=80 | 70.22%n=80 | 56.92%n=80 | 40.05%n=80 | -30.19 pp | -29.69 pp | -56.64 pp | -29.64 pp | -42.94 pp | -59.81 pp |
| semantic_similarity_and_factual_correctness | 77.09%n=80 | 70.75%n=80 | 71.17%n=80 | 45.77%n=80 | 80.07%n=80 | 73.68%n=80 | 32.02%n=80 | -6.34 pp | -5.92 pp | -31.32 pp | +2.98 pp | -3.41 pp | -45.07 pp |
| comparative_judgement_transitivity | 93.15%n=80 | 81.18%n=80 | 76.90%n=80 | 46.29%n=80 | 69.87%n=80 | 42.03%n=80 | 41.29%n=80 | -11.97 pp | -16.25 pp | -46.86 pp | -23.28 pp | -51.12 pp | -51.86 pp |
| refund_eligible_hard | 77.29%n=80 | 71.62%n=80 | 69.18%n=80 | 55.28%n=80 | 69.10%n=80 | 56.11%n=80 | 53.15%n=80 | -5.67 pp | -8.11 pp | -22.01 pp | -8.19 pp | -21.18 pp | -24.14 pp |
| formality_level | 88.51%n=80 | 77.58%n=80 | 79.93%n=80 | 27.39%n=80 | 64.44%n=80 | 65.03%n=80 | 50.57%n=80 | -10.93 pp | -8.58 pp | -61.12 pp | -24.07 pp | -23.48 pp | -37.94 pp |
| recoverability_judgement | 90.84%n=80 | 79.23%n=80 | 79.84%n=80 | 20.33%n=80 | 74.79%n=80 | 82.45%n=80 | 26.14%n=80 | -11.61 pp | -11.00 pp | -70.51 pp | -16.05 pp | -8.39 pp | -64.70 pp |
| algorithm_data_structure_appropriateness | 74.01%n=80 | 69.19%n=80 | 69.63%n=80 | 60.21%n=80 | 69.10%n=80 | 74.03%n=80 | 37.67%n=80 | -4.82 pp | -4.38 pp | -13.80 pp | -4.91 pp | +0.02 pp | -36.34 pp |
| secret_leak_hard | 92.56%n=80 | 76.41%n=80 | 74.57%n=80 | 50.17%n=80 | 66.59%n=80 | 46.39%n=80 | 50.18%n=80 | -16.15 pp | -17.99 pp | -42.39 pp | -25.97 pp | -46.17 pp | -42.38 pp |
| scientific_hypothesis_plausibility | 92.90%n=80 | 77.63%n=80 | 78.44%n=80 | 29.80%n=80 | 80.56%n=80 | 63.06%n=80 | 34.49%n=80 | -15.27 pp | -14.46 pp | -63.10 pp | -12.34 pp | -29.84 pp | -58.41 pp |
| counterfactual_plausibility | 74.25%n=80 | 67.82%n=80 | 65.37%n=80 | 55.59%n=80 | 70.12%n=80 | 97.09%n=80 | 27.24%n=80 | -6.43 pp | -8.88 pp | -18.66 pp | -4.13 pp | +22.84 pp | -47.01 pp |
| translation_correctness | 99.77%n=80 | 81.41%n=80 | 82.02%n=80 | 27.48%n=80 | 80.22%n=80 | 67.10%n=80 | 19.71%n=80 | -18.36 pp | -17.75 pp | -72.29 pp | -19.55 pp | -32.67 pp | -80.06 pp |
| negation_polarity_tracking | 76.79%n=80 | 70.08%n=80 | 70.57%n=80 | 51.29%n=80 | 74.02%n=80 | 69.65%n=80 | 45.36%n=80 | -6.71 pp | -6.22 pp | -25.50 pp | -2.77 pp | -7.14 pp | -31.43 pp |
| comparative_language_reasoning | 79.33%n=80 | 73.23%n=80 | 73.61%n=80 | 46.09%n=80 | 75.93%n=80 | 73.58%n=80 | 36.25%n=80 | -6.10 pp | -5.72 pp | -33.24 pp | -3.40 pp | -5.75 pp | -43.08 pp |
| abstraction_level_matching | 91.78%n=80 | 62.60%n=80 | 63.31%n=80 | 40.73%n=80 | 84.14%n=80 | 80.01%n=80 | 35.67%n=80 | -29.18 pp | -28.47 pp | -51.05 pp | -7.64 pp | -11.77 pp | -56.11 pp |
| causal_chain_validation | 93.36%n=80 | 73.48%n=80 | 74.33%n=80 | 50.85%n=80 | 80.58%n=80 | 65.06%n=80 | 20.66%n=80 | -19.88 pp | -19.03 pp | -42.51 pp | -12.78 pp | -28.30 pp | -72.70 pp |
| tool_action_selection | 81.76%n=80 | 86.68%n=80 | 87.17%n=80 | 25.96%n=80 | 85.80%n=80 | 71.42%n=80 | 20.16%n=80 | +4.92 pp | +5.41 pp | -55.80 pp | +4.04 pp | -10.34 pp | -61.60 pp |
| performance_bottleneck_attribution | 94.81%n=80 | 76.72%n=80 | 77.39%n=80 | 18.27%n=80 | 82.94%n=80 | 88.51%n=80 | 23.31%n=80 | -18.09 pp | -17.42 pp | -76.54 pp | -11.87 pp | -6.30 pp | -71.50 pp |
| form_completeness | 99.29%n=80 | 72.99%n=80 | 73.85%n=80 | 50.05%n=80 | 65.00%n=80 | 59.68%n=80 | 42.52%n=80 | -26.30 pp | -25.44 pp | -49.24 pp | -34.29 pp | -39.61 pp | -56.77 pp |
| scope_reasoning | 89.00%n=80 | 74.29%n=80 | 74.85%n=80 | 43.62%n=80 | 77.05%n=80 | 70.61%n=80 | 33.99%n=80 | -14.71 pp | -14.15 pp | -45.38 pp | -11.95 pp | -18.39 pp | -55.01 pp |
| incident_severity_hard | 91.93%n=80 | 78.24%n=80 | 80.89%n=80 | 25.94%n=80 | 71.92%n=80 | 79.66%n=80 | 35.11%n=80 | -13.69 pp | -11.04 pp | -65.99 pp | -20.01 pp | -12.27 pp | -56.82 pp |
| veterinary_triage | 88.80%n=80 | 72.70%n=80 | 75.41%n=80 | 39.59%n=80 | 58.10%n=80 | 48.50%n=80 | 83.81%n=80 | -16.10 pp | -13.39 pp | -49.21 pp | -30.70 pp | -40.30 pp | -4.99 pp |
| dialogue_coherence | 91.26%n=80 | 76.78%n=80 | 77.57%n=80 | 21.65%n=80 | 77.36%n=80 | 82.15%n=80 | 40.93%n=80 | -14.48 pp | -13.69 pp | -69.61 pp | -13.90 pp | -9.11 pp | -50.33 pp |
| competing_hypothesis_discrimination | 95.12%n=80 | 64.77%n=80 | 65.42%n=80 | 48.28%n=80 | 78.13%n=80 | 87.96%n=80 | 28.39%n=80 | -30.35 pp | -29.70 pp | -46.84 pp | -16.99 pp | -7.16 pp | -66.73 pp |
| review_sentiment_hard | 94.92%n=80 | 78.76%n=80 | 81.63%n=80 | 27.05%n=80 | 75.85%n=80 | 68.12%n=80 | 42.18%n=80 | -16.16 pp | -13.29 pp | -67.87 pp | -19.07 pp | -26.80 pp | -52.74 pp |
| symptom_triage | 80.85%n=80 | 72.61%n=80 | 74.86%n=80 | 34.28%n=80 | 58.15%n=80 | 51.46%n=80 | 96.51%n=80 | -8.24 pp | -5.99 pp | -46.57 pp | -22.70 pp | -29.39 pp | +15.66 pp |
| fact_vs_inference_classification | 76.70%n=80 | 74.62%n=80 | 74.86%n=80 | 46.64%n=80 | 75.77%n=80 | 74.94%n=80 | 46.02%n=80 | -2.08 pp | -1.84 pp | -30.06 pp | -0.93 pp | -1.76 pp | -30.68 pp |
| reference_coreference_resolution | 98.86%n=80 | 81.61%n=80 | 82.28%n=80 | 46.48%n=80 | 63.51%n=80 | 61.08%n=80 | 35.92%n=80 | -17.25 pp | -16.58 pp | -52.38 pp | -35.35 pp | -37.78 pp | -62.94 pp |
| temporal_ordering | 99.99%n=80 | 88.28%n=80 | 88.73%n=80 | 31.16%n=80 | 49.08%n=80 | 78.18%n=80 | 34.45%n=80 | -11.71 pp | -11.26 pp | -68.83 pp | -50.91 pp | -21.81 pp | -65.54 pp |
| quantifier_reasoning_in_natural_text | 77.84%n=80 | 74.29%n=80 | 74.69%n=80 | 48.39%n=80 | 74.81%n=80 | 76.69%n=80 | 46.15%n=80 | -3.55 pp | -3.15 pp | -29.45 pp | -3.03 pp | -1.15 pp | -31.69 pp |
| probability_calibration_language | 98.81%n=80 | 72.50%n=80 | 72.96%n=80 | 32.32%n=80 | 74.77%n=80 | 88.14%n=80 | 37.85%n=80 | -26.31 pp | -25.85 pp | -66.49 pp | -24.04 pp | -10.67 pp | -60.96 pp |
| cross_domain_synthesis | 86.75%n=80 | 76.86%n=80 | 71.43%n=80 | 39.13%n=80 | 79.42%n=80 | 62.65%n=80 | 62.37%n=80 | -9.89 pp | -15.32 pp | -47.62 pp | -7.33 pp | -24.10 pp | -24.38 pp |
| idempotency_judgements | 88.96%n=80 | 72.34%n=80 | 73.14%n=80 | 46.04%n=80 | 74.52%n=80 | 97.50%n=80 | 29.42%n=80 | -16.62 pp | -15.82 pp | -42.92 pp | -14.44 pp | +8.54 pp | -59.54 pp |
| support_department_hard | 92.41%n=80 | 79.34%n=80 | 79.77%n=80 | 34.34%n=80 | 82.06%n=80 | 80.13%n=80 | 34.21%n=80 | -13.07 pp | -12.64 pp | -58.07 pp | -10.35 pp | -12.28 pp | -58.20 pp |
| data_quality_validation | 98.48%n=80 | 73.45%n=80 | 74.11%n=80 | 49.96%n=80 | 68.32%n=80 | 89.06%n=80 | 28.89%n=80 | -25.03 pp | -24.37 pp | -48.52 pp | -30.16 pp | -9.42 pp | -69.59 pp |
| embodied_time_aware_world_views | 79.00%n=80 | 76.86%n=80 | 73.55%n=80 | 54.91%n=80 | 77.40%n=80 | 64.61%n=80 | 56.55%n=80 | -2.14 pp | -5.45 pp | -24.09 pp | -1.60 pp | -14.39 pp | -22.45 pp |
| serialization_compatibility | 80.97%n=80 | 71.43%n=80 | 72.20%n=80 | 52.54%n=80 | 65.96%n=80 | 96.94%n=80 | 43.15%n=80 | -9.54 pp | -8.77 pp | -28.43 pp | -15.01 pp | +15.97 pp | -37.82 pp |
| travel_policy_violation | 83.61%n=80 | 79.61%n=80 | 76.28%n=80 | 56.19%n=80 | 71.98%n=80 | 50.26%n=80 | 65.88%n=80 | -4.00 pp | -7.33 pp | -27.42 pp | -11.63 pp | -33.35 pp | -17.73 pp |
| structured_schema_decisions | 99.18%n=80 | 91.68%n=80 | 92.16%n=80 | 27.43%n=80 | 86.07%n=80 | 49.99%n=80 | 37.60%n=80 | -7.50 pp | -7.02 pp | -71.75 pp | -13.11 pp | -49.19 pp | -61.58 pp |
| rationale_to_answer_consistency | 93.75%n=80 | 73.89%n=80 | 71.52%n=80 | 74.00%n=80 | 62.55%n=80 | 61.27%n=80 | 50.53%n=80 | -19.86 pp | -22.23 pp | -19.75 pp | -31.20 pp | -32.48 pp | -43.22 pp |
| invoice_anomaly_detection | 84.00%n=80 | 81.77%n=80 | 82.33%n=80 | 51.53%n=80 | 76.29%n=80 | 80.94%n=80 | 32.92%n=80 | -2.23 pp | -1.67 pp | -32.47 pp | -7.71 pp | -3.06 pp | -51.08 pp |
| email_thread_state_understanding | 71.62%n=80 | 77.07%n=80 | 77.96%n=80 | 73.51%n=80 | 87.04%n=80 | 72.58%n=80 | 31.85%n=80 | +5.45 pp | +6.34 pp | +1.89 pp | +15.42 pp | +0.96 pp | -39.77 pp |
| sampling_statistical_interpretation | 93.35%n=80 | 82.88%n=80 | 83.65%n=80 | 38.96%n=80 | 82.27%n=80 | 86.63%n=80 | 30.16%n=80 | -10.47 pp | -9.70 pp | -54.39 pp | -11.08 pp | -6.72 pp | -63.19 pp |
| security_policy_risk | 85.80%n=80 | 78.49%n=80 | 78.58%n=80 | 45.54%n=80 | 77.52%n=80 | 81.64%n=80 | 52.21%n=80 | -7.31 pp | -7.22 pp | -40.26 pp | -8.28 pp | -4.16 pp | -33.59 pp |
| embodied_affordances | 92.69%n=80 | 83.81%n=80 | 81.09%n=80 | 49.61%n=80 | 80.53%n=80 | 51.20%n=80 | 61.82%n=80 | -8.88 pp | -11.60 pp | -43.08 pp | -12.16 pp | -41.49 pp | -30.87 pp |
| commercial_price_plausibility | 100.00%n=80 | 93.25%n=80 | 93.86%n=80 | 19.62%n=80 | 86.78%n=80 | 88.46%n=80 | 18.94%n=80 | -6.75 pp | -6.14 pp | -80.38 pp | -13.22 pp | -11.54 pp | -81.06 pp |
| warranty_claim_eligible | 71.89%n=80 | 89.46%n=80 | 87.45%n=80 | 56.28%n=80 | 77.26%n=80 | 62.18%n=80 | 56.84%n=80 | +17.57 pp | +15.56 pp | -15.61 pp | +5.37 pp | -9.71 pp | -15.05 pp |
| noisy_ocr_reconstruction | 97.41%n=80 | 90.10%n=80 | 90.80%n=80 | 18.43%n=80 | 98.99%n=80 | 86.31%n=80 | 20.13%n=80 | -7.31 pp | -6.61 pp | -78.98 pp | +1.58 pp | -11.10 pp | -77.28 pp |
| command_line_semantics | 84.75%n=80 | 80.59%n=80 | 81.38%n=80 | 60.22%n=80 | 71.80%n=80 | 76.67%n=80 | 47.17%n=80 | -4.16 pp | -3.37 pp | -24.53 pp | -12.95 pp | -8.08 pp | -37.58 pp |
| boundary_condition_judgement | 90.27%n=80 | 87.35%n=80 | 81.82%n=80 | 71.31%n=80 | 94.47%n=80 | 56.75%n=80 | 23.81%n=80 | -2.92 pp | -8.45 pp | -18.96 pp | +4.20 pp | -33.52 pp | -66.46 pp |
| concurrency_correctness | 98.09%n=80 | 90.46%n=80 | 91.20%n=80 | 41.65%n=80 | 91.64%n=80 | 76.90%n=80 | 16.10%n=80 | -7.63 pp | -6.89 pp | -56.44 pp | -6.45 pp | -21.19 pp | -81.99 pp |
| document_workflow_state | 100.00%n=80 | 95.39%n=80 | 95.82%n=80 | 25.90%n=80 | 76.91%n=80 | 83.12%n=80 | 31.91%n=80 | -4.61 pp | -4.18 pp | -74.10 pp | -23.09 pp | -16.88 pp | -68.09 pp |
| information_retrieval_relevance | 99.26%n=80 | 95.12%n=80 | 95.62%n=80 | 24.85%n=80 | 95.56%n=80 | 100.00%n=80 | 0.01%n=80 | -4.14 pp | -3.64 pp | -74.41 pp | -3.70 pp | +0.74 pp | -99.25 pp |
| world_model_consistency | 97.01%n=80 | 86.26%n=80 | 80.51%n=80 | 44.77%n=80 | 94.12%n=80 | 52.21%n=80 | 55.63%n=80 | -10.75 pp | -16.50 pp | -52.24 pp | -2.89 pp | -44.80 pp | -41.38 pp |
| failure_mode_classification | 99.70%n=80 | 85.48%n=80 | 85.83%n=80 | 40.08%n=80 | 90.72%n=80 | 91.42%n=80 | 21.32%n=80 | -14.22 pp | -13.87 pp | -59.62 pp | -8.98 pp | -8.28 pp | -78.38 pp |
| graph_path_relevance | 99.70%n=80 | 86.71%n=80 | 87.33%n=80 | 13.61%n=80 | 71.61%n=80 | 99.34%n=80 | 58.14%n=80 | -12.99 pp | -12.37 pp | -86.09 pp | -28.09 pp | -0.36 pp | -41.56 pp |
| entity_resolution | 86.84%n=80 | 75.29%n=80 | 74.84%n=80 | 69.47%n=80 | 73.65%n=80 | 69.30%n=80 | 67.96%n=80 | -11.55 pp | -12.00 pp | -17.37 pp | -13.19 pp | -17.54 pp | -18.88 pp |
| incident_severity | 87.87%n=80 | 80.20%n=80 | 82.01%n=80 | 34.40%n=80 | 72.76%n=80 | 73.52%n=80 | 88.20%n=80 | -7.67 pp | -5.86 pp | -53.47 pp | -15.11 pp | -14.35 pp | +0.33 pp |
| social_norm_reasoning | 97.30%n=80 | 87.02%n=80 | 87.44%n=80 | 32.03%n=80 | 95.45%n=80 | 80.00%n=80 | 40.75%n=80 | -10.28 pp | -9.86 pp | -65.27 pp | -1.85 pp | -17.30 pp | -56.55 pp |
| product_catalog_matching | 99.86%n=80 | 92.60%n=80 | 93.09%n=80 | 28.65%n=80 | 92.13%n=80 | 85.70%n=80 | 28.27%n=80 | -7.26 pp | -6.77 pp | -71.21 pp | -7.73 pp | -14.16 pp | -71.59 pp |
| grammar_judgement | 97.76%n=80 | 92.86%n=80 | 93.59%n=80 | 38.37%n=80 | 93.77%n=80 | 79.31%n=80 | 25.51%n=80 | -4.90 pp | -4.17 pp | -59.39 pp | -3.99 pp | -18.45 pp | -72.25 pp |
| geographical_consistency | 75.22%n=80 | 87.13%n=80 | 83.10%n=80 | 37.40%n=80 | 87.29%n=80 | 71.71%n=80 | 79.82%n=80 | +11.91 pp | +7.88 pp | -37.82 pp | +12.07 pp | -3.51 pp | +4.60 pp |
| animal_affordance_constraint_reasoning | 91.81%n=80 | 80.84%n=80 | 78.73%n=80 | 51.35%n=80 | 76.85%n=80 | 72.53%n=80 | 71.95%n=80 | -10.97 pp | -13.08 pp | -40.46 pp | -14.96 pp | -19.28 pp | -19.86 pp |
| live_data_requirement | 87.48%n=80 | 86.66%n=80 | 80.95%n=80 | 61.49%n=80 | 72.44%n=80 | 72.69%n=80 | 62.39%n=80 | -0.82 pp | -6.53 pp | -25.99 pp | -15.04 pp | -14.79 pp | -25.09 pp |
| code_quality_judgements | 89.53%n=80 | 85.18%n=80 | 84.18%n=80 | 51.23%n=80 | 84.57%n=80 | 73.72%n=80 | 58.24%n=80 | -4.35 pp | -5.35 pp | -38.30 pp | -4.96 pp | -15.81 pp | -31.29 pp |
| secret_leak | 90.88%n=80 | 89.53%n=80 | 87.73%n=80 | 63.83%n=80 | 78.54%n=80 | 56.63%n=80 | 59.51%n=80 | -1.35 pp | -3.15 pp | -27.05 pp | -12.34 pp | -34.25 pp | -31.37 pp |
| environment_portability_judgements | 95.25%n=80 | 86.41%n=80 | 87.30%n=80 | 49.45%n=80 | 88.84%n=80 | 98.29%n=80 | 21.38%n=80 | -8.84 pp | -7.95 pp | -45.80 pp | -6.41 pp | +3.04 pp | -73.87 pp |
| compatibility_matching | 80.79%n=80 | 89.64%n=80 | 84.99%n=80 | 35.91%n=80 | 92.65%n=80 | 84.66%n=80 | 58.64%n=80 | +8.85 pp | +4.20 pp | -44.88 pp | +11.86 pp | +3.87 pp | -22.15 pp |
| cognitive_bias_illogic_detection | 95.01%n=80 | 89.96%n=80 | 90.28%n=80 | 53.69%n=80 | 92.83%n=80 | 91.25%n=80 | 14.81%n=80 | -5.05 pp | -4.73 pp | -41.32 pp | -2.18 pp | -3.76 pp | -80.20 pp |
| complexity_estimation | 99.32%n=80 | 95.40%n=80 | 95.90%n=80 | 20.60%n=80 | 95.40%n=80 | 96.15%n=80 | 26.54%n=80 | -3.92 pp | -3.42 pp | -78.72 pp | -3.92 pp | -3.17 pp | -72.78 pp |
| explanation_necessity_judgement | 90.29%n=80 | 87.01%n=80 | 81.33%n=80 | 40.71%n=80 | 91.06%n=80 | 82.49%n=80 | 57.19%n=80 | -3.28 pp | -8.96 pp | -49.58 pp | +0.77 pp | -7.80 pp | -33.10 pp |
| embodied_unknowns | 83.30%n=80 | 90.01%n=80 | 85.25%n=80 | 45.05%n=80 | 87.01%n=80 | 98.24%n=80 | 41.93%n=80 | +6.71 pp | +1.95 pp | -38.25 pp | +3.71 pp | +14.94 pp | -41.37 pp |
| lead_qualification | 90.46%n=80 | 80.92%n=80 | 83.61%n=80 | 39.15%n=80 | 73.10%n=80 | 82.14%n=80 | 82.98%n=80 | -9.54 pp | -6.85 pp | -51.31 pp | -17.36 pp | -8.32 pp | -7.48 pp |
| summary_completeness | 99.88%n=80 | 90.65%n=80 | 91.40%n=80 | 21.94%n=80 | 68.45%n=80 | 81.61%n=80 | 79.00%n=80 | -9.23 pp | -8.48 pp | -77.94 pp | -31.43 pp | -18.27 pp | -20.88 pp |
| typed_jev_native_replay | 84.59%n=80 | 83.63%n=80 | 83.53%n=80 | 52.11%n=80 | 80.57%n=80 | 81.34%n=80 | 68.92%n=80 | -0.96 pp | -1.06 pp | -32.48 pp | -4.02 pp | -3.25 pp | -15.67 pp |
| commit_intent | 91.65%n=80 | 89.77%n=80 | 90.31%n=80 | 51.85%n=80 | 69.45%n=80 | 89.98%n=80 | 51.85%n=80 | -1.88 pp | -1.34 pp | -39.80 pp | -22.20 pp | -1.67 pp | -39.80 pp |
| class_label_leakage_detection | 74.48%n=80 | 73.45%n=80 | 70.19%n=80 | 68.71%n=80 | 79.29%n=80 | 89.38%n=80 | 81.45%n=80 | -1.03 pp | -4.29 pp | -5.77 pp | +4.81 pp | +14.90 pp | +6.97 pp |
| root_cause_attribution | 99.49%n=80 | 83.87%n=80 | 84.68%n=80 | 95.01%n=80 | 69.41%n=80 | 15.90%n=80 | 88.79%n=80 | -15.62 pp | -14.81 pp | -4.48 pp | -30.08 pp | -83.59 pp | -10.70 pp |
| review_sentiment | 95.84%n=80 | 86.10%n=80 | 88.05%n=80 | 30.48%n=80 | 83.54%n=80 | 72.70%n=80 | 80.59%n=80 | -9.74 pp | -7.79 pp | -65.36 pp | -12.30 pp | -23.14 pp | -15.25 pp |
| frustration_level | 82.74%n=80 | 78.06%n=80 | 79.25%n=80 | 40.04%n=80 | 82.41%n=80 | 81.05%n=80 | 94.28%n=80 | -4.68 pp | -3.49 pp | -42.70 pp | -0.33 pp | -1.69 pp | +11.54 pp |
| program_behavior_prediction | 94.75%n=80 | 87.49%n=80 | 88.26%n=80 | 54.04%n=80 | 78.97%n=80 | 94.56%n=80 | 43.29%n=80 | -7.26 pp | -6.49 pp | -40.71 pp | -15.78 pp | -0.19 pp | -51.46 pp |
| biological_signal_understanding | 99.76%n=80 | 88.66%n=80 | 89.41%n=80 | 51.96%n=80 | 88.22%n=80 | 92.76%n=80 | 32.25%n=80 | -11.10 pp | -10.35 pp | -47.80 pp | -11.54 pp | -7.00 pp | -67.51 pp |
| contradiction_resolution | 97.65%n=80 | 92.86%n=80 | 93.60%n=80 | 46.19%n=80 | 90.59%n=80 | 88.30%n=80 | 37.46%n=80 | -4.79 pp | -4.05 pp | -51.46 pp | -7.06 pp | -9.35 pp | -60.19 pp |
| invariant_detection | 96.06%n=80 | 87.56%n=80 | 85.00%n=80 | 65.99%n=80 | 84.58%n=80 | 67.98%n=80 | 59.48%n=80 | -8.50 pp | -11.06 pp | -30.07 pp | -11.48 pp | -28.08 pp | -36.58 pp |
| dimensional_consistency | 92.15%n=80 | 92.55%n=80 | 89.07%n=80 | 68.32%n=80 | 90.92%n=80 | 60.36%n=80 | 53.41%n=80 | +0.40 pp | -3.08 pp | -23.83 pp | -1.23 pp | -31.79 pp | -38.74 pp |
| document_field_association | 100.00%n=80 | 98.61%n=80 | 98.84%n=80 | 19.39%n=80 | 99.50%n=80 | 100.00%n=80 | 31.20%n=80 | -1.39 pp | -1.16 pp | -80.61 pp | -0.50 pp | +0.00 pp | -68.80 pp |
| version_compatibility | 100.00%n=80 | 86.20%n=80 | 86.92%n=80 | 40.28%n=80 | 85.59%n=80 | 82.19%n=80 | 68.01%n=80 | -13.80 pp | -13.08 pp | -59.72 pp | -14.41 pp | -17.81 pp | -31.99 pp |
| theory_of_mind_false_belief_reasoning | 93.20%n=80 | 82.56%n=80 | 83.52%n=80 | 74.51%n=80 | 83.55%n=80 | 82.22%n=80 | 49.72%n=80 | -10.64 pp | -9.68 pp | -18.69 pp | -9.65 pp | -10.98 pp | -43.48 pp |
| database_query_reasoning | 99.05%n=80 | 87.38%n=80 | 87.92%n=80 | 69.52%n=80 | 76.37%n=80 | 91.94%n=80 | 37.17%n=80 | -11.67 pp | -11.13 pp | -29.53 pp | -22.68 pp | -7.11 pp | -61.88 pp |
| action_item_extraction_verification | 82.09%n=80 | 85.65%n=80 | 82.03%n=80 | 68.66%n=80 | 90.12%n=80 | 80.42%n=80 | 60.73%n=80 | +3.56 pp | -0.06 pp | -13.43 pp | +8.03 pp | -1.67 pp | -21.36 pp |
| action_item_assignment | 85.94%n=80 | 84.24%n=80 | 83.30%n=80 | 56.88%n=80 | 86.67%n=80 | 82.45%n=80 | 70.35%n=80 | -1.70 pp | -2.64 pp | -29.06 pp | +0.73 pp | -3.49 pp | -15.59 pp |
| phishing_email | 91.75%n=80 | 91.11%n=80 | 86.96%n=80 | 67.65%n=80 | 75.95%n=80 | 88.40%n=80 | 48.98%n=80 | -0.64 pp | -4.79 pp | -24.10 pp | -15.80 pp | -3.35 pp | -42.77 pp |
| granularity_selection | 97.06%n=80 | 87.67%n=80 | 88.17%n=80 | 69.31%n=80 | 92.16%n=80 | 75.72%n=80 | 41.42%n=80 | -9.39 pp | -8.89 pp | -27.75 pp | -4.90 pp | -21.34 pp | -55.64 pp |
| long_context_distractor_resistance | 100.00%n=80 | 99.45%n=80 | 99.55%n=80 | 30.46%n=80 | 99.89%n=80 | 100.00%n=80 | 24.27%n=80 | -0.55 pp | -0.45 pp | -69.54 pp | -0.11 pp | +0.00 pp | -75.73 pp |
| content_type | 96.60%n=80 | 90.70%n=80 | 91.39%n=80 | 46.64%n=80 | 79.64%n=80 | 84.42%n=80 | 65.46%n=80 | -5.90 pp | -5.21 pp | -49.96 pp | -16.96 pp | -12.18 pp | -31.14 pp |
| object_permanence_state_tracking | 100.00%n=80 | 98.49%n=80 | 98.72%n=80 | 28.80%n=80 | 96.44%n=80 | 98.75%n=80 | 35.15%n=80 | -1.51 pp | -1.28 pp | -71.20 pp | -3.56 pp | -1.25 pp | -64.85 pp |
| stopping_decision_sufficiency | 85.00%n=80 | 92.61%n=80 | 89.22%n=80 | 73.32%n=80 | 92.74%n=80 | 95.94%n=80 | 28.81%n=80 | +7.61 pp | +4.22 pp | -11.68 pp | +7.74 pp | +10.94 pp | -56.19 pp |
| booking_accounting_mapping | 99.59%n=80 | 86.50%n=80 | 86.84%n=80 | 53.76%n=80 | 90.63%n=80 | 89.69%n=80 | 51.46%n=80 | -13.09 pp | -12.75 pp | -45.83 pp | -8.96 pp | -9.90 pp | -48.13 pp |
| cross_document_reconciliation | 99.50%n=80 | 98.66%n=80 | 98.87%n=80 | 41.64%n=80 | 99.35%n=80 | 99.14%n=80 | 22.34%n=80 | -0.84 pp | -0.63 pp | -57.86 pp | -0.15 pp | -0.36 pp | -77.16 pp |
| physical_commonsense | 93.46%n=80 | 90.68%n=80 | 86.89%n=80 | 38.89%n=80 | 95.40%n=80 | 71.67%n=80 | 82.75%n=80 | -2.78 pp | -6.57 pp | -54.57 pp | +1.94 pp | -21.79 pp | -10.71 pp |
| suspicious_transaction | 88.25%n=80 | 96.34%n=80 | 93.74%n=80 | 69.45%n=80 | 90.15%n=80 | 49.09%n=80 | 73.63%n=80 | +8.09 pp | +5.49 pp | -18.80 pp | +1.90 pp | -39.16 pp | -14.62 pp |
| urgency_hard | 96.49%n=80 | 97.25%n=80 | 95.16%n=80 | 59.18%n=80 | 94.54%n=80 | 60.47%n=80 | 57.72%n=80 | +0.76 pp | -1.33 pp | -37.31 pp | -1.95 pp | -36.02 pp | -38.77 pp |
| expertise_level_estimation_of_text | 99.43%n=80 | 89.27%n=80 | 89.82%n=80 | 55.91%n=80 | 91.29%n=80 | 74.21%n=80 | 61.09%n=80 | -10.16 pp | -9.61 pp | -43.52 pp | -8.14 pp | -25.22 pp | -38.34 pp |
| adversarial_agent_reasoning | 96.46%n=80 | 86.39%n=80 | 87.31%n=80 | 75.52%n=80 | 91.55%n=80 | 89.45%n=80 | 36.29%n=80 | -10.07 pp | -9.15 pp | -20.94 pp | -4.91 pp | -7.01 pp | -60.17 pp |
| synthetic_data_plausibility | 96.95%n=80 | 92.35%n=80 | 91.37%n=80 | 57.98%n=80 | 99.30%n=80 | 82.02%n=80 | 43.53%n=80 | -4.60 pp | -5.58 pp | -38.97 pp | +2.35 pp | -14.93 pp | -53.42 pp |
| temporal_graph_consistency | 93.91%n=80 | 95.62%n=80 | 91.92%n=80 | 66.18%n=80 | 74.12%n=80 | 94.91%n=80 | 50.02%n=80 | +1.71 pp | -1.99 pp | -27.73 pp | -19.79 pp | +1.00 pp | -43.89 pp |
| email_intent_hard | 99.20%n=80 | 96.65%n=80 | 96.99%n=80 | 36.82%n=80 | 98.14%n=80 | 99.44%n=80 | 40.37%n=80 | -2.55 pp | -2.21 pp | -62.38 pp | -1.06 pp | +0.24 pp | -58.83 pp |
| domain_identification | 97.10%n=80 | 93.32%n=80 | 93.79%n=80 | 50.87%n=80 | 96.33%n=80 | 98.09%n=80 | 38.85%n=80 | -3.78 pp | -3.31 pp | -46.23 pp | -0.77 pp | +0.99 pp | -58.25 pp |
| constraint_satisfaction_judgement | 96.39%n=80 | 92.84%n=80 | 91.22%n=80 | 70.05%n=80 | 93.48%n=80 | 91.20%n=80 | 33.91%n=80 | -3.55 pp | -5.17 pp | -26.34 pp | -2.91 pp | -5.19 pp | -62.48 pp |
| content_moderation | 98.29%n=80 | 90.13%n=80 | 90.51%n=80 | 59.26%n=80 | 81.73%n=80 | 77.59%n=80 | 72.18%n=80 | -8.16 pp | -7.78 pp | -39.03 pp | -16.56 pp | -20.70 pp | -26.11 pp |
| refund_eligible | 81.79%n=80 | 93.84%n=80 | 91.10%n=80 | 54.43%n=80 | 92.42%n=80 | 77.18%n=80 | 82.49%n=80 | +12.05 pp | +9.31 pp | -27.36 pp | +10.63 pp | -4.61 pp | +0.70 pp |
| medical_chronology_consistency | 98.00%n=80 | 93.00%n=80 | 88.96%n=80 | 68.70%n=80 | 95.91%n=80 | 67.95%n=80 | 61.95%n=80 | -5.00 pp | -9.04 pp | -29.30 pp | -2.09 pp | -30.05 pp | -36.05 pp |
| gaming_report_type | 95.95%n=80 | 92.31%n=80 | 92.89%n=80 | 50.27%n=80 | 80.94%n=80 | 77.03%n=80 | 85.13%n=80 | -3.64 pp | -3.06 pp | -45.68 pp | -15.01 pp | -18.92 pp | -10.82 pp |
| routing_intent_support_priority | 94.11%n=80 | 88.96%n=80 | 89.49%n=80 | 49.35%n=80 | 89.48%n=80 | 90.02%n=80 | 73.15%n=80 | -5.15 pp | -4.62 pp | -44.76 pp | -4.63 pp | -4.09 pp | -20.96 pp |
| return_reason | 98.79%n=80 | 95.81%n=80 | 96.19%n=80 | 44.59%n=80 | 95.51%n=80 | 95.28%n=80 | 48.64%n=80 | -2.98 pp | -2.60 pp | -54.20 pp | -3.28 pp | -3.51 pp | -50.15 pp |
| contains_spoiler | 95.17%n=80 | 93.51%n=80 | 90.96%n=80 | 62.70%n=80 | 91.26%n=80 | 84.30%n=80 | 57.94%n=80 | -1.66 pp | -4.21 pp | -32.47 pp | -3.91 pp | -10.87 pp | -37.23 pp |
| weather_alert_severity | 97.45%n=80 | 95.80%n=80 | 97.45%n=80 | 45.23%n=80 | 94.60%n=80 | 98.91%n=80 | 46.43%n=80 | -1.65 pp | +0.00 pp | -52.22 pp | -2.85 pp | +1.46 pp | -51.02 pp |
| support_department | 93.31%n=80 | 89.47%n=80 | 89.87%n=80 | 45.03%n=80 | 90.73%n=80 | 85.69%n=80 | 81.87%n=80 | -3.84 pp | -3.44 pp | -48.28 pp | -2.58 pp | -7.62 pp | -11.44 pp |
| code_review_intent | 95.27%n=80 | 91.39%n=80 | 91.92%n=80 | 39.31%n=80 | 92.21%n=80 | 95.81%n=80 | 70.35%n=80 | -3.88 pp | -3.35 pp | -55.96 pp | -3.06 pp | +0.54 pp | -24.92 pp |
| pii_sensitive_data_classification | 95.63%n=80 | 95.51%n=80 | 95.99%n=80 | 44.84%n=80 | 94.01%n=80 | 96.88%n=80 | 54.06%n=80 | -0.12 pp | +0.36 pp | -50.79 pp | -1.62 pp | +1.25 pp | -41.57 pp |
| language_dialect_identification | 100.00%n=80 | 94.53%n=80 | 94.97%n=80 | 84.19%n=80 | 96.80%n=80 | 88.47%n=80 | 19.87%n=80 | -5.47 pp | -5.03 pp | -15.81 pp | -3.20 pp | -11.53 pp | -80.13 pp |
| biological_system_reasoning | 88.18%n=80 | 96.24%n=80 | 94.16%n=80 | 62.71%n=80 | 97.64%n=80 | 81.02%n=80 | 59.83%n=80 | +8.06 pp | +5.98 pp | -25.47 pp | +9.46 pp | -7.16 pp | -28.35 pp |
| fair_housing_violation | 90.60%n=80 | 88.30%n=80 | 86.48%n=80 | 60.41%n=80 | 83.83%n=80 | 89.74%n=80 | 80.45%n=80 | -2.30 pp | -4.12 pp | -30.19 pp | -6.77 pp | -0.86 pp | -10.15 pp |
| mood_atmosphere_understanding | 98.96%n=80 | 93.53%n=80 | 94.11%n=80 | 67.08%n=80 | 94.40%n=80 | 99.01%n=80 | 33.13%n=80 | -5.43 pp | -4.85 pp | -31.88 pp | -4.56 pp | +0.05 pp | -65.83 pp |
| evidence_provenance | 100.00%n=80 | 88.59%n=80 | 88.88%n=80 | 77.25%n=80 | 64.16%n=80 | 62.70%n=80 | 99.64%n=80 | -11.41 pp | -11.12 pp | -22.75 pp | -35.84 pp | -37.30 pp | -0.36 pp |
| ocr_ngram_based_text_recognition | 100.00%n=80 | 99.40%n=80 | 99.50%n=80 | 41.03%n=80 | 99.67%n=80 | 100.00%n=80 | 42.03%n=80 | -0.60 pp | -0.50 pp | -58.97 pp | -0.33 pp | +0.00 pp | -57.97 pp |
| conversation_state_tracking | 100.00%n=80 | 98.50%n=80 | 98.74%n=80 | 54.39%n=80 | 92.49%n=80 | 99.03%n=80 | 38.70%n=80 | -1.50 pp | -1.26 pp | -45.61 pp | -7.51 pp | -0.97 pp | -61.30 pp |
| allergen_present | 91.85%n=80 | 94.74%n=80 | 92.67%n=80 | 57.83%n=80 | 92.65%n=80 | 79.30%n=80 | 76.15%n=80 | +2.89 pp | +0.82 pp | -34.02 pp | +0.80 pp | -12.55 pp | -15.70 pp |
| taxonomy_category_assignment | 100.00%n=80 | 97.75%n=80 | 98.05%n=80 | 58.53%n=80 | 97.02%n=80 | 100.00%n=80 | 34.17%n=80 | -2.25 pp | -1.95 pp | -41.47 pp | -2.98 pp | +0.00 pp | -65.83 pp |
| dietary_vegan | 92.99%n=80 | 98.00%n=80 | 96.26%n=80 | 39.71%n=80 | 96.83%n=80 | 87.83%n=80 | 73.92%n=80 | +5.01 pp | +3.27 pp | -53.28 pp | +3.84 pp | -5.16 pp | -19.07 pp |
| soft_vs_hard_constraint_discrimination | 99.74%n=80 | 91.83%n=80 | 92.49%n=80 | 56.52%n=80 | 93.56%n=80 | 93.42%n=80 | 58.58%n=80 | -7.91 pp | -7.25 pp | -43.22 pp | -6.18 pp | -6.32 pp | -41.16 pp |
| tone_register_classification | 98.90%n=80 | 89.37%n=80 | 89.92%n=80 | 60.47%n=80 | 88.83%n=80 | 92.82%n=80 | 66.35%n=80 | -9.53 pp | -8.98 pp | -38.43 pp | -10.07 pp | -6.08 pp | -32.55 pp |
| narrative_next_event_plausibility | 100.00%n=80 | 96.82%n=80 | 97.23%n=80 | 45.06%n=80 | 97.61%n=80 | 100.00%n=80 | 50.65%n=80 | -3.18 pp | -2.77 pp | -54.94 pp | -2.39 pp | +0.00 pp | -49.35 pp |
| evidence_sufficiency | 93.00%n=80 | 99.56%n=80 | 98.50%n=80 | 58.81%n=80 | 99.17%n=80 | 99.97%n=80 | 38.85%n=80 | +6.56 pp | +5.50 pp | -34.19 pp | +6.17 pp | +6.97 pp | -54.15 pp |
| local_rule_over_world_knowledge_judgement | 95.55%n=80 | 99.23%n=80 | 97.71%n=80 | 72.87%n=80 | 98.66%n=80 | 47.94%n=80 | 79.05%n=80 | +3.68 pp | +2.16 pp | -22.68 pp | +3.11 pp | -47.61 pp | -16.50 pp |
| unknown_known_state_classification | 99.46%n=80 | 97.30%n=80 | 97.69%n=80 | 64.26%n=80 | 91.01%n=80 | 96.71%n=80 | 46.39%n=80 | -2.16 pp | -1.77 pp | -35.20 pp | -8.45 pp | -2.75 pp | -53.07 pp |
| relationship_permission_reasoning | 94.89%n=80 | 89.56%n=80 | 87.17%n=80 | 53.56%n=80 | 95.18%n=80 | 87.87%n=80 | 88.37%n=80 | -5.33 pp | -7.72 pp | -41.33 pp | +0.29 pp | -7.02 pp | -6.52 pp |
| consistency_under_option_permutation | 89.54%n=80 | 88.72%n=80 | 84.22%n=80 | 58.03%n=80 | 92.35%n=80 | 93.73%n=80 | 93.43%n=80 | -0.82 pp | -5.32 pp | -31.51 pp | +2.81 pp | +4.19 pp | +3.89 pp |
| document_type_classification | 100.00%n=80 | 91.96%n=80 | 92.35%n=80 | 83.46%n=80 | 90.26%n=80 | 100.00%n=80 | 43.47%n=80 | -8.04 pp | -7.65 pp | -16.54 pp | -9.74 pp | +0.00 pp | -56.53 pp |
| urgency | 95.81%n=80 | 98.40%n=80 | 97.20%n=80 | 84.78%n=80 | 97.18%n=80 | 70.18%n=80 | 58.22%n=80 | +2.59 pp | +1.39 pp | -11.03 pp | +1.37 pp | -25.63 pp | -37.59 pp |
| reading_level | 94.92%n=80 | 95.04%n=80 | 96.60%n=80 | 45.92%n=80 | 93.96%n=80 | 91.52%n=80 | 84.86%n=80 | +0.12 pp | +1.68 pp | -49.00 pp | -0.96 pp | -3.40 pp | -10.06 pp |
| glossary_adherence | 98.28%n=80 | 97.35%n=80 | 95.57%n=80 | 77.30%n=80 | 91.86%n=80 | 81.09%n=80 | 61.88%n=80 | -0.93 pp | -2.71 pp | -20.98 pp | -6.42 pp | -17.19 pp | -36.40 pp |
| scale_plausibility | 94.84%n=80 | 86.04%n=80 | 82.01%n=80 | 73.98%n=80 | 97.06%n=80 | 79.68%n=80 | 90.87%n=80 | -8.80 pp | -12.83 pp | -20.86 pp | +2.22 pp | -15.16 pp | -3.97 pp |
| ad_policy_violation | 92.03%n=80 | 92.62%n=80 | 91.09%n=80 | 76.96%n=80 | 99.03%n=80 | 98.81%n=80 | 54.20%n=80 | +0.59 pp | -0.94 pp | -15.07 pp | +7.00 pp | +6.78 pp | -37.83 pp |
| recipe_cuisine | 99.82%n=80 | 97.76%n=80 | 97.93%n=80 | 41.64%n=80 | 98.39%n=80 | 94.87%n=80 | 79.05%n=80 | -2.06 pp | -1.89 pp | -58.18 pp | -1.43 pp | -4.95 pp | -20.77 pp |
| log_event_sequence_diagnosis | 93.49%n=80 | 95.17%n=80 | 95.78%n=80 | 72.21%n=80 | 95.70%n=80 | 99.89%n=80 | 57.31%n=80 | +1.68 pp | +2.29 pp | -21.28 pp | +2.21 pp | +6.40 pp | -36.18 pp |
| definition_adherence | 96.98%n=80 | 99.85%n=80 | 99.32%n=80 | 49.95%n=80 | 99.35%n=80 | 88.25%n=80 | 77.66%n=80 | +2.87 pp | +2.34 pp | -47.03 pp | +2.37 pp | -8.73 pp | -19.32 pp |
| hazmat_shipping | 90.88%n=80 | 99.49%n=80 | 98.60%n=80 | 64.39%n=80 | 94.51%n=80 | 99.61%n=80 | 64.29%n=80 | +8.61 pp | +7.72 pp | -26.49 pp | +3.63 pp | +8.73 pp | -26.59 pp |
| emotion_cause_attribution | 100.00%n=80 | 99.30%n=80 | 99.41%n=80 | 64.25%n=80 | 95.55%n=80 | 99.94%n=80 | 53.50%n=80 | -0.70 pp | -0.59 pp | -35.75 pp | -4.45 pp | -0.06 pp | -46.50 pp |
| contains_pii | 96.44%n=80 | 96.87%n=80 | 94.51%n=80 | 71.16%n=80 | 93.77%n=80 | 92.68%n=80 | 71.02%n=80 | +0.43 pp | -1.93 pp | -25.28 pp | -2.67 pp | -3.76 pp | -25.42 pp |
| question_duplicate | 94.03%n=80 | 98.63%n=80 | 96.85%n=80 | 63.10%n=80 | 80.92%n=80 | 95.18%n=80 | 91.03%n=80 | +4.60 pp | +2.82 pp | -30.93 pp | -13.11 pp | +1.15 pp | -3.00 pp |
| category_error_detection | 99.80%n=80 | 95.07%n=80 | 95.57%n=80 | 87.04%n=80 | 93.49%n=80 | 84.89%n=80 | 65.16%n=80 | -4.73 pp | -4.23 pp | -12.76 pp | -6.31 pp | -14.91 pp | -34.64 pp |
| topic_keyword_fidelity | 100.00%n=80 | 92.26%n=80 | 92.92%n=80 | 62.18%n=80 | 94.80%n=80 | 100.00%n=80 | 79.01%n=80 | -7.74 pp | -7.08 pp | -37.82 pp | -5.20 pp | +0.00 pp | -20.99 pp |
| character_goal_consistency | 95.49%n=80 | 99.09%n=80 | 97.60%n=80 | 70.78%n=80 | 98.99%n=80 | 77.23%n=80 | 84.70%n=80 | +3.60 pp | +2.11 pp | -24.71 pp | +3.50 pp | -18.26 pp | -10.79 pp |
| minimal_sufficient_explanation_selection | 100.00%n=80 | 94.59%n=80 | 95.18%n=80 | 49.00%n=80 | 97.99%n=80 | 100.00%n=80 | 87.19%n=80 | -5.41 pp | -4.82 pp | -51.00 pp | -2.01 pp | +0.00 pp | -12.81 pp |
| set_coverage_diversity_judgement | 100.00%n=80 | 97.58%n=80 | 97.87%n=80 | 40.62%n=80 | 98.56%n=80 | 100.00%n=80 | 93.98%n=80 | -2.42 pp | -2.13 pp | -59.38 pp | -1.44 pp | +0.00 pp | -6.02 pp |
| home_service_routing | 99.84%n=80 | 98.20%n=80 | 98.44%n=80 | 46.04%n=80 | 99.42%n=80 | 93.87%n=80 | 94.54%n=80 | -1.64 pp | -1.40 pp | -53.80 pp | -0.42 pp | -5.97 pp | -5.30 pp |
| churn_risk | 97.90%n=80 | 93.76%n=80 | 95.94%n=80 | 65.20%n=80 | 94.11%n=80 | 89.85%n=80 | 95.13%n=80 | -4.14 pp | -1.96 pp | -32.70 pp | -3.79 pp | -8.05 pp | -2.77 pp |
| humor_explanation_correctness | 100.00%n=80 | 86.33%n=80 | 87.04%n=80 | 71.12%n=80 | 93.23%n=80 | 97.11%n=80 | 98.61%n=80 | -13.67 pp | -12.96 pp | -28.88 pp | -6.77 pp | -2.89 pp | -1.39 pp |
| historical_fact_verification | 96.02%n=80 | 99.20%n=80 | 97.73%n=80 | 55.94%n=80 | 97.75%n=80 | 94.72%n=80 | 97.35%n=80 | +3.18 pp | +1.71 pp | -40.08 pp | +1.73 pp | -1.30 pp | +1.33 pp |
| voice_assistant_intent | 99.45%n=80 | 98.44%n=80 | 98.68%n=80 | 54.93%n=80 | 98.34%n=80 | 96.54%n=80 | 92.70%n=80 | -1.01 pp | -0.77 pp | -44.52 pp | -1.11 pp | -2.91 pp | -6.75 pp |
| city_service_request | 99.70%n=80 | 97.99%n=80 | 98.24%n=80 | 53.50%n=80 | 98.32%n=80 | 92.67%n=80 | 98.71%n=80 | -1.71 pp | -1.46 pp | -46.20 pp | -1.38 pp | -7.03 pp | -0.99 pp |
| oncall_route | 99.15%n=80 | 98.03%n=80 | 98.25%n=80 | 52.55%n=80 | 97.83%n=80 | 98.75%n=80 | 94.76%n=80 | -1.12 pp | -0.90 pp | -46.60 pp | -1.32 pp | -0.40 pp | -4.39 pp |
| multi_label_topic_classification | 95.87%n=80 | 99.37%n=80 | 98.21%n=80 | 55.38%n=80 | 99.38%n=80 | 99.95%n=80 | 99.95%n=80 | +3.50 pp | +2.34 pp | -40.49 pp | +3.51 pp | +4.08 pp | +4.08 pp |
| expense_category | 99.31%n=80 | 97.62%n=80 | 97.84%n=80 | 59.80%n=80 | 98.21%n=80 | 99.91%n=80 | 95.89%n=80 | -1.69 pp | -1.47 pp | -39.51 pp | -1.10 pp | +0.60 pp | -3.42 pp |
| app_review_intent | 99.80%n=80 | 91.58%n=80 | 91.64%n=80 | 76.64%n=80 | 95.93%n=80 | 99.75%n=80 | 98.92%n=80 | -8.22 pp | -8.16 pp | -23.16 pp | -3.87 pp | -0.05 pp | -0.88 pp |
| email_intent | 99.80%n=80 | 98.74%n=80 | 98.93%n=80 | 63.64%n=80 | 99.11%n=80 | 98.65%n=80 | 97.21%n=80 | -1.06 pp | -0.87 pp | -36.16 pp | -0.69 pp | -1.15 pp | -2.59 pp |
| news_topic | 99.81%n=80 | 99.28%n=80 | 99.40%n=80 | 69.78%n=80 | 97.86%n=80 | 91.67%n=80 | 98.73%n=80 | -0.53 pp | -0.41 pp | -30.03 pp | -1.95 pp | -8.14 pp | -1.08 pp |
| source_reliability_in_context | 100.00%n=80 | 97.86%n=80 | 98.14%n=80 | 82.84%n=80 | 98.90%n=80 | 99.87%n=80 | 79.80%n=80 | -2.14 pp | -1.86 pp | -17.16 pp | -1.10 pp | -0.13 pp | -20.20 pp |
| word_sense_disambiguation | 99.99%n=80 | 99.29%n=80 | 99.40%n=80 | 88.61%n=80 | 99.19%n=80 | 100.00%n=80 | 79.68%n=80 | -0.70 pp | -0.59 pp | -11.38 pp | -0.80 pp | +0.01 pp | -20.31 pp |
| delivery_exception | 99.84%n=80 | 97.24%n=80 | 97.57%n=80 | 80.89%n=80 | 99.09%n=80 | 100.00%n=80 | 92.08%n=80 | -2.60 pp | -2.27 pp | -18.95 pp | -0.75 pp | +0.16 pp | -7.76 pp |
| medical_differential_reasoning_benchmarks | 99.80%n=80 | 97.51%n=80 | 97.84%n=80 | 82.49%n=80 | 98.76%n=80 | 100.00%n=80 | 92.02%n=80 | -2.29 pp | -1.96 pp | -17.31 pp | -1.04 pp | +0.20 pp | -7.78 pp |
| document_type | 100.00%n=80 | 99.45%n=80 | 99.56%n=80 | 93.28%n=80 | 99.59%n=80 | 100.00%n=80 | 78.22%n=80 | -0.55 pp | -0.44 pp | -6.72 pp | -0.41 pp | +0.00 pp | -21.78 pp |
| contract_clause_type | 99.94%n=80 | 99.34%n=80 | 99.44%n=80 | 71.80%n=80 | 99.92%n=80 | 100.00%n=80 | 99.85%n=80 | -0.60 pp | -0.50 pp | -28.14 pp | -0.02 pp | +0.06 pp | -0.09 pp |
| graph_knowledge | 100.00%n=80 | 99.55%n=80 | 99.64%n=80 | 99.94%n=80 | 99.63%n=80 | 100.00%n=80 | 84.00%n=80 | -0.45 pp | -0.36 pp | -0.06 pp | -0.37 pp | +0.00 pp | -16.00 pp |