Harvey LAB-AA v1.1 adds a hallucination gate to its legal agent benchmark: leader Grok 4.7 scores 9.4%, and over 60% of otherwise passing work contains a material error

Why it matters

【Key points】

On Oct 8, Artificial Analysis and legal AI company Harvey released Harvey LAB-AA v1.1. Across 120 private legal tasks in 24 practice areas, every deliverable is now checked against its source documents, and any material hallucination zeroes the task.

On the new headline metric Grok 4.7 leads at 9.4%, ahead of Muse Spark 1.3 (8.9%), GPT-6 Astra (8.6%), GPT-6.1 Sol (6.9%), Claude Fable 5.1 (6.4%), Kimi K3 (5.3%) and Claude Opus 5.5 (4.2%). Without the gate Muse Spark 1.3 would lead at 26.7%, but two thirds of those passes contain a material hallucination. Over 60% of otherwise passing results across models do. GPT-6 Astra had 4 across 120 tasks; Gemini 3.8 Flash averaged 13.96 per task; every open-weights model averaged at least 2.09.

【Verifiable】

Announcement: https://artificialanalysis.ai/articles/harvey-lab-aa-v1-1

Leaderboard: https://artificialanalysis.ai/evaluations/harvey-lab-aa

Post: https://x.com/ArtificialAnlys/status/2108264572545310824

Rubrics are graded by GPT-6 Sol, Grok 4.7 and Claude Opus 5.5; GPT-6 Sol runs the hallucination check. Scores are not comparable with v1.0.

【Our read】

1. The headline is that the best score is 9.4%. Most models clear 85%+ of rubric criteria, yet fewer than one in ten deliverables are both complete and free of material errors. Human review is still mandatory.

2. Gating on hallucinations reshuffles the ranking: completeness and reliability are different skills, and completeness-only boards overrate models that write a lot and invent details.

3. Treat absolute numbers with care: checkers found 57 to 470 material errors on the same work, and the strictest was chosen. GPT-6 Astra scored zero under all six, so its result is not checker bias.