Scale AI's Humanity's Sixth Sense benchmark: humans score 93.1% on intuitive visual reasoning, the best model, GPT-6 Astra, 53.6%

Why it matters

Key points

Scale AI's research arm Scale Labs and the startup Elorian released a new benchmark, Humanity's Sixth Sense (HSS), in the early hours of October 8 Beijing time. It tests intuitive visual reasoning: looking at an image or short video and inferring what the scene does not show outright. Can two more books fit in the gaps on a shelf? Which side of the frame did a traveller enter from? Are the chains on a plough taut or slack? Are the people next to an aircraft crew or visitors?

The gap is wide. Twenty human annotators, answering under the same free-form conditions as the models, scored 93.1%. The best of 25 models, GPT-6 Astra at maximum reasoning effort, scored 53.6%; the median model scored 30.9%, and most were below 40%. Next came GPT-6.1 Sol at 46.6%, Claude Opus 5.5 at 44.6% and Gemini 3.8 Flash at 41.6%. The highest Chinese model was Qwen3.8-Max at 33.0% in ninth place; GLM-5.3-Flash scored 26.8%, Kimi K3 25.5% and Xiaomi's MiMo v2.6 about 25%.

A few details stand out. On tasks people answer at a glance, models spent about 4,000 reasoning tokens per task on average and often still got them wrong. Video was harder than still images for 23 of the 25 models, by 7.3 points on average. Social understanding (who defers to whom, what the people in a scene know, unwritten rules) was the weakest domain for 21 of the 25 models, averaging 24.4% against 34.1% for the other three domains.

Verifiable

Scale Labs blog post, dated October 7, 2026 (US time): https://labs.scale.com/blog/introducing-humanitys-sixth-sense . Leaderboard and methodology: https://labs.scale.com/leaderboard/hss . Paper page: https://labs.scale.com/papers/humanitys-sixth-sense . The dataset is public on Hugging Face: https://huggingface.co/datasets/ScaleAI/HSS . Scale Labs announcement post: https://x.com/ScaleAILabs/status/2107920376756117903 .

Method: 522 open-ended tasks, 288 on images and 234 on video (17.6 hours in total), across 4 domains and 11 subdomains with 723 rubric criteria. Of 3,466 tasks written, 15.1% survived three independent review rounds. Each model answers each task three times and the pass rate is averaged; the plus-or-minus figure is the half-width of a 95% confidence interval. Answers are graded by Claude Opus 5 as an automated judge. The stated limitations include video supplied without audio or transcripts, possible judge bias such as favouring its own vendor's models, and the chance that some publicly sourced images or clips appear in training data.

Our take

What makes this benchmark worth following is that it is nowhere near saturated. On most popular benchmarks frontier models are at or above human level; here roughly 40 points still separate people from the best model. For AI agents moving into homes, cars and offices, judging whether something will fit, whether a person has authority, or what happens next is a daily requirement, not a bonus.

The 4,000-token average says the problem is not too little thinking. Models are using long written reasoning on problems meant to be solved with a glance, and the longer they reason, the easier it is to drift away from the cues actually in the scene. Our view is that the gap is more likely to close through training on more video and real-world data and through better visual representations than through bigger reasoning budgets. Chinese models trail the Western flagships by a clear margin on this board, which suggests multimodal intuition is one of the areas where their gap to the top tier is still large.

Read the leaderboard with some care. Scale's core business is supplying human-labelled data to model developers, so the conclusion that models lack human intuition and need more human-written data suits its commercial interests. The judge is an Anthropic model while Claude ranks near the top, so self-preference is possible; Scale says it checked agreement across judges, but the figures are in the paper. The confidence intervals for second through fourth place overlap, so their order should not be over-read. What to watch next: whether other labs and independent evaluators can reproduce these scores from the public dataset, and whether the next generation of models narrows the gap on video and social understanding.