# Humanity's Last Exam Benchmark: How “Humanity’s Last Exam” Pushes GPT-4-Level AI Beyond MMLU

Canonical URL: https://www.adaline.ai/blog/humanitys-last-exam-ai-benchmark
LLM text URL: https://www.adaline.ai/blog/humanitys-last-exam-ai-benchmark/llms.txt
Published: 2025-05-08T00:00:00.000Z
Modified: 2025-05-08T17:01:31.864Z
Author: Nilesh Barla
Category: Research
Visibility: public
Reading time: 15 min
Topics: Research, Adaline, AI agent observability, agent evals, self-improving agents

## Summary

Brief notes on the multi-modal HLE benchmark built to test frontier models after traditional AI leaderboards hit saturation.

## Article

# Benchmark Saturation & the Need for a “Last Exam”

TL;DR: Humanity's Last Exam addresses benchmark saturation where top AI models score >90% on traditional tests, making progress measurement impossible. Created by experts across 100+ domains, HLE questions stump even the best models (scoring <30%), revealing true AI limitations. Its innovative design includes anti-gaming features, multi-modal testing, and calibration measurement, providing a more realistic assessment of AI capabilities for product teams and policymakers.

## The Current AI Evaluation Crisis

The AI community faces a serious problem. Our best tools for measuring language model capabilities have reached their limits. Leading[ ](https://scale.com/blog/humanitys-last-exam-results)[benchmark saturation](https://scale.com/blog/humanitys-last-exam-results) occurs when top models consistently score above 90% on standardized tests.

This saturation creates a false ceiling that masks actual progress. When GPT-4 scored over 90% on MMLU (Massive Multitask Language Understanding), the benchmark effectively lost its ability to distinguish meaningful improvements in newer models.

```csv
Benchmark	GPT-4 Evaluated few-shot	GPT-3.5 Evaluated few-shot	LM SOTA Best external LM evaluated few-shot	SOTA Best external model (includes benchmark-specific training)
MMLU Multiple-choice questions in 57 subjects (professional & academic)	86.4% 5-shot	70.0% 5-shot	70.7% 5-shot U-PaLM	75.2% 5-shot Flan-PaLM
HellaSwag Commonsense reasoning around everyday events	95.3% 10-shot	85.5% 10-shot	84.2% LLAMA (validation set)	85.6% ALUM
AI2 Reasoning Challenge (ARC) Grade-school multiple choice science questions. Challenge-set.	96.3% 25-shot	85.2% 25-shot	84.2% 8-shot PaLM	85.6% ST-MOE
WinoGrande Commonsense reasoning around pronoun resolution	87.5% 5-shot	81.6% 5-shot	84.2% 5-shot PALM	85.6% 5-shot PALM
HumanEval Python coding tasks	67.0% 0-shot	48.1% 0-shot	26.2% 0-shot PaLM	65.8% CodeT + GPT-3.5
DROP (f1 score) Reading comprehension & arithmetic.	80.9 3-shot	64.1 3-shot	70.8 1-shot PaLM	88.4
```

**Source**:[ GPT-4](https://openai.com/index/gpt-4-research/)

The pattern repeats across multiple evaluation standards:

- MMLU exceeded 90% accuracy with the latest models
- MATH dataset quickly progressed from <10% to >90% in just three years
- GSM8K showing similar saturation patterns

This rapid saturation limits our understanding of AI capabilities at the development frontier.

### Why Traditional Benchmarks Fall Short

Traditional benchmarks fail to reveal crucial differences between top-performing models. This leads to three significant problems:

1. [Measurement limitations] When models max out scores, we cannot accurately measure improvement
2. [False confidence] Teams develop unrealistic expectations about model capabilities
3. [Hidden weaknesses] Critical gaps in knowledge remain undetected despite high scores

The[ ](https://www.zdnet.com/article/reasoning-ai-models-are-overwhelming-the-benchmark-tests-its-time-for-human-evaluation/)[post-MMLU era](https://www.zdnet.com/article/reasoning-ai-models-are-overwhelming-the-benchmark-tests-its-time-for-human-evaluation/) demands new evaluation approaches. Research teams at Scale AI and the Center for AI Safety recognized this evaluation crisis when developing Humanity’s Last Exam.

## The Need for a Final Academic Benchmark

Product teams face significant risks when relying on saturated benchmarks. High scores on existing tests create a misleading impression of model capabilities and reliability. These scores rarely translate to real-world performance.

> When I released the MATH benchmark in 2021, the best model scored less than 10%; few predicted that scores higher than 90% would be achieved just three years later.
> -- Dan Hendrycks, CAIS co-founder, highlights the speed of benchmark obsolescence.

This rapid progression underscores why we need increasingly complex standards. Humanity’s Last Exam represents an attempt to create the final closed-ended academic benchmark with sufficient headroom to measure progress for years to come accurately.

The exam focuses on questions that specifically test the limits of AI knowledge at the frontiers of human expertise. We can truly understand their capabilities and limitations only by pushing models to their breaking points.

# Origins of Humanity’s Last Exam (HLE)

## 

Image: https://a-us.storyblok.com/f/1023026/1011x710/fe14e60a01/a-sample-of-hle-dataset.png

_A sample of HLE dataset_** | Source**: [Humanity’s Last Exam](https://static.scale.com/uploads/654197dc94d34f66c0f5184e/Publication%20Ready%20Humanity's%20Last%20Exam.pdf)

HLE emerged as a joint initiative between[ ](https://scale.com/blog/humanitys-last-exam-results)[Scale AI](https://scale.com/blog/humanitys-last-exam-results) and the Center for AI Safety (CAIS). The project represents an ambitious effort to create what its founders call the "final closed-ended academic benchmark with broad subject coverage."

The benchmark was developed under the leadership of Dan Hendrycks, CAIS co-founder and executive director, and Alexandr Wang from Scale AI. Their vision was to create an evaluation that would remain challenging even as AI systems continue to advance rapidly.

### Creating the Ultimate Challenge

Image: https://a-us.storyblok.com/f/1023026/1360x444/3c88dcaf06/the-development-workflow-diagram-of-hle.png

_The development workflow diagram of HLE _**| Source**: [Humanity’s Last Exam](https://static.scale.com/uploads/654197dc94d34f66c0f5184e/Publication%20Ready%20Humanity's%20Last%20Exam.pdf)

The development process for this[ ](https://lastexam.ai/)[crowdsourced AI benchmark](https://lastexam.ai/) followed a rigorous path:

1. Global expert recruitment (nearly 1,000 contributors)
2. Collection of 70,000+ initial question submissions
3. Selection of 13,000 questions for human expert review
4. Final curation to 3,000 questions for public release

> What might be the ultimate test, meticulously distilled and designed to challenge the world's most advanced models at the frontiers of intelligence.
> -- Summer Yue, Director of Research at Scale AI, described the effort as creating.

## The Expert Network Behind HLE

The benchmark draws its strength from its diverse contributor base:

```csv
Contributor Stats	Numbers
Total contributors	Nearly 1,000
Academic institutions	500+
Countries represented	50+
Contributor backgrounds	Professors, researchers, PhD holders
```

To incentivize high-quality submissions, the project established a $500,000 prize pool. This included:

- $5,000 USD for each of the top 50 questions
- $500 USD for each of the next 500 best submissions

The substantial prizes attracted world-class expertise across domains. Contributors also received co-authorship opportunities on the final research paper.

## A Frontier Knowledge Test

What sets HLE apart is its focus on questions at the absolute edge of human knowledge. The questions intentionally push beyond standard academic tests into territory where even specialized experts might struggle.

The exam covers multiple formats, including text-only and multi-modal challenges with images and diagrams. Each question underwent testing against frontier AI models to ensure it truly represents a challenging evaluation standard.

> By bringing together elite global expertise, HLE aims to measure how close AI systems are to matching specialist human knowledge across diverse academic fields.

# Inside the HLE Blueprint: Multi-Modal, Expert-Level, Closed-Ended

## Anatomy of an Ultimate Challenge

Humanity’s Last Exam contains approximately 3000 questions spanning over 100 subject areas. The benchmark’s architecture follows a carefully designed blueprint to ensure maximum effectiveness as a[ ](https://lastexam.ai/)[multi-modal benchmark](https://lastexam.ai/).

The question formats include:

- 76% short-answer (exact-match) questions
- 24% multiple-choice questions
- 10-14% vision-based tasks requiring image analysis

This diversity ensures the benchmark tests both reasoning capabilities and visual understanding across academic domains.

### Question Distribution by Field

Image: https://a-us.storyblok.com/f/1023026/1350x624/5557cbfe7e/question-distribution-of-hle.png

_Question distribution of HLE_** | Source**: [Humanity’s Last Exam](https://static.scale.com/uploads/654197dc94d34f66c0f5184e/Publication%20Ready%20Humanity's%20Last%20Exam.pdf)

The exam covers an extensive range of academic disciplines:

```csv
Subject Category,Example Fields
Mathematics,"Pure math, Applied math, Statistics"
Sciences,"Physics, Chemistry, Biology, Medicine"
Humanities,"History, Philosophy, Linguistics"
Engineering,"Computer Science, Electrical, Civil"
Arts,"Musicology, Art History, Classical Ballet"
Other,"Chess, Trivia, Law"
```

Each subject features questions at a graduate or expert level, requiring deep domain knowledge to solve.

## Preventing Memorization with Private Test Sets

A key innovation in HLE's design is its two-part structure:

- Public development set for research and model improvement
- Private[ ](https://lastexam.ai/)[held-out evaluation set](https://lastexam.ai/) to prevent gaming

The private test subset allows ongoing assessment of model overfitting. When models perform significantly better on public questions than on similar private ones, it reveals potential memorization rather than true understanding.

Each question in the final dataset underwent confirmation as:

- Precise and unambiguous
- Solvable with domain knowledge
- Non-searchable via simple internet queries
- Original or representing non-trivial synthesis of published information

This meticulous attention to quality makes HLE uniquely positioned to serve as a durable benchmark for the most advanced AI systems, potentially for years to come, despite rapid model improvements.

# Leaderboard Deep-Dive: GPT-4o, Gemini 2.5, o3 vs Humans

## Early Results Show Significant AI Knowledge Gaps

The initial[ ](https://lastexam.ai/)[SEAL leaderboard](https://lastexam.ai/) results for Humanity's Last Exam reveal striking limitations in current AI systems. Even the most advanced models struggle with expert-level questions, achieving surprisingly low scores:

```csv
Model	Accuracy (%) ↑	Calibration Error (%) ↓
o3	20.3	34
Gemini 2.5 Pro	18.4	71
o4-mini	18.1	57
o3-mini*	13.4	80
Gemini 2.5 Flash	12.1	80
DeepSeek-R1*	8.5	73
Claude 3.7 Sonnet (16K)	8	80
o1	8	83
GPT-4.5 Preview	5.4	85
GPT-4.1	5.4	89
Claude 3.5 Sonnet	4.1	84
GPT-4o	2.7	89
```

These single-digit scores contrast sharply with the >90% accuracy these same models achieve on other benchmarks like MMLU. This confirms HLE's effectiveness at testing the frontiers of AI capabilities.

### Rapid Progress Through Specialized Techniques

Image: https://a-us.storyblok.com/f/1023026/776x916/8a446b216e/performance-graph-of-openai-models-on-hle.png

_Performance graph of OpenAI models on HLE _**| Source**: [Introducing OpenAI o3 and o4-mini](https://openai.com/index/introducing-o3-and-o4-mini/)

Despite initial low scores, improvement has happened quickly. By April 2025, newer models showed notable progress:

1. o3-high: 20.3% accuracy
2. OpenAI "Deep Research": 26.6% accuracy

This rapid climb suggests two important insights:

- Specialized research techniques can overcome initial limitations
- The ceiling for improvement remains far from human expert performance

## The Calibration Problem

Beyond raw accuracy,[ ](https://lastexam.ai/)[AI calibration error](https://lastexam.ai/) presents a serious concern. The RMS calibration error measures how well models assess their own confidence. Perfect calibration means a model claiming 70% confidence should be correct 70% of the time.

Current models show poor calibration on HLE:

- RMS errors between 70-90% across all tested systems
- Models frequently express high confidence in incorrect answers
- This pattern reveals a fundamental reliability problem

```python
# Simplified calibration error calculation

RMS_Error = sqrt(mean((confidence - correctness)²))
```

This unreliable self-assessment creates serious trust issues. When models confidently provide incorrect answers (hallucinate), users face greater risks than when models acknowledge uncertainty.

### The Reasoning Token Cost

Image: https://a-us.storyblok.com/f/1023026/1368x434/8a534c3b61/plot-showing-average-completion-tokens-of-reasoning-models.png

_Plot showing average completion tokens of reasoning models_** | Source**: [Humanity’s Last Exam](https://static.scale.com/uploads/654197dc94d34f66c0f5184e/Publication%20Ready%20Humanity's%20Last%20Exam.pdf)

Models with "reasoning" capabilities use significantly more computation. The token count comparison reveals this trade-off:

- **Standard models:** 200-800 tokens per response
- **Reasoning models:** 2,000-8,000 tokens per response

This 10x increase in computational resources yields only 2-3x improvement in[ ](https://www.zdnet.com/article/reasoning-ai-models-are-overwhelming-the-benchmark-tests-its-time-for-human-evaluation/)[GPT-4 HLE score](https://www.zdnet.com/article/reasoning-ai-models-are-overwhelming-the-benchmark-tests-its-time-for-human-evaluation/). This efficiency gap highlights the immense computational costs of approaching human-level expertise.

While progress continues, the benchmark confirms that a substantial gap remains between even the most advanced AI systems and human expert knowledge across academic domains.

# Why HLE Matters for AI Safety & Policy

## Revealing True AI Limitations

Humanity's Last Exam serves a crucial function beyond standard benchmarking. It exposes fundamental gaps in frontier AI systems that might otherwise remain hidden. These revelations have direct implications for[ ](https://lastexam.ai/)[AI safety metrics](https://lastexam.ai/) and governance approaches.

The benchmark's results highlight two critical safety concerns:

1. [Reasoning limitations] Even top models achieve <30% accuracy on expert questions
2. [Overconfidence problem] Models express high confidence in wrong answers

This combination of poor performance and poor calibration creates significant risk. AI systems deployed in critical domains may make mistakes without appropriate caution signals to users.

### Informing Research Priorities

HLE's detailed breakdowns guide alignment researchers toward specific improvement areas:

```csv
Problem Area	HLE Finding	Research Direction
Calibration	80-90% RMS error	Uncertainty quantification
Multi-step reasoning	Low accuracy on complex tasks	Chain-of-thought improvements
Domain expertise	Uneven performance across fields	Specialized training approaches
Robustness	Sensitivity to question format	Consistency enforcement
```

By targeting these weaknesses, researchers can develop more reliable AI systems that acknowledge their limitations appropriately.

## Implications for AI Governance

Policymakers must consider HLE results when developing[ ](https://scale.com/blog/humanitys-last-exam-results)[AGI policy benchmarks](https://scale.com/blog/humanitys-last-exam-results). The exam offers a dual narrative with significant policy implications:

### Progress Signal

The rapid improvement from initial scores (3-5%) to more recent results (20-26%) demonstrates:

- Frontier models advance quickly
- Research tactics can overcome initial limitations
- The ceiling for performance remains undefined

### Warning Signal

Despite progress, critical gaps remain:

- Expert-level knowledge still exceeds current AI capabilities
- Self-assessment abilities lag behind raw performance
- Computational costs for improvements grow exponentially

This nuanced picture helps regulators balance innovation with caution. HLE provides concrete evidence that advanced systems still have significant limitations, especially in specialized knowledge domains.

For safety researchers, the benchmark offers a standardized way to measure both progress and remaining challenges. This shared reference point enables more productive discussions about risk assessment and appropriate guardrails for increasingly capable systems.

As models continue improving on HLE, the benchmark will help track the narrowing gap between AI capabilities and human expertise across academic disciplines.

# The Benchmark Lifecycle: What Comes After HLE?

## Predicting HLE's Useful Lifespan

How long will Humanity's Last Exam remain challenging? The[ ](https://lastexam.ai/)[benchmark lifecycle](https://lastexam.ai/) of previous tests suggests caution in making predictions. Dan Hendrycks notes that the MATH benchmark moved from <10% to >90% accuracy in just three years—far faster than experts anticipated.

For HLE, competing predictions exist:

```csv
Timeframe	Forecast
End of 2025	>50% accuracy possible
2-3 years	Near-saturation (>80%)
5+ years	Continued relevance due to private test set
```

The private held-out test set provides crucial protection against overfitting. This design element may extend HLE's useful life significantly compared to previous benchmarks.

## Beyond Traditional Question-Answer Tests

When HLE eventually saturates, what comes next? Several emerging evaluation approaches may take its place:

1. [AI-generated evaluations] Tests that evolve alongside model capabilities
2. [Interactive assessment] Dynamic evaluations that adapt based on responses
3. [Real-world agent evaluations] Testing AI behavior in simulated environments

Researchers increasingly recognize that[ ](https://www.zdnet.com/article/reasoning-ai-models-are-overwhelming-the-benchmark-tests-its-time-for-human-evaluation/)[benchmarking approaches](https://www.zdnet.com/article/reasoning-ai-models-are-overwhelming-the-benchmark-tests-its-time-for-human-evaluation/) must evolve alongside AI capabilities. Static benchmarks have inherent limitations regardless of difficulty.

> HLE may be the last academic exam we need to give to models, but it is far from the last benchmark for AI.
> -- The HLE creators themselves note

The future of AI evaluation will likely shift toward human-in-the-loop assessment methods. These approaches combine automated testing with human judgment to create more nuanced and adaptable evaluation standards. This hybrid approach may prove more resistant to saturation than even the most challenging fixed benchmarks.

# Conclusion

Humanity's Last Exam (HLE) represents a crucial innovation in AI evaluation. This benchmark contains approximately 3,000 expert-crafted questions across 100+ academic domains, specifically designed to challenge even the most advanced AI systems.

HLE stands apart through several key features:

- Multi-modal testing (text and image-based questions)
- Graduate-level difficulty requiring specialized knowledge
- Public and private evaluation sets to prevent gaming
- Rigorous development by nearly 1,000 global experts

Current leading models achieve only 2-20% accuracy on HLE—a stark contrast to their >90% performance on previous benchmarks. This performance gap reveals important limitations in frontier AI systems.

## Why Product Leaders Should Pay Attention

For product leaders working with AI technologies, HLE offers valuable insights:

1. [Realistic capability assessment] HLE provides an honest picture of what today's most advanced models can and cannot do, helping set appropriate expectations
2. [Risk identification] The poor calibration scores (70-90% error) highlight reliability issues that could affect product trustworthiness
3. [Development roadmap guidance] The benchmark identifies specific areas where improvements would yield the greatest benefits
4. [Safety considerations] HLE helps teams understand where human oversight remains necessary in AI-augmented workflows

As AI capabilities advance, HLE will serve as a consistent reference point for measuring progress. Product teams can use HLE scores to make informed decisions about which technologies to adopt and how to implement them responsibly. By understanding both the rapid improvements and persistent limitations revealed by this benchmark, leaders can build more realistic roadmaps for AI integration in their products.