HumanEval
Observability & Evals · Free + paid
OpenAI's benchmark for evaluating code generation
At a glance
- Made by
- OpenAI
- Available on
- CLI
- Open source
- Public code (license name is not in the source text)
- GitHub
- 3,402 stars · MIT
What is HumanEval?
HumanEval is a hand-written evaluation set, and this repository provides an evaluation harness for the HumanEval problem dataset. When users save model-generated code completions in JSON Lines format, the evaluate_functional_correctness command runs the tests and computes metrics such as pass@1, pass@10, and pass@100. Each result records whether it passed, timed out, or failed. Because it executes untrusted model-generated code, it strongly advises against using it outside a security sandbox.
- Hand-written set of code generation evaluation problems
- Generated results input in JSON Lines format
- Functional correctness evaluation with the evaluate_functional_correctness command
- pass@k metric calculation (k values configurable)
- Per-result record of pass, timeout, or failure
Who uses HumanEval and for what
- Evaluating the code generation ability of language models trained on code
Pricing
Open source (free)
GitHub에 공개된 벤치마크 데이터셋·평가 하네스 코드뿐이고 유료 플랜·관리형 클라우드 언급 없음
공식 페이지 확인 2026-10-14(산출 규칙 2026-10-15)
AI tools similar to HumanEval (alternatives)
HumanEval vs. alternatives at a glance
| AI | Pricing | Combos on the map | Connected tools | Free | Cheapest monthly | Korean |
|---|---|---|---|---|---|---|
| HumanEval | Free + paid | 0 | 7 | — | — | — |
| Braintrust | Free + paid | 0 | 7 | Yes | $249/mo | No |
| DeepEval | Free + paid | 10 | 7 | — | — | No |
| EleutherAI LM Evaluation Harness | Free + paid | 0 | 7 | — | — | — |
| LangSmith | Free + paid | 10 | 5 | — | — | — |
1 of these 5 tools have a free plan. Cheapest paid plan: Braintrust ($249/mo).
Combo and connection counts are based on the relationships recorded on AI Galaxy Map (2026-10-26); they are not a quality ranking.
HumanEval on AI Galaxy Map
- One of 27 AI tools in the Observability & Evals field.
- Connected to 7 other tools — #1 in its field.
Frequently asked questions
Can I use HumanEval for free?
Open source (free). Prices can change, so check the official site before paying.
What AI can I use instead of HumanEval?
Tools that do the same job, such as Braintrust, DeepEval, EleutherAI LM Evaluation Harness, LangSmith, are connected to it on the map.
Sources (official pages)
Checked on the official pages on 2026-10-14. Prices and features can change, so check the official site again before paying.