EleutherAI LM Evaluation Harness
Observability & Evals · Free + paid
Framework for few-shot language model evaluation
At a glance
- Made by
- EleutherAI
- Available on
- CLI
- Open source
- Open source (license name not in the provided text)
- GitHub
- 14,213 stars · MIT
What is EleutherAI LM Evaluation Harness?
Language Model Evaluation Harness is a unified framework for testing generative language models on many evaluation tasks. It implements more than 60 standard academic benchmarks and hundreds of subtasks and variants. It can evaluate models loaded through transformers, GPT-NeoX, and Megatron-DeepSpeed, as well as models hosted with vLLM, SGLang, and OpenAI-compatible APIs. It supports CLI subcommands (run, ls, validate) and YAML configuration files.
- 60+ standard academic benchmarks and hundreds of subtasks
- Support for transformers, GPT-NeoX, and Megatron-DeepSpeed models
- vLLM and SGLang backend support
- API model evaluation (batch and asynchronous requests)
- Configuration-based task creation and Jinja2 prompt design
- Open LLM Leaderboard task bundle
- CLI subcommands (run, ls, validate) and YAML configuration
- Register model backends, filters, and metrics through plugins
Who uses EleutherAI LM Evaluation Harness and for what
- Comparing language model performance across multiple academic benchmarks
Pricing
Open source (free)
GitHub에 공개된 평가 프레임워크(pip 설치 라이브러리·CLI)만 있고, 받은 본문에 유료 플랜이나 관리형 클라우드 언급 없음
공식 페이지 확인 2026-10-14(산출 규칙 2026-10-15)
AI tools similar to EleutherAI LM Evaluation Harness (alternatives)
EleutherAI LM Evaluation Harness vs. alternatives at a glance
| AI | Pricing | Combos on the map | Connected tools | Free | Cheapest monthly | Korean |
|---|---|---|---|---|---|---|
| EleutherAI LM Evaluation Harness | Free + paid | 0 | 7 | — | — | — |
| Braintrust | Free + paid | 0 | 7 | Yes | $249/mo | No |
| DeepEval | Free + paid | 10 | 7 | — | — | No |
| HumanEval | Free + paid | 0 | 7 | — | — | — |
| LangSmith | Free + paid | 10 | 5 | — | — | — |
1 of these 5 tools have a free plan. Cheapest paid plan: Braintrust ($249/mo).
Combo and connection counts are based on the relationships recorded on AI Galaxy Map (2026-10-26); they are not a quality ranking.
EleutherAI LM Evaluation Harness on AI Galaxy Map
- One of 27 AI tools in the Observability & Evals field.
- Connected to 7 other tools — #1 in its field.
Frequently asked questions
Can I use EleutherAI LM Evaluation Harness for free?
Open source (free). Prices can change, so check the official site before paying.
What AI can I use instead of EleutherAI LM Evaluation Harness?
Tools that do the same job, such as Braintrust, DeepEval, HumanEval, LangSmith, are connected to it on the map.
Sources (official pages)
Checked on the official pages on 2026-10-14. Prices and features can change, so check the official site again before paying.