HarmBench

HarmBench is a standardized evaluation framework for rigorously evaluating automated red teaming methods. It can be used in two ways: evaluating red teaming methods against multiple LLMs, or conversely evaluating an LLM against multiple red teaming methods. It compared 18 red teaming methods and 33 target LLMs and defense techniques at large scale. It also presents an efficient adversarial training method to improve LLM robustness.

• Evaluating a red teaming method against a set of LLMs
• Evaluating an LLM against a set of red teaming methods
• Evaluation pipeline of test case generation → response generation → response evaluation
• run_pipeline.py, a tool that runs the pipeline automatically
• Supports transformers-compatible LLMs, closed APIs, and multimodal models
• Provides adversarial training code and models

Open source (free)

연구용 오픈소스 평가 프레임워크(논문·GitHub 공개), 요금 페이지는 비어 있음

공식 페이지 확인 2026-10-14(산출 규칙 2026-10-15)

• One of 21 AI tools in the AI Security & Safety field.
• Connected to 6 other tools — #1 in its field.