Snorkel

Snorkel develops specialized training data and environments so that frontier labs and AI teams can differentiate their models and agents. It combines task design, programmatic checks, calibrated expert review, and realistic evaluation environments to create measurable learning signals. It repeats a cycle of measuring (evaluating) where a model fails, curating data and environments to match those failures, and improving until performance rises. It originated at the Stanford AI Lab and also runs public benchmark leaderboards.

• Developing expert-curated datasets for frontier AI
• Building realistic evaluation environments with programmatic pass/fail criteria
• Rubric-based pipelines involving calibrated experts
• Developing evaluators (model-based and rule-based) and meta-evaluation of the evaluators themselves
• Covers more than 1,000 specialized domains
• Public benchmark leaderboards for agentic coding, computer use, enterprise tasks, and more
• Building custom specialized agents

Enterprise (contact for pricing)

Custom quotes based on data labeling/RLHF volume and quality level are typical

Estimate based on AI knowledge - please verify on the official site

• One of 8 AI tools in the AI Data & Human Feedback field.
• Connected to 1 other tools — #5 in its field.