Captum

Captum is a scalable open-source model interpretability library built on PyTorch. Using algorithms such as Integrated Gradients, TCAV, and TracIn, it shows which features, training examples, and concepts contributed to a model's prediction. It supports models across multiple modalities such as vision and text, and can be used with little modification to the original neural network. Researchers can implement new algorithms and compare them with existing ones, and it also offers adversarial attacks and minimal input perturbation for counterfactual explanations.

• Interpreting multimodal models such as vision and text
• Supports PyTorch models with minimal modification
• Interpretation algorithms such as Integrated Gradients, TCAV, and TracIn
• Implementations of saliency map, smoothgrad, and vargrad
• Adversarial attacks and minimal input perturbation (counterfactual explanations)
• Extensible structure that makes it easy to implement and benchmark new algorithms
• Fast integration with torchvision, torchtext, and more

Open source + paid cloud-managed plans

Open-source edition is free, SaaS managed edition is a paid subscription based on usage/seats

Estimate based on AI knowledge - please verify on the official site

• One of 27 AI tools in the Observability & Evals field.
• Connected to 4 other tools — #15 in its field.