LLaVA

LLaVA (Large Language and Vision Assistant) is a large multimodal model for general-purpose visual and language understanding. It connects a CLIP ViT-L/14 vision encoder to a Vicuna language model with a simple projection matrix. It was the first attempt at visual instruction tuning, using multimodal instruction-following data generated with language-only GPT-4. It was fine-tuned for two uses, everyday-user visual chat and science QA, and the authors state that combined with GPT-4 it reached the best performance on Science QA at 92.53%. Data, model, and code are released, but licensed for research use only.

• CLIP ViT-L/14 vision encoder connected to the Vicuna LLM
• 158K multimodal instruction data generated with GPT-4 (LLaVA-Instruct-150K)
• Two-stage instruction tuning (feature-alignment pretraining → end-to-end fine-tuning)
• LLaVA-1.5: best performance on 11 benchmarks, trained in about 1 day on a single 8-A100 node
• Optical character recognition (OCR) examples
• LLaVA-NeXT: processes 4x more pixels

Open source (free, weights/code released)

Released free under an open-source license with no extra cost for self-hosting (usage-based pricing when offered as a cloud API)

Estimate based on AI knowledge - please verify on the official site

• One of 22 AI tools in the Multimodal & Perception field.
• Connected to 5 other tools — #11 in its field.