StyleTTS2

StyleTTS 2 is a research model that aims for human-level text-to-speech (TTS) using style diffusion and adversarial training with large pretrained speech language models (SLMs). It treats style as a latent variable of a diffusion model, so it generates a style that fits the text without a reference speech. It publishes samples for single speaker (LJSpeech), multi-speaker (VCTK), zero-shot speaker adaptation (LibriTTS), and long-form reading, and provides training and inference code, pretrained models, and a fine-tuning script for new speakers on GitHub.

• Generates a speaking style that fits the text without a reference speech (style diffusion)
• Adversarial training that uses large speech language models such as WavLM as discriminators
• Zero-shot speaker adaptation — similar generalization with 250 times less data than VALL-E
• Long paragraph reading (long-form narration)
• Provides a script for fine-tuning new speakers from the base multi-speaker model
• Training and inference code and pretrained models are public

Open source (free, weights/code released)

Released free under an open-source license with no extra cost for self-hosting (usage-based pricing when offered as a cloud API)

Estimate based on AI knowledge - please verify on the official site

• One of 60 AI tools in the Foundation Models field.
• Connected to 5 other tools — #16 in its field.