The new Text2Speech model, Bark, was just introduced, and it has constraints on voice cloning and permits prompts to ensure user safety. However, scientists have decoded the audio samples, freed the instructions from constraints, and made them available in an accessible Jupyter notebook. Now, using just 5-10 seconds of audio/text samples, it is possible to clone a whole audio file.
What is Bark?
Suno’s groundbreaking Bark text-to-audio model is built on GPT-style models and can produce natural-sounding speech in several languages, in addition to music, noise, and basic sound effects. Suno developed the Bark text-to-audio paradigm using a transformer. In addition to making a natural-sounding speech in several languages, Bark can also create music, ambient noise, and basic sound effects. The model can also generate facial expressions, including smiling, frowning, and sobbing.
Bark uses GPT-style models to create speech with minimum fine-tuning, resulting in voices with a wide range of expressions and emotions that accurately reflect subtleties in tone, pitch, and rhythm. It’s an amazing experience that makes you question whether or not you’re talking to real people. Bark has impressively clear and accurate voice generation capabilities in several languages, including Mandarin, French, Italian, and Spanish.
How does it work?
Bark employs GPT-style models to produce audio from scratch, just as Vall-E and other incredible work in the area. In contrast to Vall-E, high-level semantic tokens incorporate the first text prompt instead of phonemes. Therefore, it may generalize to non-speech sounds, such as music lyrics or sound effects in the training data, in addition to speech. The entire waveform is then created by converting the semantic tokens into audio codec tokens using a second model.
Features
- Bark has built-in support for several languages and can automatically detect the user’s input language. While English presently has the highest quality, other languages will improve as one scale. Therefore, Bark will use the natural accent for the corresponding languages when presented with code-switched text.
- Bark is capable of producing any form of sound imaginable, including music. There is no fundamental distinction between speech and music in Bark’s mind. On occasion, though, Bark will instead create music based on words.
- Bark can replicate every nuance of a human voice, including timbre, pitch, inflection, and prosody. The model also works to save environmental sounds, music, and other inputs. Due to Bark’s automated language recognition, you may utilize a German history prompt with English content, for instance. As a result, the resulting audio typically has a German accent.
- Users can specify a certain character’s voice by providing prompts like NARRATOR, MAN, WOMAN, etc. These directions are only sometimes followed, especially if another audio history direction is supplied that conflicts with the first.
Performance
CPU and GPU (pytorch 2.0+, CUDA 11.7, and CUDA 12.0) implementations of Bark have been validated. Bark can produce near real-time audio on current GPUs using PyTorch every night. Bark demands running transformer models with over a hundred million parameters. Inference times might be 10–100 times slower on older GPUs, the default collab, or a CPU
By Dhanshree Shripad Shenwai
Dhanshree Shenwai is a Computer Science Engineer and has a good experience in FinTech companies covering Financial, Cards & Payments and Banking domain with keen interest in applications of AI. She is enthusiastic about exploring new technologies and advancements in today’s evolving world making everyone’s life easy.
