Nvidia, the tech giant known for its graphics processing units, has introduced a remarkable new AI audio model called Fugatto. This innovative tool pushes the boundaries of sound synthesis, offering capabilities that were previously thought impossible.
Synthesising the Unheard
Fugatto stands out for its ability to create sounds that have never existed before. By combining various audio traits and characteristics, the model can produce unique auditory experiences. For instance, it can generate the sound of "saxophones barking" or "ambulance sirens singing in a choir". This level of audio manipulation opens up new possibilities for musicians, sound designers, and creative professionals across various industries.
The model's versatility is evident in its capacity to handle a wide range of audio tasks. From transforming speech and music to generating entirely new sound effects, Fugatto acts as a comprehensive tool for audio manipulation. Nvidia researchers describe it as a "Swiss Army knife for sound", highlighting its multifaceted capabilities.
The Science Behind the Sound
The development of Fugatto involved a complex process of data collection and model training. Nvidia researchers utilised a vast array of open-source audio datasets, totalling over 50,000 hours of audio. These samples were meticulously annotated with synthetic captions, describing various audio traits such as gender, emotion, and acoustic qualities.
To enable the model to understand relational comparisons, the researchers employed datasets where one factor remained constant while others changed. This approach allowed Fugatto to learn the nuances of different audio characteristics, such as distinguishing between instruments or identifying emotional tones in speech.
The final model, with 2.5 billion parameters, was trained using 32 Nvidia tensor cores. This extensive training process resulted in a system capable of performing a variety of audio quality tests reliably.
Implications for the Creative Industry
Fugatto's potential applications span numerous fields, from music production to video game sound design. The model's ability to prototype songs, create dynamic video game scores, and assist in international ad targeting showcases its versatility.
However, Nvidia emphasises that Fugatto is not intended to replace human creativity. Instead, it should be viewed as a powerful new tool for audio artists to expand their creative possibilities. As Ido Zmishlany, a producer and songwriter, noted, "With AI, we're writing the next chapter of music. We have a new instrument, a new tool for making music—and that's super exciting."
While Fugatto is not yet available for public testing, the samples provided on Nvidia's website demonstrate its potential to revolutionise the audio industry. As AI continues to evolve, tools like Fugatto are set to redefine the boundaries of sound creation and manipulation, offering exciting new avenues for artistic expression.
