Voice Conversion

Replicating one individual's voice through another person.

What's voice conversion?

Voice conversion allows you to change one individual's voice to resemble another's. This is achieved using a technique known as voice cloning. This method captures the essence of the desired voice, allowing the generated speech to mirror the target's vocal identity while maintaining the original speech's tone.

Uses

Advanced voice conversion and voice cloning technologies hold the promise of reshaping content creation, distribution, and engagement across multiple sectors. These technologies not only streamline production processes and cut costs but also present unique income opportunities for individuals who lend their voices for algorithm training.

In the realm of filmmaking:

** Actors can share voice databases, allowing producers to craft audio tracks without on-site or studio recording.

** Correcting mispronounced lines during post-production becomes seamless.

** The tech can recreate voices of historical personalities for fictional scenarios or even resurrect voices of past actors.

For video game developers:

**Voice corrections and experimentation can be instantaneous, eliminating the need for actor's on-site presence.

Medical applications include:

**Providing voice restoration for patients, such as those affected by throat cancer treatments, allowing them to communicate in their original tone.

For consumer technology:

**Virtual assistants can be personalized, letting users interact with familiar voices rather than unfamiliar ones.

In advertising:

**Synthetic voiceovers that sound human-like sidestep rights and royalty concerns. Additionally, when a specific recognizable voice is desired, the tech can clone an actor's voice, removing the need for prolonged recording sessions.

Audiobook and podcast sectors represent additional industries where voice cloning and conversion can enhance production and editing of immersive content.

ElevenLabs Voice Conversion

While we do offer voice conversion software in our suite of tools at Eleven, the primary focus of our research into voice cloning and synthesis is the creation of our flagship product set to launch next year: the identity-preserving automatic dubbing solution.

Our vision is to seamlessly make spoken content universally accessible, retaining the speaker's original voice, with just a button's press. Take an educational YouTube video in English, for example. If a viewer only understands Spanish, they miss out. While captions offer a workaround, we're aiming for a richer, more immersive experience. Our technology would enable the content to be delivered in fluent Spanish, all while sounding like the original speaker.

The beauty of voice cloning lies in its ability to maintain the uniqueness of the speaker's voice. This technology enables us to craft new messages in diverse languages, maintaining the essence of the original speaker.

Voice conversion is crucial as it ensures the authenticity of emotions, intentions, and delivery style is intact, offering an immersive experience. Our sophisticated multi-language models are trained to interpret content in the source language and replicate it in the desired language with perfect intonation.

Process

To change one person's voice to sound like another's – that is, to shift from the source voice to the target voice – we require an algorithm that captures the essence of the source voice and layers it with the characteristics of the target voice. A fitting comparison can be made with face-swapping apps where you can blend your facial features with someone else's to create a composite image.

The process involves analyzing an image of a face and identifying its key attributes. The points illustrated in the example below act as boundaries within which the features of the second face are superimposed.

Image credit: Elevenlabs.io

In voice conversion, the algorithm must be able to capture the unique attributes of the target speech. To achieve this, the algorithm is trained using a vast array of speech samples from the target. It dissects these samples down to their most basic components - think of these as the "building blocks" of speech. At its core, speech is made up of sentences. Sentences consist of words, and words are constructed from phonemes. It's these phonemes that carry the distinguishing features of the target speech and serve as the foundational level at which the algorithm functions.

Image credit: Elevenlabs.io

In voice conversion, the challenge is to present the content of the source speech using the phonemes of the target speech. This process is analogous to face-swapping: the more points or markers you use to capture one face's features, the more restricted the mapped face becomes. Similarly, in voice conversion, the emphasis we place on the target speech can potentially overshadow the source speech's nuances. If we prioritize the target speech too much, we risk deviating from the original content of the source speech.

On the other hand, if the target speech is not given enough emphasis, we might not capture its distinct characteristics. For instance, trying to convey someone's furious shouting in the calm demeanor of Morgan Freeman's voice poses a dilemma. Overemphasizing the original emotions could dilute the iconic nature of Freeman's voice, while focusing too much on replicating his voice could mute the raw emotion of the original speech.

Ethics

The rapid advancements in voice cloning have inevitably brought forth ethical concerns. The potential misuse of this technology is alarming, with incidents such as the 2020 case where scammers utilized audio deepfakes to mimic a CEO’s voice and successfully authorized a bank transfer of $35 million. Such technology, which can persuasively make someone appear to say something they never did, naturally stirs anxieties about misinformation, defamation, and fraudulent activities. Additionally, voice conversion prompts debates on copyright violations, especially when content is produced without the explicit consent of the voice owner.

At Eleven, we are deeply committed to ethical use:

01. We collaborate only with clients who strictly adhere to our Terms of Service, which explicitly prohibit any malicious intentions, including spreading misinformation, defamation, fraud, or any other illicit or harmful actions.

02. Any video content synthesized by Eleven is distinctly marked with a watermark indicating its AI-generated nature. Similarly, our audio content has an explicit file descriptor. When replicating well-known voices, it is strictly for illustrative purposes and is done in non-controversial settings.

03. We remain steadfast in our commitment to support voice owners and their licensors in upholding their rights.

Feedback is invaluable. If you believe there are ways to bolster our ethical approach, please reach out to us at ethics@elevenlabs.io.

While the potential for misuse exists, we firmly believe that the predominant narrative shouldn't be fear-driven. Instead, our focus should be on establishing stringent safeguards during the developmental phases. This ensures that while risks are minimized, we can collectively harness the immense benefits this technology promises for society at large.

Future

The horizons of voice conversion and cloning technology are vast, poised to reshape sectors from filmmaking and television to game development, podcasts, audiobooks, and advertising. Yet, their reach isn't limited to just commercial arenas; they extend into vital areas like medicine, education, and broader communication channels.