A voice that stands in for you
The voicemail that sounds like nobody
When you are away, something answers for you. Today it answers in a voice that is not yours.
You are in a meeting, or asleep, or on a flight. Someone who matters to you calls and gets a recorded stranger. The words may be right. The voice tells them, before any word does, that you are not there.
A stand-in is an assistant that answers in a person's own words and voice while they are away. It needs two things: a way to speak like the person, and a way to sound like them. This piece is about the second.
The question was plain. From one ordinary recording of a person, how close can a generated voice get to the real one, and how do we know?
A recording is not a voice
Playing back what someone said is easy. Saying something they never said, in their voice, is the problem.
A recording holds sentences. A voice can say any sentence. Cutting old sentences together gets you a ransom note: the right speaker, the wrong rhythm, and nothing for the sentence that was never recorded.
Cloning a voice means generating new speech that a listener would take for that person. The recording is only a sample to learn from. The clip people actually have is short, made on a phone, in a room that echoes.
So there are two versions of the problem. One starts from a clean, quiet recording. The other starts from the clip a real person would upload. They turn out to be very different.
Two questions, one score
Likeness and correct words are multiplied together, so a voice that sounds right and says the wrong thing scores badly.
A clone is judged on two things at once. The first is likeness: how close the generated speech is to the person. The second is the words: did it say what it was asked to say.
Likeness is put on a scale with two fixed ends. At the bottom is a stranger: how similar other people's voices are to this person. At the top is the person themselves: how similar their own other recordings are. A clone at the top of that scale is as close to the person as the person is to themselves.
The words are checked by transcribing the generated speech and counting the words that came out wrong. The two are multiplied, so neither can hide the other.
A clean test and a hard one
Every result is reported on a clean recording and on a poor one, because good systems all pass the clean test.
The test has two conditions. In the clean one the sample is a quiet, studio-quality clip. In the hard one the sample is shorter and has been degraded the way a phone or a video recording is: a narrow band, room echo and background noise.
In both conditions the clone is compared with the person's real, clean voice. A clone made from a noisy clip is not excused for being noisy.
Two guards sit beside the score. A separate measure of audio quality must not fall, and speech must be generated quickly enough to hold a conversation. The speakers are a small set of public-domain recordings, men and women, reading everyday sentences.
On a clean recording, the gap closes
From a clean clip the clone reached the top of the likeness scale, with fewer than one word in a hundred wrong.
From a clean recording, the clone is as close to the person as their own other recordings. On the likeness scale it reached the top, the level of the real speaker.
It also said the right words. Fewer than one word in a hundred came out wrong, and most of those were a matter of how a number is written down, not a slip of speech.
And it was quick: speech was generated faster than it takes to play. All of this ran on the person's own machine, with no cloud voice service.
On a real recording, about four fifths of the way
From a short, noisy, echoing clip, every approach tried fell to about four fifths of the person's likeness.
The clip a real person uploads is the real problem. With a short, degraded sample, every approach we tried fell to about the same place: roughly four fifths of the way from a stranger to the person.
The poor recording also leaked into the result. Measured audio quality of the generated speech was about a quarter lower than from a clean clip, because the noise in the sample came through in the voice.
This is the useful finding. Which voice system you pick matters much less than what recording you give it.
Closing part of the gap
On poor recordings about a third of the remaining gap was closed, at more than twice the generating time.
Part of the gap can be bought back. On poor recordings, the best configuration closed about a third of the distance that remained between the clone and the person.
It was not free. That configuration takes more than twice as long to generate each sentence, which pushes it just past real time. For a spoken reply that a caller is waiting for, the quicker configuration is the one to use; for a message prepared in advance, the closer one.
Overall, across clean and poor recordings together, the best configuration scored a few percent above where the work began.
What the score cannot hear
The likeness measure is an instrument, not a listener. It is blind to rhythm, and the test is small.
Likeness on an instrument is necessary, not sufficient. The measure compares the colour of a voice. It does not hear rhythm, emphasis or the small artefacts a person notices at once. The final judgement is a human ear, and no listening test with people has been run.
The test is small: a handful of public-domain speakers and a short list of sentences, in one language. A difference of a point or two between configurations is inside the noise.
The hard condition is a simulation of a poor recording. It is not a collection of real ones.
Whose voice, and what is on request
Only a voice you own or have permission for. The comparison is public; the figures and the system are on request.
Speak As Me is for your own voice. It is meant for a voice you own or have explicit permission to use, and the stand-in is told to say that it is an AI when someone sincerely asks.
What is public is the shape of the result: how close, on what kind of recording, at what cost in time. Which voice systems were compared, how the recording is prepared and what was changed are not published. The measured scores for every configuration are in the full results paper. If you want to hear it or try it on your own voice, ask.
Step 2: the paper