Speech Synthesis From Continuous Features Using Per-Token Latent Diffusion
Abstract
We present SALAD, a zero-shot text-to-speech(TTS) autoregressive model that operates over continuous speech representations. SALAD utilizes a per-token diffusion process to refine and predict the continuous representation for the next time step. We explore two additional variants of our method, extending popular discrete speech synthesis techniques. We implement discrete baselines for each variant and conduct a comparative analysis of discrete versus continuous speech modeling techniques. Our results demonstrate that both continuous and discrete approaches are highly competent, and that SALAD achieves a superior intelligibility score while obtaining comparable speech quality and speaker similarity metrics with the ground-truth audio.
Systems
We compare the following systems:
Continuous T2A: our text-to-acoustic SALAD variant that predicts continuous acoustic features and discrete semantic tokens in parallel.
Discrete T2A: our discrete text-to-acoustic that predicts semantic and RVQ acoustic features in parallel.
Continuous NAR: our non-autoregressive continuous MaskGIT variant that predicts continuous acoustic features based on semantic tokens.
Discrete NAR: our SoundStorm discrete semantic-to-acoustic model, that uses the improved random unmasking method instead of confidence-based unmasking.
Samples
Text
Prompt
Continuous T2A
Discrete T2A
Continuous NAR
Discrete NAR
Ground Truth
edison had installed his historic first great central station system in new york on the multiple arc system covered by his feeder and main invention which resulted in a notable saving in the cost of conductors as against a straight two wire system throughout of the tree kind
and wylder laughed too more suddenly and noisily than the humour of the joke seemed quite to call for and glanced a grim look from the corners of his eyes on lake but the gallant captain did not seem to perceive it and after a few seconds more he handed it very innocently back to missus dorothy only remarking
delivered in a strong tone of assent announced the gratification the savage would receive in witnessing such an exhibition of weakness in an enemy so long hated and so much feared
which thou dost scatter on every little blade of grass that springs revives the milked cow and tames the fire breathing steed
but anne had begun to suffer just before the holidays and charlotte watched over her younger sisters with the jealous vigilance of some wild creature that changes her very nature if danger threatens her young
she tried this morning an air or two upon the piano sang a simple song in a sweet but slightly metallic voice and then seating herself by the open window read philip's letter
doubts now arose and some discussion followed whether or not it was desirable for ben zoof to accompany his master
but in the rest of the work the power of language seems to fail him and the dramatic form is wholly given up
with a further advance in culture this tabu may change into simple custom of a more or less rigorous character but whatever be the theoretical basis of the distinction which is maintained whether it be a tabu or a larger conventionality the features of the conventional scheme of consumption do not change easily
hilda was very nice to him and he sat on the edge of his chair flushed with his conversational efforts and moving his chin about nervously over his high collar
now delia contrived to obtain a great influence and ascendency over the minds of the children by means of these dolls
let him come in then said the king and as if colbert had been listening at the door for the purpose of keeping himself au courant with the conversation he entered as soon as the king had pronounced his name to the two courtiers
in a few days an officer came with a requisition from governor shannon and took the prisoner by land to westport and afterwards from there to kansas city and leavenworth
i remained there alone for many hours but i must acknowledge that before i left the chambers i had gradually brought myself to look at the matter in another light
the pleasant graveyard of my soul with sentimental cypress trees and flowers is filled that i may stroll in meditation at my ease
now you will see in these studies that the moment the white is inclosed properly and harmonized with the other hues it becomes somehow more precious and pearly than the white paper and that i am not afraid to leave a whole field of untreated white paper all round it being sure that even the little diamonds in the round window will tell as jewels if they are gradated justly