Text-to-speech (ElevenLabs-style)

01 · CONTEXT

What the system needs to do

The case asks for the design of a system that turns text into realistic speech, in the style of commercial synthetic-voice services. It is a classic interview problem in ML system design. Educative's Generative AI System Design course includes a module on designing a realistic speech generation system; this case does not reproduce that module and relies on scientific papers and a voice vendor's public documentation.

02 · TWO STAGES

From text to spectrogram, from spectrogram to waveform

One architecture splits the problem into two stages. In Tacotron 2, a sequence-to-sequence network maps characters to mel spectrograms, followed by a modified WaveNet that acts as a vocoder and synthesizes the time-domain waveform.

1. The acoustic model maps characters to mel spectrograms.

2. The vocoder synthesizes the time-domain waveform from the spectrogram.

3. In the diagram, the last edge connects the vocoder to the application.

The vocoder is the piece that varies the most. WaveNet is autoregressive: each audio sample is conditioned on all the previous ones. Another family uses generative adversarial networks (GANs) to produce raw waveforms, such as HiFi-GAN.

There are also single-stage models. According to the VITS paper, some recent end-to-end models had sample quality below that of two-stage systems; VITS proposes a parallel end-to-end method that, according to the authors, generates more natural audio than the then-current two-stage models.

The same text can be spoken in many ways, with different tones and rhythms. VITS also proposes a stochastic duration predictor to synthesize speech with diverse rhythms from text.

03 · QUALITY

How to measure whether the voice sounds natural

A metric cited in the papers is MOS (mean opinion score), a subjective human evaluation. Tacotron 2 reported a MOS of 4.53, comparable to the 4.58 MOS of professionally recorded speech.

04 · STREAMING

Serving audio while the text is still arriving

When the text arrives gradually, the design depends on the communication protocols. ElevenLabs' TTS WebSocket API was built to generate audio from partial text, and the documentation recommends it when text is streamed or generated in chunks and when word-to-audio alignment is needed.

1. The client sends text to the gateway gradually, in chunks.

2. The segmenter splits the text into segments before the model.

3. In the diagram, the acoustic model connects to the voice profile store.

4. The vocoder synthesizes the waveform from the acoustic representation.

5. The audio returns to the client in chunks, for real-time playback.

Streaming has a cost. Because generations are partial, there is buffering, which can cause slightly higher latency than a standard HTTP request, and working with WebSockets can be harder and more complex.

The choice of model is also a trade-off. According to the documentation, Flash v2.5 offers ultra-low latency of 75 ms, while Multilingual v2 delivers the highest audio quality, with more nuanced expression. For long texts, the documentation recommends splitting into segments and using streaming; it also gives specific guidance for keeping a natural prosody flow between chunks.

05 · VOICE CLONING

A chosen voice and a cloned voice

An example of serving several voices with the same model is in WaveNet: a single WaveNet captures the characteristics of many speakers and switches between them by conditioning on the speaker identity.

In a commercial service, cloning comes in two modalities. Instant Voice Cloning is created from short audio samples, under two minutes. Professional Voice Cloning requires more extensive training audio for the highest fidelity.

The documentation describes a voice-captcha technology that verifies that professional clones are created from the person's own voice samples.