write-up kokoro-fastapi
Adding zero-shot voice-cloning* to Kokoro-82M
Kokoro-82M is still charting at 10M+ downloads a month, and voice cloning is one of the features the community has been asking for. I took a crack at it over a weekend, with under $20 in GPU.
Still charting at 10M+ downloads/month and gaining; there have been newer text-to-speech models since its release about 2 years ago, but Kokoro-82M set the standard for lightweight high-quality speech.
It gave me my first exposure to maintaining a popular open source project via remsky/Kokoro-FastAPI. I recently took a crack at addressing one of the features the community has been asking for there: Voice Cloning.
TLDR: see the HuggingFace repo for samples and usage
Background
Why was Kokoro so popular?
It is almost shockingly lightweight for such high quality speech, running most places without issue (including in-browser via kokoro.js). A big part of that win is credited to the curated grapheme-to-phoneme library (hexgrad/misaki) which handles pronunciation.
Why there’s a gap?
Kokoro ships without its style encoder, which is what maps reference audio to a new trained voice. Pre-trained voice packs can combine for some customization, but the secret sauce used to add a truly new voice is retained by original model creator [hexgrad].
How does the stock system work?
The diagram below can help with an intuition, it is a swappable embedding tensor built from a speaker’s voice:
Have others tackled voice creation?
Other community approaches did great work in this direction (kokoclone, kvoicewalk, KokoroVoices, along with voice pack bundles and RVC post-processing), but weren’t quite the right fit here on speed and flexibility. I aimed to build on the model’s strengths and appeal, focusing on:
- Stock generation speed, quality, compatibility
- Minimal impact on time-to-first-audio from reference
- Voices should be genuinely new, guided by the reference.
Evaluation
Unofficially I set a few requirements:
- Keep the budget reasonable for an exploratory project.
- It should only require a single pass through the model at inference time. This softened to better control processing, and led to a small helper library, inno-kokoro (GitHub, PyPI).
- Vincent Price as a reference should be compellingly spooky.
Details on the metrics used during training can be found on HuggingFace.
Development
Exploration
Initial short run investigations helped to understand the problem space quickly and iterate. I drew from the karpathy/autoresearch approach when getting started to set up my workspace.
Agentic CLI tooling was used during iteration, troubleshooting, and to keep tabs on experiment runs & compare results as part of the development process. To corral this, I set a few hooks on a sandboxed folder including a budget tracker, enforcing runs as auto-labelled with a clean commit hash, and an equipped subagent written to flag insight or errors based on local testing and continuous monitoring of the runs against my own working notes & assumptions.
Training
More detail can be found on HuggingFace, but briefly:
- Runs were via HF Jobs (a10g-small) w/ Weights & Biases
- The v0.1.0 model was about $7, including distillation of the encoder, trained with audiobook readers and archival audio
- The v0.2.0 (prosody head) was about $10, improving pitch range at a small cost to mean pitch error, with VoxPopuli and Emilia (YODAS) added to the mix
- Microsoft’s UniSpeech-SAT (380mb fp32) was distilled onto WeSpeaker ResNet34 (14mb fp16), and baked into the model for portability.
Architecture
Results
Speed & Quality
Send a reference clip along with the text you want spoken, and it comes back sub-second in the cloned voice for most cases. Zero-shot timings are dependent on machine specs, but generally:
- 0.05s per 1s of input audio (a 30s reference in 1.4s) [CPU]
- 0.3s total (5-30s reference, after initial model load) [GPU]
All packs generate fully compatible with base Kokoro and maintained high UTMOS (naturalness) scoring. And if you like the voice, it’s saved, reusable, and shareable as a ~500kb file (the same as stock).
Identity
By the numbers it captures about a third across the evaluation set. Is it a strong deepfake tool? I wouldn’t use it for that, no. A more accurate name would be clone-tuning.
For zero-shot, reliable, and unique voice creation guided by reference audio, runnable on base Kokoro-82M (inheriting its speed, compatibility, and Misaki G2P quality basis), the Inno Clone-Tuner delivers. And by ear, it can get pretty close depending on the reference audio.
- Amelia Earhart was surprisingly good, despite using heavily degraded archival audio as the input.
- Jane Goodall’s example is intentionally included as a partial miss. It has aspects of her speaking style, but degrades towards stock speech as though from a younger speaker.
- Vincent Price, you can be the judge, but I think it hit the mark.
Reach out
Community feedback so far has been positive and useful, and that is always welcome. If you are interested in chatting about aspects of the development process, I’d love to hear from you as well.