A TTS benchmark harness, five models in, nine to go

Fourth write-up out of the experiment register, and the first one that isn't a pass/fail hypothesis test. feature/tts-voice-clone-benchmark is building a Concourse-orchestrated harness to compare roughly 14 local zero-shot voice-cloning models on identical inputs. This is a progress snapshot, not a verdict: five models are wired and validated so far, nine are not.

What it's actually testing

The source is a Reddit thread ("Best AI Voice Cloning in 2026") plus its top comments, cross-checked against what's actually feasible on this rig. Zero-shot cloning only: the source article's per-model rows used 45 minutes of fine-tuning data per voice, which isn't a fair "same inputs" comparison against a harness that hands every model the same short reference clip and walks away. That reference clip is fixed and documented: reference_24k_mono.wav, LibriSpeech test-clean speaker 6930, utterance 6930-75918-0003, 23.3 seconds, with the exact script text held constant across every model too.

Sample 0 from each of the five validated models, in order (XTTS-v2, Chatterbox Turbo, OmniVoice, dots.tts, MOSS-TTS-Local-Transformer-v1.5), pulled from the model's own /benchmark response rather than from git, since generated benchmark audio isn't committed. All five clone the same reference_24k_mono.wav speaker reading the same script text. Durations and formats check out against this article's own numbers: XTTS-v2 12.94s/24kHz mono, Chatterbox Turbo 13.88s/24kHz mono, OmniVoice 12.62s/24kHz mono, dots.tts 11.2s/48kHz mono, MOSS-TTS-Local-v1.5 12.72s/48kHz stereo (the only one of the five that renders in stereo, this model's own native audio-tokenizer format).

The first four models' own exited Docker containers (where the earlier pass of this article pulled its samples from) are still on disk, but the actual per-model Python environments that produced them are gone: the worktree this branch was registered against degraded into a broken, root-owned stub before this session started, taking every .venv/ with it. Re-running any of the first four from scratch means reinstalling each model's environment again, not just re-triggering the job; nothing about the harness pattern itself changed.

Every model gets its own directory under tools/tts-bench/<model>/ with a FastAPI server exposing /health, /load, /benchmark, and /unload, running as its own Docker Compose service so it gets real GPU access, the same host-level-passthrough pattern already used for comfyui-local. A tts-benchmark-<model> Concourse job, mutexed through gpu-lock, drives each one. Only one model's service can be up at a time: this rig's single 24GB 3090 and a home partition sitting at 98% full can't hold two models' weights simultaneously.

What "validated" means here

Validated means each model ran a real /benchmark call through Concourse and gpu-lock, produced actual generated audio, and returned real measured numbers: load time, per-sample generation time, peak VRAM via the model process's own torch.cuda stats, and real-time factor (RTF). It does not mean anyone listened to the output and judged voice-cloning quality; no MOS score or by-ear quality call is recorded for any of the four. This harness measures whether a model runs and how expensive it is, not whether it sounds right yet.

Five models cleared that bar, in this order:

• XTTS-v2 (Coqui): first validated, build #5 after three unrelated bugs (a transformers version that dropped a symbol coqui-tts still expected, a missing torchcodec dependency, and a root-owned cache directory from Docker's bind-mount auto-create). Load 67.7s, generation 3.3-5.3s/sample, peak VRAM ~1.9-2.0GB, RTF 2.4-3.7x.
• Chatterbox Turbo (Resemble AI): second validated, one new bug (a watermarking dependency silently disabled itself because setuptools>=81 no longer ships pkg_resources). Load 7.8s, generation 3.6-12.8s/sample, peak VRAM ~3.3GB, RTF 1.1-3.7x.
• OmniVoice (k2-fsa): third validated, one new container-level bug (Triton needed a full C toolchain for its first JIT compile, which neither prior model had exercised). Load 2.1s, generation 1.9-4.5s/sample, peak VRAM ~3.86GB, RTF up to 6.5x, the fastest of the five.
• dots.tts (rednote-hilab): fourth validated, no new bugs, first attempt clean. Load 24.2s, generation 22.8-39.0s/sample, peak VRAM ~5.6-5.7GB, RTF 0.29-0.44, the only one of the five that runs slower than real time on this box.
• MOSS-TTS-Local-Transformer-v1.5 (OpenMOSS): fifth validated, two new bugs (torchaudio needed torchcodec pinned to exactly 0.9, since the latest release requires a newer torch than this model's own pin and crashed with an ABI mismatch; separately, this shim's own /unload handler left roughly 13GB of VRAM stuck allocated across two failed builds because dropping the model reference alone didn't free it before torch.cuda.empty_cache() ran, fixed with an explicit gc.collect() first). Load 8.5s, generation 9.5-11.3s/sample, peak VRAM ~12.2GB, RTF 1.1-1.35x, native 48kHz stereo output. Worth flagging: this repo's own candidate list named this row "Local-Transformer 1.7B," but OpenMOSS ships three different "v1.5"-adjacent checkpoints and that figure describes a different, older one; the checkpoint actually run here is a 4B backbone, the smaller of the two true "v1.5" releases.

Where the other ten stand

Of the roughly 14 candidates, this is where things actually sit as of this snapshot:

• Not started, no blockers known: Chatterbox (base), CosyVoice 3, VibeVoice 1.5B, IndexTTS-2, MOSS-TTS-v1.5 (the other "v1.5" checkpoint, the 8B flagship, distinct from the 4B Local-Transformer variant validated above).
• Feasibility checked, not scaffolded: Audio8-TTS-Preview-0.1b (Audio8-AI/Audio8_TTS on GitHub; licensing is fine for this use, needs a git clone install rather than a plain pip install), Fish S2 Pro (Fish-Speech; flagged rather than guessed at: unclear if the open-source repo actually ships the "S2 Pro" checkpoint or only the plain "S2," since "S2 Pro" also exists as a separate paid hosted tier).
• Partially wired, unvalidated, and explicitly not to be duplicated: Qwen3-TTS. A separate branch already built a service and a smoke-test job for this one, merged to main, but it OOM'd once against a concurrent render and was never re-validated afterward; nothing is currently running on its port. The benchmark harness's own notes flag that a later candidate ("faster-qwen-tts") should extend that existing work rather than scaffold a fifth Qwen3-TTS install from scratch.
• Explicitly out of scope: ElevenLabs Professional, a paid cloud API with no local VRAM/GPU-time to capture. Worth a one-line mention in a future final report, not a row in the comparison table.

Where it actually stands

Ongoing, five of roughly fourteen wired and validated, nine remaining. This is not a hypothesis test with a pass/fail outcome: the register correctly tracks it as a non-A/B benchmark build-out. No cross-model comparison or ranking exists yet; what exists is a working harness pattern (proven four times over after the first model's install debugging) and five models' worth of raw performance numbers, with no quality assessment layered on top of any of them. The honest state of this branch today is "harness works, five data points collected," not "here's the best model."