What is speaker diarization?
The "who spoke when" problem, solved automatically.
You have an audio file with several people talking — an interview, a podcast, a meeting recording, a movie clip, a YouTube reaction, a lecture with Q&A. Speaker diarization is the AI task of figuring out, second by second, which speaker is talking at any given moment, and then bundling each speaker's contribution into its own clean audio track.
Drop a 30-minute podcast with 3 hosts → get back speaker_1.wav, speaker_2.wav, speaker_3.wav, each containing only that one person's lines (every other voice silenced). The split is based on voice signature (timbre, pitch, formants), not channel separation — it works on a single mono mix.
Headline use case — voice cloning for AI video
The problem. You're making an AI video where a specific person speaks — yourself, a public-domain figure, an actor whose voice you have license to use. To clone their voice with RVC / XTTS / OpenVoice / F5-TTS / your model of choice, you need 5 to 30 minutes of clean, single-speaker training data. The catch: the source material you have is almost never clean. It's an interview where someone else asks the questions. A podcast where two co-hosts overlap. A movie scene where the target speaks for 12 seconds before another character cuts in. A lecture with audience Q&A. A documentary voice-over with B-roll dialogue underneath.
The old workflow. Open Audacity, scrub through 30 minutes of audio, manually cut every segment where the wrong speaker talks, splice the rest back together. Two to three hours of tedious work per dataset. Any mistake contaminates the clone — one stray word from the wrong speaker and your AI character starts borrowing someone else's vowels.
The new workflow. Drop the file in ExpSoft Diarizer, set Number of speakers to "auto" (or pin it to 2/3/4 if you know), click Run. A few minutes later (real time on a 4060 Ti, ~5× real time on CPU) you have one .wav per speaker. Listen to each, identify your target, feed that single track to your voice-cloning trainer. Two hours of cleanup compressed to two minutes of compute.
Other use cases for AI video creators
👄
Lip-sync prep (Wav2Lip / SadTalker / Hedra / LatentSync)
Lip-sync tools assume one speaker. Multi-voice input confuses the timing model and produces glitchy mouths. Diarize first, feed only the target speaker's track.
🎬
AI dubbing — swap one voice, keep everything else
Replace one character's voice in a scene while preserving the other speakers and the ambient bed. Diarize, regenerate the target track only with your TTS / clone, mix back over the original.
📝
Speaker-labeled captions for shorts
Whisper transcribes but doesn't know who's talking. Combine its timestamps with diarization output to get "Host: … Guest: …" captions ready for TikTok / Shorts / Reels — the kind of fast-cut content where speaker changes need to be visible at a glance.
🎤
Podcast / interview post-production
Apply different EQ, compression and noise reduction per speaker independently. The target wears a Shure SM7B at home, the guest is on a Zoom call from a hotel room — you can only fix that asymmetry if their tracks are separated first.
🎧
Audio-driven AI music / sound design
ACE-Step's REMIX, Stable Audio's audio-conditioned generation, MMAudio's video-to-audio — all degrade when the input has overlapping voices over the music. Diarize, isolate the music+ambience, then feed clean.
🔗
Building voice datasets at scale
Want to train a multi-speaker TTS, a voice-conversion model, or a custom LoRA on a specific actor? Run Diarizer over a folder of 50 podcast episodes via Batch Processing, regroup the speakers across files, and you have hours of pre-cleaned per-speaker audio in one overnight job.
