FlowTSE: Target Speaker Extraction with Flow Matching

Aviv Navon, Aviv Shamsian, Yael Segal-Feldman, Neta Glazer, Gil Hetz, Joseph Keshet

aiOla Research

InterSpeech 2025

Paper

Abstract

Target speaker extraction (TSE) aims to isolate a specific speaker's speech from a mixture using speaker enrollment as a reference. While most existing approaches are discriminative, recent generative methods for TSE achieve strong results. However, generative methods for TSE remain mostly underexplored, with many methods relying on complex pipelines and pretrained components, leading to computational overhead. In this work, we present FlowTSE, a simple yet effective TSE approach based on conditional flow matching. Our model receives an enrollment audio sample and a mixed speech signal, both represented as mel-spectrograms, with the objective of extracting the target speaker's clean speech. Furthermore, for tasks where phase reconstruction is crucial, we propose a novel vocoder conditioned on the complex STFT of the mixed signal, enabling improved phase estimation. Experimental results on standard TSE benchmarks show that FlowTSE matches or outperforms strong baselines.

Architecture

Model Architecture

LibriMix "mix-both" Samples

Mixture input Enrollment Clean Ours
1. Clean text: "I suppose it's the wet season will you have to cut them too"
2. Clean text: "Hilda was very nice to him, and he sat on the edge of his chair, flushed with his conversational..."
3. Clean text: "And I have no one ready to whom I can give up the archives of the Government."
4. Clean text: "He stood still in deference to their calls and parried..."
5. Clean text: "The Seasons, allies of Spring, followed him closely, to form a quadrille..."
6. Clean text: "Another and far more important reason than the delivery of a pair of..."