Textraction

SC–MC Text-Guided Extraction and Speaker Enhancement

Aviad Eisenberg  ·  Sharon Gannot  ·  Shlomo E. Chazan

Faculty of Engineering, Bar-Ilan University  ·  OriginAI, Israel

A single unified model that extracts a desired speaker — or any subset of speakers and noise sources — from a mixture, using only a natural-language prompt. The same network handles both single-channel and multi-channel inputs, conditions on either semantic attributes or relative spatial descriptions, and reduces to a state-of-the-art speech enhancer when only one speaker is present. Prompts are free-form and may specify entities to keep, to remove, or both.

Unified SC & MC

One architecture for single- and multi-microphone inputs, exploiting spatial cues when present.

Free-form prompts

Keep / remove any subset of up to two speakers and two noises — robust to paraphrasing.

Lightweight text encoder

A fine-tuned E5 retrieval model, contrastively tuned for keep-vs-remove polarity — no heavy LLM.

Relative spatial cues

Describes speakers by relative position (near/far, left/right) in plain language.

Overview & demo walkthrough
Textraction model architecture
Model architecture

Multi-Channel Extraction

Two speakers and two noise sources are captured by a two-microphone array in a reverberant room. A free-form prompt names which entities to keep or remove, and the model reconstructs exactly that subset — exploiting spatial diversity when it is available.

Example 1
Mixture (input)
Prompt

Remove the weak speaker.

Example 2
Mixture (input)
Prompt

Kindly eliminate the loudspeaker from the recording.

Example 3
Mixture (input)
Prompt

Eliminate the gentle mallet instrument playing a delicate melody and the guy speaker.

Example 4
Mixture (input)
Prompt

Exclude the running motor vehicle engine, the softly purring cat, and the loud guy speaker, while keeping the female weak speaker.

Example 5
Mixture (input)
Prompt

Preserve the chime ringing, exclude the speakers, and keep the door opening and closing sounds.

Single-Channel Extraction

The same text-guided extraction from a single microphone, with no spatial cues at all. Separation is driven purely by the semantic attributes in the prompt (gender, relative loudness, noise identity).

Example 1
Mixture (input)
Prompt

Maintain the female, continue with the motorboat engine sound, and exclude the cat's meowing.

Example 2
Mixture (input)
Prompt

Kindly eliminate the guy.

Example 3
Mixture (input)
Prompt

Maintain the presence of the person playing a musical instrument, keep the water splashing and gurgling sounds, and eliminate both the man weak speaker and the female loud speaker.

Example 4
Mixture (input)
Prompt

Eliminate the sound of the race car accelerating and revving, as well as the female.

Example 5
Mixture (input)
Prompt

Kindly eliminate the guy speaker and the steady rhythm of the acoustic guitar.

Example 6
Mixture (input)
Prompt

Maintain the Drum and Bass music in the background and continue featuring the guy.

Spatially-Guided Extraction

Here the prompt describes a speaker by relative position rather than voice attributes — e.g. “the voice closer to the array” or “the speaker on the left.” The model resolves the relative spatial relation between the two talkers.

Example 1
Mixture (input)
Prompt

Isolate the speaker found to the left side

Example 2
Mixture (input)
Prompt

Separate the speaker situated toward the left

Example 3
Mixture (input)
Prompt

Extract the speaker situated toward the left

Example 4
Mixture (input)
Prompt

Keep the voice who is farther away and oriented more to the right

Example 5
Mixture (input)
Prompt

Isolate the voice that is nearer and facing more leftward

Multi-Channel Enhancement

A single target speaker buried in heavy background noise, recorded on two microphones. The prompt simply asks to keep the speaker / remove the noise; enhancement is handled as a special case of the same extraction model. These examples were chosen for their very low input SNR to show the denoising under hard conditions.

Example 1input SNR −6.2 dB
Mixture (input)
Prompt

Remove all background noise and preserve only the primary speaker.

Example 2input SNR −1.6 dB
Mixture (input)
Prompt

Extract the human voice and discard any background noise.

Example 3input SNR −1.3 dB
Mixture (input)
Prompt

Clean this clip by removing all noise and retaining only the human voice.

Example 4input SNR −0.9 dB
Mixture (input)
Prompt

Isolate the vocal track and eliminate unwanted noise such as traffic, people talking, or machinery.

Example 5input SNR −0.9 dB
Mixture (input)
Prompt

Suppress noise and enhance the clarity of the vocal track.

Single-Channel Enhancement

The same denoising from a single microphone. On this task Textraction surpasses the dedicated HD-DEMUCS-TF baseline, despite being a general text-guided extractor. Again, the noisiest inputs were picked on purpose.

Example 1input SNR −5.7 dB
Mixture (input)
Prompt

Remove background chatter and retain only the speaker.

Example 2input SNR −1.6 dB
Mixture (input)
Prompt

Remove background chatter and retain only the main voice.

Example 3input SNR −0.7 dB
Mixture (input)
Prompt

Denoise this recording so that only the speaker remains clearly audible.

Example 4input SNR −0.7 dB
Mixture (input)
Prompt

Suppress noise and enhance the clarity of the primary speaker.

Example 5input SNR −0.6 dB
Mixture (input)
Prompt

Keep only the human voice and eliminate everything else.