Falcon

Causal audio-visual target speaker extraction model from the paper: a single-pass multi-scale separator trained with an encoder-space speaker contrastive loss (0.5 M trainable parameters).

Code, training and evaluation: https://github.com/JusperLee/Falcon. Paper: Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes (NeurIPS 2026), Wendi Sang*, Kai Li*, Yifan Li, Jianqiang Huang, Xiaolin Hu (*equal contribution).

File	Config in the code repo	Audio network	Total params
falcon.pt	configs/Falcon.yml	FalconSpeakerContrastive	11.72 M

Total parameters include the frozen 11.2 M-parameter FRCNN lip-reading encoder. The file holds only state_dict (audio_model.*, video_model.*), the architecture name and the training epoch; load it with torch.load(..., weights_only=True). config.yaml is a copy of configs/Falcon.yml from the code repo; the Hub counts downloads by this file, and FalconRuntime requests it on every load.

Usage

git clone https://github.com/JusperLee/Falcon.git
cd Falcon/studio && pip install -r requirements.txt

import numpy as np
from falcon_runtime import FalconRuntime

model = FalconRuntime.from_pretrained("JusperLee/Falcon")      # downloads falcon.pt on first use
audio = np.zeros(32000, dtype=np.float32)              # 2 s mono 16 kHz mixture
mouth = np.zeros((50, 96, 96), dtype=np.uint8)         # 25 fps grayscale mouth crops of the target speaker
target = model.infer(audio, mouth)                     # (32000,) target speech

Whole clips: python infer_file.py --audio mix.wav --mouth mouth.npy --out target.wav --model JusperLee/Falcon. Live separation in the browser and in Zoom / Tencent Meeting: see studio/README.md in the code repo.

Input: mono 16 kHz audio and synchronized 25 fps grayscale 96×96 mouth crops (center-cropped to 88×88 and normalized inside the runtime).

License

Apache License 2.0, the same as the code repository.