Diffusers documentation
Kandinsky S2V 1.0
Kandinsky S2V 1.0
Kandinsky S2V 1.0 根据参考帧和语音轨道生成头像视频。帧中的人物会被动画化以匹配语音——整个身体都会动起来，而不仅仅是头部。语音是条件输入，而非输出：模型不会生成音频，最终结果的音轨就是您传入的轨道。
两种条件均通过 Qwen2.5-Omni-7B 处理： 其隐藏状态编码提示词，音频塔编码语音，语音通过逐帧对齐的交叉注意力适配器进入 Transformer，因此每个潜在帧仅关注其覆盖的语音。片段以 5 秒（25 fps 下为 125 帧）为单位生成，每个片段携带前序片段的 FramePack 记忆，从而确保较长的音轨在片段边界处保持一致性。
Kandinsky S2V 1.0 仅以模块化流水线的形式集成；它没有标准的DiffusionPipeline类。
检查点
蒸馏检查点将 PiflowScheduler 配置为 shift=10.0 和 n_grid=5，而非其默认值。
ModularPipeline.from_pretrained 会从检查点中选取各个模块。各步骤的默认值如上所述。Classifier-free guidance 是 guider 组件，而非调用参数：基础检查点中该值默认为 3.0。蒸馏后的检查点不包含 guider，也没有 negative_prompt，因为 guidance 已经内化到学生模型中。
用法
import librosa
import torch
from diffusers import ModularPipeline
from diffusers.utils import encode_video, load_image
pipe = ModularPipeline.from_pretrained("kandinskylab/Kandinsky-S2V-1.0-Lite-distill-Diffusers")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")
image = load_image("reference.png")
audio, _ = librosa.load("speech.wav", sr=16000) # a mono waveform at 16 kHz, for the model
speech, sample_rate = librosa.load("speech.wav", sr=None) # the original track, added to the output video
video = pipe(image=image, audio=audio, output="videos")[0]
encode_video(
video,
fps=25,
output_path="avatar.mp4",
audio=torch.from_numpy(speech)[None],
audio_sample_rate=sample_rate,
)当所有组件都位于设备上时，Pro 检查点无法与 7B 语音和文本编码器同时放入 80 GB 的 GPU 中。请通过 ComponentsManager 将它们卸载：
import librosa
import torch
from diffusers import ComponentsManager, ModularPipeline
from diffusers.utils import encode_video, load_image
manager = ComponentsManager()
pipe = ModularPipeline.from_pretrained("kandinskylab/Kandinsky-S2V-1.0-Pro-Diffusers", components_manager=manager)
pipe.load_components(dtype=torch.bfloat16)
manager.enable_auto_cpu_offload(device="cuda", memory_reserve_margin="40GiB")
image = load_image("reference.png")
audio, _ = librosa.load("speech.wav", sr=16000)
speech, sample_rate = librosa.load("speech.wav", sr=None)
video = pipe(image=image, audio=audio, output="videos")[0]
encode_video(
video,
fps=25,
output_path="avatar.mp4",
audio=torch.from_numpy(speech)[None],
audio_sample_rate=sample_rate,
)margin 是指模型移至 GPU 时，在权重之外预留的内存。
它需要容纳一个数据块的 VAE 解码，在 512x768 分辨率下约为 33 GiB：如果使用默认的 3 GB，transformer 会在解码期间一直驻留在 GPU 上，从而导致内存耗尽。
如果设置为 40 GiB，在 80 GB 的 GPU 上峰值约为 50 GiB。请在 load_components 之后启用 offload，以便其应用于每个组件。
备注
image是一个 PIL 图像，audio是一个 16 kHz 的单声道浮点波形，取值范围为 [-1, 1]，以张量或形状为(samples,)的数组形式提供；该 pipeline 不会读取文件。- 顶层块（
text_encoder、vae_encoder、denoise、decode）也可以独立运行； 单独运行每个块时，请传入相同的num_latent_frames和fps。 - 除非传入
num_chunks，否则分块数量取决于语音轨道的长度。无论如何，视频都会被裁剪至与轨道长度一致，因此超出其末尾的分块会被计算并丢弃，同时会发出警告。 - 在引导机制下，无条件分支会替换为负向提示词 并 静音语音，因此引导是联合作用于文本和语音的。
KandinskyS2VModularPipeline
class diffusers.KandinskyS2VModularPipeline
< source >( blocks: diffusers.modular_pipelines.modular_pipeline.ModularPipelineBlocks | None = Nonepretrained_model_name_or_path: str | os.PathLike | None = Nonecomponents_manager: diffusers.modular_pipelines.components_manager.ComponentsManager | None = Nonecollection: str | None = Noneworkflow: str | None = Nonemodular_config_dict: dict[str, typing.Any] | None = Noneconfig_dict: dict[str, typing.Any] | None = None**kwargs )
用于 Kandinsky S2V 1.0 语音到视频生成的 ModularPipeline。
KandinskyS2VAutoBlocks
基于 Kandinsky S2V 1.0 的模块化语音转视频生成流水线：根据参考帧和语音轨道，逐块生成虚拟形象视频。
组件：
text_encoder (Qwen2_5OmniThinkerForConditionalGeneration) tokenizer (Qwen2_5OmniProcessor) text_encoder_2
(CLIPTextModel) tokenizer_2 (CLIPTokenizer) vae (AutoencoderKLHunyuanVideo) video_processor
(VideoProcessor) transformer (KandinskyS2VTransformer3DModel) scheduler
(FlowMatchEulerDiscreteScheduler) guider (ClassifierFreeGuidance)
Inputs:
prompt (str, optional, defaults to The video presents a stable medium close-up of a subject facing the
camera directly. The subject exhibits natural, lifelike body dynamics: the head and shoulders move fluidly
with the rhythm of speech. The lighting is even, maintaining a balanced natural exposure to clearly define
facial features without washout. Throughout the sequence, the subject engages in continuous speech marked by
precise, high-fidelity articulation. The lips move in a rhythmic, natural pattern: alternating between brief
firm closures and relaxed openings. During these openings, the tips of the upper teeth are momentarily and
realistically visible, exhibiting natural enamel texture. The upper lip lifts subtly over the teeth during
enunciation, while the jaw performs minimal vertical oscillations synchronized with the speech cadence.
Facial micro-expressions are active yet restrained; eyebrows rise and lower slightly with minute asymmetry.
The eyes maintain a steady, engaged gaze forward, punctuated by smooth, deliberate blinks. ):
The prompt to guide the generation.
negative_prompt (str, optional):
The prompt for the unconditional pass of the guidance. Defaults to DEFAULT_NEGATIVE_PROMPT when the
guidance needs one.
audio (Tensor | ndarray):
The speech track that drives the generation: a mono float waveform in [-1, 1] at 16000 Hz, shaped
(samples,).
num_latent_frames (int, optional, defaults to 32):
Latent frames per chunk; 32 gives 125 frames (5 s at 25 fps).
fps (int, optional, defaults to 25):
Frame rate the chunk length is computed at.
num_chunks (int, optional):
Number of chunks to generate. Defaults to however many cover the speech track; the video is trimmed to
the track either way.
image (Image):
The reference frame to animate.
height (int, optional):
The height in pixels of the generated video. Defaults to the 512 bucket closest to the reference frame.
width (int, optional):
The width in pixels of the generated video, chosen together with height.
generator (Generator, optional):
Torch generator for deterministic generation.
num_inference_steps (int, optional, defaults to 50):
The number of denoising steps.
scale_factor (tuple, optional, defaults to (1.0, 2.0, 2.0)):
Per-axis RoPE scale for the resolution bucket.
output_type (str, optional, defaults to pil):
Output format: ‘pil’, ‘np’, ‘pt’.
输出：
videos (list)：
生成的视频。
KandinskyS2VDistilledModularPipeline
class diffusers.KandinskyS2VDistilledModularPipeline
< source >( blocks: diffusers.modular_pipelines.modular_pipeline.ModularPipelineBlocks | None = Nonepretrained_model_name_or_path: str | os.PathLike | None = Nonecomponents_manager: diffusers.modular_pipelines.components_manager.ComponentsManager | None = Nonecollection: str | None = Noneworkflow: str | None = Nonemodular_config_dict: dict[str, typing.Any] | None = Noneconfig_dict: dict[str, typing.Any] | None = None**kwargs )
用于 pi-Flow 蒸馏版 Kandinsky S2V 1.0 检查点的 ModularPipeline。
Guidance 被蒸馏到学生模型中，因此其块没有 guider，也没有负面提示，每个片段只需执行一次前向传播。
KandinskyS2VDistilledAutoBlocks
使用 pi-Flow 蒸馏的 Kandinsky S2V 1.0 检查点构建的模块化语音转视频生成流水线：每个块包含 16 个片段且不使用引导，该设置被蒸馏到学生模型中。
组件：
text_encoder (Qwen2_5OmniThinkerForConditionalGeneration) tokenizer (Qwen2_5OmniProcessor) text_encoder_2
(CLIPTextModel) tokenizer_2 (CLIPTokenizer) vae (AutoencoderKLHunyuanVideo) video_processor
(VideoProcessor) transformer (KandinskyS2VTransformer3DModel) scheduler (PiflowScheduler)
Inputs:
prompt (str, optional, defaults to The video presents a stable medium close-up of a subject facing the
camera directly. The subject exhibits natural, lifelike body dynamics: the head and shoulders move fluidly
with the rhythm of speech. The lighting is even, maintaining a balanced natural exposure to clearly define
facial features without washout. Throughout the sequence, the subject engages in continuous speech marked by
precise, high-fidelity articulation. The lips move in a rhythmic, natural pattern: alternating between brief
firm closures and relaxed openings. During these openings, the tips of the upper teeth are momentarily and
realistically visible, exhibiting natural enamel texture. The upper lip lifts subtly over the teeth during
enunciation, while the jaw performs minimal vertical oscillations synchronized with the speech cadence.
Facial micro-expressions are active yet restrained; eyebrows rise and lower slightly with minute asymmetry.
The eyes maintain a steady, engaged gaze forward, punctuated by smooth, deliberate blinks. ):
The prompt to guide the generation.
audio (Tensor | ndarray):
The speech track that drives the generation: a mono float waveform in [-1, 1] at 16000 Hz, shaped
(samples,).
num_latent_frames (int, optional, defaults to 32):
Latent frames per chunk; 32 gives 125 frames (5 s at 25 fps).
fps (int, optional, defaults to 25):
Frame rate the chunk length is computed at.
num_chunks (int, optional):
Number of chunks to generate. Defaults to however many cover the speech track; the video is trimmed to
the track either way.
image (Image):
The reference frame to animate.
height (int, optional):
The height in pixels of the generated video. Defaults to the 512 bucket closest to the reference frame.
width (int, optional):
The width in pixels of the generated video, chosen together with height.
generator (Generator, optional):
Torch generator for deterministic generation.
num_inference_steps (int, optional, defaults to 16):
The number of denoising steps.
scale_factor (tuple, optional, defaults to (1.0, 2.0, 2.0)):
Per-axis RoPE scale for the resolution bucket.
output_type (str, optional, defaults to pil):
Output format: ‘pil’, ‘np’, ‘pt’.
输出：
videos (list)：
生成的视频。