Hugging Face's logo

Diffusers documentation

Helios

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v0.42.1).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

LoRA

Helios

来自北京大学、字节跳动等机构的 Helios：真正的实时长视频生成模型，作者为 Shenghai Yuan、Yuanyang Yin、Zongjian Li、Xinwei Huang、Xiao Yang、Li Yuan。

  • 我们推出了 Helios，这是首个在单张 NVIDIA H100 GPU 上以 17 FPS 运行、支持分钟级生成且质量与强大基线相当的 14B 视频生成模型。 我们在三个关键维度上取得了突破：(1) 在不使用自强制、误差库或关键帧采样等常用抗漂移启发式方法的情况下，实现了对长视频漂移的鲁棒性；(2) 在不使用 KV-cache、因果掩码或稀疏注意力等标准加速技术的情况下，实现了实时生成；以及 (3) 在不使用并行或分片框架的情况下进行训练，从而在将最多四个 14B 模型装入 80 GB GPU 显存的同时，实现了图像扩散规模的批量大小。具体而言，Helios 是一个 14B 自回归扩散模型，采用统一的输入表示，原生支持 T2V、I2V 和 V2V 任务。为了缓解长视频生成中的漂移问题，我们刻画了其典型的失效模式，并提出了简单而有效的训练策略，在训练过程中显式模拟漂移，同时从根源上消除重复运动。在效率方面，我们对历史和噪声上下文进行了大幅压缩，并减少了采样步数，使得计算成本与 1.3B 视频生成模型相当甚至更低。此外，我们引入了基础设施层面的优化，在加速推理和训练的同时降低了内存消耗。大量实验表明，Helios 在短视频和长视频生成上均持续优于以往方法。所有代码和模型均可在 [this https URL](https://pku-yuangroup.github.io/Helios-Page) 获取。

以下 Helios 模型在 Diffusers 中受支持：

  • Helios-Base：最佳质量，采用 v-prediction、标准 CFG 以及自定义的 HeliosScheduler。
  • Helios-Mid：中等权重，采用 v-prediction、CFG-Zero* 以及自定义的 HeliosScheduler。
  • Helios-Distilled：效率最佳，采用 x0-prediction 和自定义的 HeliosDMDScheduler。

点击右侧边栏中的 Helios 模型，以查看更多视频生成示例。

优化内存和推理速度

下面的示例演示了如何生成针对内存或推理速度进行优化的文本到视频。

内存
推理速度

有关各种内存节省技术的详细信息，请参阅 减少内存使用 指南。

下方的 Helios 模型需要约 6GB 的显存。

import torch
from diffusers import AutoModel, HeliosPipeline
from diffusers.hooks.group_offloading import apply_group_offloading
from diffusers.utils import export_to_video

vae = AutoModel.from_pretrained("BestWishYsh/Helios-Base", subfolder="vae", dtype=torch.float32)

# group-offloading
pipeline = HeliosPipeline.from_pretrained(
    "BestWishYsh/Helios-Base",
    vae=vae,
    dtype=torch.bfloat16
)
pipeline.enable_group_offload(
    onload_device=torch.device("cuda"),
    offload_device=torch.device("cpu"),
    offload_type="leaf_level",
    use_stream=True,
    record_stream=True,
)

prompt = """
A vibrant tropical fish swimming gracefully among colorful coral reefs in a clear, turquoise ocean. The fish has bright blue 
and yellow scales with a small, distinctive orange spot on its side, its fins moving fluidly. The coral reefs are alive with 
a variety of marine life, including small schools of colorful fish and sea turtles gliding by. The water is crystal clear, 
allowing for a view of the sandy ocean floor below. The reef itself is adorned with a mix of hard and soft corals in shades 
of red, orange, and green. The photo captures the fish from a slightly elevated angle, emphasizing its lively movements and 
the vivid colors of its surroundings. A close-up shot with dynamic movement.
"""
negative_prompt = """
Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality,
low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured,
misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards
"""

output = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_frames=99,
    num_inference_steps=50,
    guidance_scale=5.0,
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]
export_to_video(output, "helios_base_t2v_output.mp4", fps=24)

使用 Helios-Base 进行生成

下面的示例演示了如何使用 Helios-Base 根据文本、图像或视频生成视频。

用法
import torch
from diffusers import AutoModel, HeliosPipeline
from diffusers.utils import export_to_video, load_video, load_image

vae = AutoModel.from_pretrained("BestWishYsh/Helios-Base", subfolder="vae", dtype=torch.float32)

pipeline = HeliosPipeline.from_pretrained(
    "BestWishYsh/Helios-Base",
    vae=vae,
    dtype=torch.bfloat16
)
pipeline.to("cuda")  # or "mps", "xpu", "cpu"

negative_prompt = """
Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality,
low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured,
misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards
"""

# For Text-to-Video
prompt = """
A vibrant tropical fish swimming gracefully among colorful coral reefs in a clear, turquoise ocean. The fish has bright blue 
and yellow scales with a small, distinctive orange spot on its side, its fins moving fluidly. The coral reefs are alive with 
a variety of marine life, including small schools of colorful fish and sea turtles gliding by. The water is crystal clear, 
allowing for a view of the sandy ocean floor below. The reef itself is adorned with a mix of hard and soft corals in shades 
of red, orange, and green. The photo captures the fish from a slightly elevated angle, emphasizing its lively movements and 
the vivid colors of its surroundings. A close-up shot with dynamic movement.
"""

output = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_frames=99,
    num_inference_steps=50,
    guidance_scale=5.0,
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]
export_to_video(output, "helios_base_t2v_output.mp4", fps=24)

# For Image-to-Video
prompt = """
A towering emerald wave surges forward, its crest curling with raw power and energy. Sunlight glints off the translucent water, 
illuminating the intricate textures and deep green hues within the wave’s body. A thick spray erupts from the breaking crest, 
casting a misty veil that dances above the churning surface. As the perspective widens, the immense scale of the wave becomes 
apparent, revealing the restless expanse of the ocean stretching beyond. The scene captures the ocean’s untamed beauty and 
relentless force, with every droplet and ripple shimmering in the light. The dynamic motion and vivid colors evoke both awe and 
respect for nature’s might.
"""
image_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/helios/wave.jpg"

output = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    image=load_image(image_path).resize((640, 384)),
    num_frames=99,
    num_inference_steps=50,
    guidance_scale=5.0,
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]
export_to_video(output, "helios_base_i2v_output.mp4", fps=24)

# For Video-to-Video
prompt = """
A bright yellow Lamborghini Huracn Tecnica speeds along a curving mountain road, surrounded by lush green trees 
under a partly cloudy sky. The car's sleek design and vibrant color stand out against the natural backdrop, 
emphasizing its dynamic movement. The road curves gently, with a guardrail visible on one side, adding depth to 
the scene. The motion blur captures the sense of speed and energy, creating a thrilling and exhilarating atmosphere. 
A front-facing shot from a slightly elevated angle, highlighting the car's aggressive stance and the surrounding greenery.
"""
video_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/helios/car.mp4"

output = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    video=load_video(video_path),
    num_frames=99,
    num_inference_steps=50,
    guidance_scale=5.0,
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]
export_to_video(output, "helios_base_v2v_output.mp4", fps=24)

使用 Helios-Mid 进行生成

Helios-Mid 的以下示例展示了如何基于文本、图像或视频生成视频。

用法
import torch
from diffusers import AutoModel, HeliosPyramidPipeline
from diffusers.utils import export_to_video, load_video, load_image

vae = AutoModel.from_pretrained("BestWishYsh/Helios-Mid", subfolder="vae", dtype=torch.float32)

pipeline = HeliosPyramidPipeline.from_pretrained(
    "BestWishYsh/Helios-Mid",
    vae=vae,
    dtype=torch.bfloat16
)
pipeline.to("cuda")  # or "mps", "xpu", "cpu"

negative_prompt = """
Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality,
low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured,
misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards
"""

# For Text-to-Video
prompt = """
A vibrant tropical fish swimming gracefully among colorful coral reefs in a clear, turquoise ocean. The fish has bright blue 
and yellow scales with a small, distinctive orange spot on its side, its fins moving fluidly. The coral reefs are alive with 
a variety of marine life, including small schools of colorful fish and sea turtles gliding by. The water is crystal clear, 
allowing for a view of the sandy ocean floor below. The reef itself is adorned with a mix of hard and soft corals in shades 
of red, orange, and green. The photo captures the fish from a slightly elevated angle, emphasizing its lively movements and 
the vivid colors of its surroundings. A close-up shot with dynamic movement.
"""

output = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_frames=99,
    pyramid_num_inference_steps_list=[20, 20, 20],
    guidance_scale=5.0,
    use_zero_init=True,
    zero_steps=1,
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]
export_to_video(output, "helios_pyramid_t2v_output.mp4", fps=24)

# For Image-to-Video
prompt = """
A towering emerald wave surges forward, its crest curling with raw power and energy. Sunlight glints off the translucent water, 
illuminating the intricate textures and deep green hues within the wave’s body. A thick spray erupts from the breaking crest, 
casting a misty veil that dances above the churning surface. As the perspective widens, the immense scale of the wave becomes 
apparent, revealing the restless expanse of the ocean stretching beyond. The scene captures the ocean’s untamed beauty and 
relentless force, with every droplet and ripple shimmering in the light. The dynamic motion and vivid colors evoke both awe and 
respect for nature’s might.
"""
image_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/helios/wave.jpg"

output = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    image=load_image(image_path).resize((640, 384)),
    num_frames=99,
    pyramid_num_inference_steps_list=[20, 20, 20],
    guidance_scale=5.0,
    use_zero_init=True,
    zero_steps=1,
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]
export_to_video(output, "helios_pyramid_i2v_output.mp4", fps=24)

# For Video-to-Video
prompt = """
A bright yellow Lamborghini Huracn Tecnica speeds along a curving mountain road, surrounded by lush green trees 
under a partly cloudy sky. The car's sleek design and vibrant color stand out against the natural backdrop, 
emphasizing its dynamic movement. The road curves gently, with a guardrail visible on one side, adding depth to 
the scene. The motion blur captures the sense of speed and energy, creating a thrilling and exhilarating atmosphere. 
A front-facing shot from a slightly elevated angle, highlighting the car's aggressive stance and the surrounding greenery.
"""
video_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/helios/car.mp4"

output = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    video=load_video(video_path),
    num_frames=99,
    pyramid_num_inference_steps_list=[20, 20, 20],
    guidance_scale=5.0,
    use_zero_init=True,
    zero_steps=1,
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]
export_to_video(output, "helios_pyramid_v2v_output.mp4", fps=24)

使用 Helios-Distilled 进行生成

下面的示例演示了如何使用 Helios-Distilled 根据文本、图像或视频生成视频。

用法
import torch
from diffusers import AutoModel, HeliosPyramidPipeline
from diffusers.utils import export_to_video, load_video, load_image

vae = AutoModel.from_pretrained("BestWishYsh/Helios-Distilled", subfolder="vae", dtype=torch.float32)

pipeline = HeliosPyramidPipeline.from_pretrained(
    "BestWishYsh/Helios-Distilled",
    vae=vae,
    dtype=torch.bfloat16
)
pipeline.to("cuda")  # or "mps", "xpu", "cpu"

negative_prompt = """
Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality,
low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured,
misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards
"""

# For Text-to-Video
prompt = """
A vibrant tropical fish swimming gracefully among colorful coral reefs in a clear, turquoise ocean. The fish has bright blue 
and yellow scales with a small, distinctive orange spot on its side, its fins moving fluidly. The coral reefs are alive with 
a variety of marine life, including small schools of colorful fish and sea turtles gliding by. The water is crystal clear, 
allowing for a view of the sandy ocean floor below. The reef itself is adorned with a mix of hard and soft corals in shades 
of red, orange, and green. The photo captures the fish from a slightly elevated angle, emphasizing its lively movements and 
the vivid colors of its surroundings. A close-up shot with dynamic movement.
"""

output = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_frames=240,
    pyramid_num_inference_steps_list=[2, 2, 2],
    guidance_scale=1.0,
    is_amplify_first_chunk=True,
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]
export_to_video(output, "helios_distilled_t2v_output.mp4", fps=24)

# For Image-to-Video
prompt = """
A towering emerald wave surges forward, its crest curling with raw power and energy. Sunlight glints off the translucent water, 
illuminating the intricate textures and deep green hues within the wave’s body. A thick spray erupts from the breaking crest, 
casting a misty veil that dances above the churning surface. As the perspective widens, the immense scale of the wave becomes 
apparent, revealing the restless expanse of the ocean stretching beyond. The scene captures the ocean’s untamed beauty and 
relentless force, with every droplet and ripple shimmering in the light. The dynamic motion and vivid colors evoke both awe and 
respect for nature’s might.
"""
image_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/helios/wave.jpg"

output = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    image=load_image(image_path).resize((640, 384)),
    num_frames=240,
    pyramid_num_inference_steps_list=[2, 2, 2],
    guidance_scale=1.0,
    is_amplify_first_chunk=True,
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]
export_to_video(output, "helios_distilled_i2v_output.mp4", fps=24)

# For Video-to-Video
prompt = """
A bright yellow Lamborghini Huracn Tecnica speeds along a curving mountain road, surrounded by lush green trees 
under a partly cloudy sky. The car's sleek design and vibrant color stand out against the natural backdrop, 
emphasizing its dynamic movement. The road curves gently, with a guardrail visible on one side, adding depth to 
the scene. The motion blur captures the sense of speed and energy, creating a thrilling and exhilarating atmosphere. 
A front-facing shot from a slightly elevated angle, highlighting the car's aggressive stance and the surrounding greenery.
"""
video_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/helios/car.mp4"

output = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    video=load_video(video_path),
    num_frames=240,
    pyramid_num_inference_steps_list=[2, 2, 2],
    guidance_scale=1.0,
    is_amplify_first_chunk=True,
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]
export_to_video(output, "helios_distilled_v2v_output.mp4", fps=24)

文本到视频展示

提示生成的视频
一位维京战士驾驶着一辆载满乘客的现代城市公交车。这位维京人留着向后扎起的金色长发，蓄着胡须，头戴毛皮镶边的头盔，身披铠甲。他身穿传统的束腰外衣和长裤，但同时也系着安全带，全神贯注地穿梭于繁忙的街道。公交车内部陈设典型，一排排座位上坐满了正在度过日常生活的各色乘客。外部镜头展示了熙熙攘攘的城市环境，包括高楼大厦和车流。中景镜头聚焦于手握方向盘的维京人，并偶尔穿插他坚毅表情的特写。
一段纪录片风格的自然摄影画面，由一辆向左行驶的车载摄像机拍摄，捕捉到一只螃蟹迅速爬入其洞穴的瞬间。这只螃蟹拥有坚硬、呈绿褐色的甲壳和长长的螯，以坚定的速度在沙地上移动。当它钻入沙中时，身体微微拱起，身后留下一条细小的痕迹。背景是一片浅滩，散落着岩石和贝壳，地平线处可见海岸线柔和的弧度。照片具有自然、真实的质感，突出了螃蟹的自然动态以及沙子的纹理。这是一个从略微俯视角度拍摄的近景镜头。

图像转视频展示

图像提示生成的视频
一辆时尚的红色起亚汽车在阴云密布的天空下沿着乡间公路疾驰，周围田野和树木向远处延伸，其模糊的动态效果凸显了车辆的现代设计与动感姿态。车身光亮的漆面反射着阴沉的天空，突显出其流线型轮廓和运动化的姿态。车牌上写着“KIA 626”，车辆的前大灯已开启，进一步增强了运动感和活力。道路平缓弯曲，车辆略微偏离中心位置，营造出向前冲的动势。动态的前四分之三视角捕捉到了车辆在连绵起伏的丘陵和零星树木这一宁静背景下的强劲气场。
特写镜头捕捉到一只毛茸茸的橙色猫咪，它有着醒目的绿色眼睛和白色胡须，正专注地凝视着镜头。猫咪的毛发柔软且打理得十分整洁，呈现出温暖的橙色与奶油色的混合色调。它那双大而富有表现力的眼睛是鲜亮的绿色，流露出好奇与警觉。猫咪的鼻子小巧粉嫩，嘴巴微微张开，隐约露出粉色的舌头。背景柔和虚化，暗示着一个色调中性的舒适室内环境。照片采用了浅景深，焦点清晰地落在猫咪的面部，而背景则保持模糊。这是一个从略微俯视角度拍摄的特写镜头。

交互式视频展示

提示生成的视频
提示词可以在此处找到
提示词可在此处找到

资源

通过以下资源进一步了解 Helios。

HeliosPipeline

class diffusers.HeliosPipeline

< >

( tokenizer: AutoTokenizertext_encoder: UMT5EncoderModelvae: AutoencoderKLWanscheduler: HeliosSchedulertransformer: HeliosTransformer3DModel )

参数

  • tokenizer (T5Tokenizer) — 来自 T5 的分词器， 具体为 google/umt5-xxl 变体。
  • text_encoder（T5EncoderModel）—— T5，具体为 google/umt5-xxl 变体。
  • transformer (HeliosTransformer3DModel) — 用于对输入潜变量进行去噪的条件 Transformer。
  • scheduler (HeliosScheduler) — 用于与 transformer 配合，对编码后的图像潜变量进行去噪的调度器。
  • vae (AutoencoderKLWan) — 变分自编码器（VAE）模型，用于将视频编码为潜在表示，以及从潜在表示解码为视频。

使用 Helios 进行文本生成视频 / 图像生成视频 / 视频生成视频的 Pipeline。

该模型继承自 DiffusionPipeline。有关为所有 pipeline 实现的通用方法（下载、保存、在特定设备上运行等），请参阅父类文档。

__call__

< >

( prompt: str | list[str] = Nonenegative_prompt: str | list[str] = Noneheight: int = 384width: int = 640num_frames: int = 132num_inference_steps: int = 50sigmas: list = Noneguidance_scale: float = 5.0num_videos_per_prompt: int | None = 1generator: typing.Union[torch.Generator, list[torch.Generator], NoneType] = Nonelatents: typing.Optional[torch.Tensor] = Noneprompt_embeds: typing.Optional[torch.Tensor] = Nonenegative_prompt_embeds: typing.Optional[torch.Tensor] = Noneoutput_type: str | None = 'np'return_dict: bool = Trueattention_kwargs: dict[str, typing.Any] | None = Nonecallback_on_step_end: typing.Union[typing.Callable[[int, int], NoneType], diffusers.callbacks.PipelineCallback, diffusers.callbacks.MultiPipelineCallbacks, NoneType] = Nonecallback_on_step_end_tensor_inputs: list = ['latents']max_sequence_length: int = 512image: typing.Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list[PIL.Image.Image], list[numpy.ndarray], list[torch.Tensor], NoneType] = Noneimage_latents: typing.Optional[torch.Tensor] = Nonefake_image_latents: typing.Optional[torch.Tensor] = Noneadd_noise_to_image_latents: bool = Trueimage_noise_sigma_min: float = 0.111image_noise_sigma_max: float = 0.135video: typing.Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list[PIL.Image.Image], list[numpy.ndarray], list[torch.Tensor], NoneType] = Nonevideo_latents: typing.Optional[torch.Tensor] = Noneadd_noise_to_video_latents: bool = Truevideo_noise_sigma_min: float = 0.111video_noise_sigma_max: float = 0.135history_sizes: list = [16, 2, 1]num_latent_frames_per_chunk: int = 9keep_first_frame: bool = Trueis_skip_first_chunk: bool = False ) → ~HeliosPipelineOutput or tuple

参数

  • prompt (str 或 list[str]，可选) — 用于引导图像生成的提示词或提示词列表。如果未定义，请改为传入 prompt_embeds。
  • negative_prompt (str 或 list[str]，可选) — 图像生成过程中需要避免的提示词。如果未定义，请改为传入 negative_prompt_embeds。在未使用引导（guidance_scale < 1）时将被忽略。
  • height (int，默认为 384) — 生成图像的高度（以像素为单位）。
  • width (int，默认为 640) — 生成图像的宽度（以像素为单位）。
  • num_frames (int，默认为 132) — 生成视频的帧数。
  • num_inference_steps (int，默认为 50) — 去噪步数。更多的去噪步数通常能生成更高质量的图像，但推理速度会变慢。
  • sigmas (list[float]，可选) — 用于去噪过程的自定义 sigmas。如果未定义，则使用调度器的默认 schedule。
  • guidance_scale (float，默认为 5.0) — Classifier-Free Diffusion Guidance 中定义的引导尺度。guidance_scale 定义为 Imagen Paper 中公式 2 的 w。通过设置 guidance_scale > 1 来启用引导尺度。较高的引导尺度会促使生成与文本 prompt 紧密关联的图像，但通常以牺牲图像质量为代价。
  • num_videos_per_prompt (int，可选，默认为 1) — 每个提示词生成的图像数量。
  • generator (torch.Generator 或 list[torch.Generator]，可选) — 用于使生成过程确定性的 torch.Generator。
  • latents (torch.Tensor, 可选) — 预先生成的、从高斯分布中采样的含噪潜变量，用作图像生成的输入。可用于以不同的提示词微调同一生成过程。如果未提供，则使用提供的随机generator进行采样，生成一个潜变量张量。
  • prompt_embeds (torch.Tensor, 可选) — 预生成的文本嵌入。可用于轻松调整文本输入（提示词加权）。如果未提供，则从 prompt 输入参数生成文本嵌入。
  • negative_prompt_embeds (torch.Tensor, 可选) — 预生成的负向文本嵌入。如果未提供，则从 negative_prompt 生成。
  • output_type (str，可选，默认为 "np") — 生成图像的输出格式。可选 PIL.Image 或 np.array。
  • return_dict (bool，可选，默认为 True) — 是否返回 HeliosPipelineOutput 而不是普通元组。
  • attention_kwargs (dict, 可选) — 一个 kwargs 字典，如果指定，将传递给 diffusers.models.attention_processor 中定义的 AttentionProcessor，该处理器在 self.processor 下定义。
  • callback_on_step_end (Callable、PipelineCallback、MultiPipelineCallbacks、可选) — 在推理过程中，每个去噪步骤结束时调用的函数，或 PipelineCallback 或 MultiPipelineCallbacks 的子类，其参数如下：callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)。callback_kwargs 将包含由 callback_on_step_end_tensor_inputs 指定的所有张量列表。
  • callback_on_step_end_tensor_inputs (list，可选) — callback_on_step_end 函数的张量输入列表。列表中指定的张量将作为 callback_kwargs 参数传递。您只能包含流水线类 ._callback_tensor_inputs 属性中列出的变量。
  • max_sequence_length (int，默认为 512) — 文本编码器的最大序列长度。如果提示词长于该长度，将被截断；如果短于该长度，将被填充至该长度。
  • image (PipelineImageInput，可选) — 用于图像到视频条件控制的输入图像。
  • image_latents (torch.Tensor, 可选) — 用于替代 image 的预编码图像潜变量。
  • fake_image_latents (torch.Tensor, 可选) — 用于条件生成过程中的可选伪图像潜变量。
  • add_noise_to_image_latents (bool，可选，默认为 True) — 是否在对图像潜变量进行去噪之前为其添加噪声。
  • image_noise_sigma_min (float，可选，默认为 0.111) — 添加到图像潜在表示的噪声的最小 sigma 值。
  • image_noise_sigma_max (float，可选，默认值为 0.135) — 添加到图像潜在表示的噪声的最大 sigma 值。
  • video (PipelineImageInput，可选) — 用于视频到视频条件控制的输入视频。
  • video_latents (torch.Tensor, 可选) — 用于替代 video 的预编码视频潜变量。
  • add_noise_to_video_latents (bool，可选，默认为 True) — 是否在对视频潜变量进行去噪之前添加噪声。
  • video_noise_sigma_min (float，可选，默认为 0.111) — 添加到视频潜变量的噪声的最小 sigma 值。
  • video_noise_sigma_max (float，可选，默认值为 0.135) — 添加到视频潜变量的噪声的最大 sigma 值。
  • history_sizes (list，可选，默认为 [16, 2, 1]) — 用于自回归分块生成的历史窗口大小。
  • num_latent_frames_per_chunk (int，可选，默认为 9) — 自回归生成过程中，每个 chunk 生成的潜在帧数量。
  • keep_first_frame (bool，可选，默认为 True) — 是否在各分块之间保留第一帧。
  • is_skip_first_chunk (bool，可选，默认为 False) — 是否跳过第一个块的生成。

返回值

~HeliosPipelineOutput 或 tuple

如果 return_dict 为 True，则返回 HeliosPipelineOutput；否则返回一个 tuple，其中第一个元素是包含生成图像的列表，第二个元素是 bool 列表，用于指示对应的生成图像是否包含“不适合工作场所”（nsfw）内容。

用于生成的 pipeline 的调用函数。

示例：

>>> import torch
>>> from diffusers.utils import export_to_video
>>> from diffusers import AutoencoderKLWan, HeliosPipeline

>>> # Available models: BestWishYsh/Helios-Base, BestWishYsh/Helios-Mid, BestWishYsh/Helios-Distilled
>>> model_id = "BestWishYsh/Helios-Base"
>>> vae = AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32)
>>> pipe = HeliosPipeline.from_pretrained(model_id, vae=vae, torch_dtype=torch.bfloat16)
>>> pipe.to("cuda")

>>> prompt = "A cat and a dog baking a cake together in a kitchen. The cat is carefully measuring flour, while the dog is stirring the batter with a wooden spoon. The kitchen is cozy, with sunlight streaming through the window."
>>> negative_prompt = "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards"

>>> output = pipe(
...     prompt=prompt,
...     negative_prompt=negative_prompt,
...     height=384,
...     width=640,
...     num_frames=132,
...     guidance_scale=5.0,
... ).frames[0]
>>> export_to_video(output, "output.mp4", fps=24)

encode_prompt

< >

( prompt: str | list[str]negative_prompt: str | list[str] | None = Nonedo_classifier_free_guidance: bool = Truenum_videos_per_prompt: int = 1prompt_embeds: typing.Optional[torch.Tensor] = Nonenegative_prompt_embeds: typing.Optional[torch.Tensor] = Nonemax_sequence_length: int = 226device: typing.Optional[torch.device] = Nonedtype: typing.Optional[torch.dtype] = None )

参数

  • prompt (str 或 list[str]，可选) — 要编码的提示
  • negative_prompt (str 或 list[str]，可选) — 用于不引导图像生成的提示词。如果未定义，则必须传入 negative_prompt_embeds。在未使用引导时会被忽略（即当 guidance_scale 小于 1 时忽略）。
  • do_classifier_free_guidance（bool，可选，默认为 True）— 是否使用 classifier free guidance。
  • num_videos_per_prompt (int，可选，默认为 1) — 每个提示词应生成的视频数量。用于放置结果嵌入的 torch 设备
  • prompt_embeds (torch.Tensor, 可选) — 预生成的文本嵌入。可用于轻松调整文本输入，例如 提示词加权。如果未提供，文本嵌入将从 prompt 输入参数生成。
  • negative_prompt_embeds (torch.Tensor, 可选) — 预生成的负向文本嵌入。可用于轻松调整文本输入，例如提示词加权。如果未提供，negative_prompt_embeds 将从 negative_prompt 输入参数生成。
  • device — (torch.device，可选)： torch 设备
  • dtype — (torch.dtype，可选)： torch 数据类型

将提示词编码为文本编码器的隐藏状态。

HeliosPyramidPipeline

class diffusers.HeliosPyramidPipeline

< >

( tokenizer: AutoTokenizertext_encoder: UMT5EncoderModelvae: AutoencoderKLWanscheduler: diffusers.schedulers.scheduling_helios.HeliosScheduler | diffusers.schedulers.scheduling_helios_dmd.HeliosDMDSchedulertransformer: HeliosTransformer3DModelis_cfg_zero_star: bool = Falseis_distilled: bool = False )

参数

  • tokenizer (T5Tokenizer) — 来自 T5 的分词器， 具体为 google/umt5-xxl 变体。
  • text_encoder（T5EncoderModel）—— T5，具体为 google/umt5-xxl 变体。
  • transformer (HeliosTransformer3DModel) — 用于对输入潜变量进行去噪的条件 Transformer。
  • scheduler ([HeliosScheduler, HeliosDMDScheduler]) — 用于与 transformer 配合，对编码后的图像潜变量进行去噪的调度器。
  • vae (AutoencoderKLWan) — 变分自编码器（VAE）模型，用于将视频编码为潜在表示，以及从潜在表示解码为视频。

使用 Helios 进行文本生成视频 / 图像生成视频 / 视频生成视频的 Pipeline。

该模型继承自 DiffusionPipeline。有关为所有 pipeline 实现的通用方法（下载、保存、在特定设备上运行等），请参阅父类文档。

__call__

< >

( prompt: str | list[str] = Nonenegative_prompt: str | list[str] = Noneheight: int = 384width: int = 640num_frames: int = 132sigmas: list = Noneguidance_scale: float = 5.0num_videos_per_prompt: int | None = 1generator: typing.Union[torch.Generator, list[torch.Generator], NoneType] = Nonelatents: typing.Optional[torch.Tensor] = Noneprompt_embeds: typing.Optional[torch.Tensor] = Nonenegative_prompt_embeds: typing.Optional[torch.Tensor] = Noneoutput_type: str | None = 'np'return_dict: bool = Trueattention_kwargs: dict[str, typing.Any] | None = Nonecallback_on_step_end: typing.Union[typing.Callable[[int, int], NoneType], diffusers.callbacks.PipelineCallback, diffusers.callbacks.MultiPipelineCallbacks, NoneType] = Nonecallback_on_step_end_tensor_inputs: list = ['latents']max_sequence_length: int = 512image: typing.Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list[PIL.Image.Image], list[numpy.ndarray], list[torch.Tensor], NoneType] = Noneimage_latents: typing.Optional[torch.Tensor] = Nonefake_image_latents: typing.Optional[torch.Tensor] = Noneadd_noise_to_image_latents: bool = Trueimage_noise_sigma_min: float = 0.111image_noise_sigma_max: float = 0.135video: typing.Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list[PIL.Image.Image], list[numpy.ndarray], list[torch.Tensor], NoneType] = Nonevideo_latents: typing.Optional[torch.Tensor] = Noneadd_noise_to_video_latents: bool = Truevideo_noise_sigma_min: float = 0.111video_noise_sigma_max: float = 0.135history_sizes: list = [16, 2, 1]num_latent_frames_per_chunk: int = 9keep_first_frame: bool = Trueis_skip_first_chunk: bool = Falsepyramid_num_inference_steps_list: list = [10, 10, 10]use_zero_init: bool | None = Truezero_steps: int | None = 1is_amplify_first_chunk: bool = False ) → ~HeliosPipelineOutput or tuple

参数

  • prompt (str 或 list[str]，可选) — 用于引导图像生成的提示词或提示词列表。如果未定义，请改为传入 prompt_embeds。
  • negative_prompt (str 或 list[str]，可选) — 图像生成过程中需要避免的提示词。如果未定义，请改为传入 negative_prompt_embeds。在未使用引导（guidance_scale < 1）时将被忽略。
  • height (int，默认为 384) — 生成图像的高度（以像素为单位）。
  • width (int，默认为 640) — 生成图像的宽度（以像素为单位）。
  • num_frames (int，默认为 132) — 生成视频的帧数。
  • sigmas (list[float]，可选) — 用于去噪过程的自定义 sigmas。如果未定义，则使用调度器的默认 schedule。
  • guidance_scale (float，默认为 5.0) — Classifier-Free Diffusion Guidance 中定义的引导尺度。guidance_scale 定义为 Imagen Paper 中公式 2 的 w。通过设置 guidance_scale > 1 来启用引导尺度。较高的引导尺度会促使生成与文本 prompt 紧密关联的图像，但通常以牺牲图像质量为代价。
  • num_videos_per_prompt (int，可选，默认为 1) — 每个提示词生成的图像数量。
  • generator (torch.Generator 或 list[torch.Generator]，可选) — 用于使生成过程确定性的 torch.Generator。
  • latents (torch.Tensor, 可选) — 预先生成的、从高斯分布中采样的含噪潜变量，用作图像生成的输入。可用于以不同的提示词微调同一生成过程。如果未提供，则使用提供的随机generator进行采样，生成一个潜变量张量。
  • prompt_embeds (torch.Tensor, 可选) — 预生成的文本嵌入。可用于轻松调整文本输入（提示词加权）。如果未提供，则从 prompt 输入参数生成文本嵌入。
  • negative_prompt_embeds (torch.Tensor, 可选) — 预生成的负向文本嵌入。如果未提供，则从 negative_prompt 生成。
  • output_type (str，可选，默认为 "np") — 生成图像的输出格式。可选 PIL.Image 或 np.array。
  • return_dict (bool，可选，默认为 True) — 是否返回 HeliosPipelineOutput 而不是普通元组。
  • attention_kwargs (dict, 可选) — 一个 kwargs 字典，如果指定，将传递给 diffusers.models.attention_processor 中定义的 AttentionProcessor，该处理器在 self.processor 下定义。
  • callback_on_step_end (Callable、PipelineCallback、MultiPipelineCallbacks、可选) — 在推理过程中，每个去噪步骤结束时调用的函数，或 PipelineCallback 或 MultiPipelineCallbacks 的子类，其参数如下：callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)。callback_kwargs 将包含由 callback_on_step_end_tensor_inputs 指定的所有张量列表。
  • callback_on_step_end_tensor_inputs (list，可选) — callback_on_step_end 函数的张量输入列表。列表中指定的张量将作为 callback_kwargs 参数传递。您只能包含流水线类 ._callback_tensor_inputs 属性中列出的变量。
  • max_sequence_length (int，默认为 512) — 文本编码器的最大序列长度。如果提示词长于该长度，将被截断；如果短于该长度，将被填充至该长度。
  • image (PipelineImageInput，可选) — 用于图像到视频条件控制的输入图像。
  • image_latents (torch.Tensor, 可选) — 用于替代 image 的预编码图像潜变量。
  • fake_image_latents (torch.Tensor, 可选) — 用于条件生成过程中的可选伪图像潜变量。
  • add_noise_to_image_latents (bool，可选，默认为 True) — 是否在对图像潜变量进行去噪之前为其添加噪声。
  • image_noise_sigma_min (float，可选，默认为 0.111) — 添加到图像潜在表示的噪声的最小 sigma 值。
  • image_noise_sigma_max (float，可选，默认值为 0.135) — 添加到图像潜在表示的噪声的最大 sigma 值。
  • video (PipelineImageInput，可选) — 用于视频到视频条件控制的输入视频。
  • video_latents (torch.Tensor, 可选) — 用于替代 video 的预编码视频潜变量。
  • add_noise_to_video_latents (bool，可选，默认为 True) — 是否在对视频潜变量进行去噪之前添加噪声。
  • video_noise_sigma_min (float，可选，默认为 0.111) — 添加到视频潜变量的噪声的最小 sigma 值。
  • video_noise_sigma_max (float，可选，默认值为 0.135) — 添加到视频潜变量的噪声的最大 sigma 值。
  • history_sizes (list，可选，默认为 [16, 2, 1]) — 用于自回归分块生成的历史窗口大小。
  • num_latent_frames_per_chunk (int，可选，默认为 9) — 自回归生成过程中，每个 chunk 生成的潜在帧数量。
  • keep_first_frame (bool，可选，默认为 True) — 是否在各分块之间保留第一帧。
  • is_skip_first_chunk (bool，可选，默认为 False) — 是否跳过第一个块的生成。
  • pyramid_num_inference_steps_list (list，可选，默认为 [10, 10, 10]) — 在 Stage 2 生成过程中，每个金字塔阶段的推理步数。
  • use_zero_init (bool，可选，默认为 True) — 是否在去噪开始时应用 CFG 零初始化。
  • zero_steps (int，可选，默认为 1) — 使用 CFG zero-init 的初始步数。
  • is_amplify_first_chunk (bool，可选，默认为 False) — 是否对第一个 chunk 放大 guidance（与 DMD 相关）。

返回值

~HeliosPipelineOutput 或 tuple

如果 return_dict 为 True，则返回 HeliosPipelineOutput；否则返回一个 tuple，其中第一个元素是包含生成图像的列表，第二个元素是 bool 列表，用于指示对应的生成图像是否包含“不适合工作场所”（nsfw）内容。

用于生成的 pipeline 的调用函数。

示例：

>>> import torch
>>> from diffusers.utils import export_to_video
>>> from diffusers import AutoencoderKLWan, HeliosPyramidPipeline

>>> # Available models: BestWishYsh/Helios-Base, BestWishYsh/Helios-Mid, BestWishYsh/Helios-Distilled
>>> model_id = "BestWishYsh/Helios-Base"
>>> vae = AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32)
>>> pipe = HeliosPyramidPipeline.from_pretrained(model_id, vae=vae, torch_dtype=torch.bfloat16)
>>> pipe.to("cuda")

>>> prompt = "A cat and a dog baking a cake together in a kitchen. The cat is carefully measuring flour, while the dog is stirring the batter with a wooden spoon. The kitchen is cozy, with sunlight streaming through the window."
>>> negative_prompt = "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards"

>>> output = pipe(
...     prompt=prompt,
...     negative_prompt=negative_prompt,
...     height=384,
...     width=640,
...     num_frames=132,
...     guidance_scale=5.0,
... ).frames[0]
>>> export_to_video(output, "output.mp4", fps=24)

encode_prompt

< >

( prompt: str | list[str]negative_prompt: str | list[str] | None = Nonedo_classifier_free_guidance: bool = Truenum_videos_per_prompt: int = 1prompt_embeds: typing.Optional[torch.Tensor] = Nonenegative_prompt_embeds: typing.Optional[torch.Tensor] = Nonemax_sequence_length: int = 226device: typing.Optional[torch.device] = Nonedtype: typing.Optional[torch.dtype] = None )

参数

  • prompt (str 或 list[str]，可选) — 要编码的提示
  • negative_prompt (str 或 list[str]，可选) — 用于不引导图像生成的提示词。如果未定义，则必须传入 negative_prompt_embeds。在未使用引导时会被忽略（即当 guidance_scale 小于 1 时忽略）。
  • do_classifier_free_guidance（bool，可选，默认为 True）— 是否使用 classifier free guidance。
  • num_videos_per_prompt (int，可选，默认为 1) — 每个提示词应生成的视频数量。用于放置结果嵌入的 torch 设备
  • prompt_embeds (torch.Tensor, 可选) — 预生成的文本嵌入。可用于轻松调整文本输入，例如 提示词加权。如果未提供，文本嵌入将从 prompt 输入参数生成。
  • negative_prompt_embeds (torch.Tensor, 可选) — 预生成的负向文本嵌入。可用于轻松调整文本输入，例如提示词加权。如果未提供，negative_prompt_embeds 将从 negative_prompt 输入参数生成。
  • device — (torch.device，可选)： torch 设备
  • dtype — (torch.dtype，可选)： torch 数据类型

将提示词编码为文本编码器的隐藏状态。

HeliosPipelineOutput

class diffusers.pipelines.helios.pipeline_output.HeliosPipelineOutput

< >

( frames: Tensor )

参数

  • frames (torch.Tensor, np.ndarray, or List[List[PIL.Image.Image]]) — 视频输出列表 - 它可以是一个长度为 batch_size, 的嵌套列表，其中每个子列表包含长度为 num_frames. 的去噪 PIL 图像序列。它也可以是一个形状为 (batch_size, num_frames, channels, height, width) 的 NumPy 数组或 Torch 张量。

Helios 流水线的输出类。

Update on GitHub

←Framepack HunyuanVideo→