旧版适配器

这些方法仍然受到支持，尤其是在 Stable Diffusion 系列的检查点上。对于新的工作，建议优先使用 LoRA、IP-Adapter 和 ControlNet。

T2I-Adapter

T2I-Adapter 是一种用于可控生成的适配器。它学习从控制信号（例如深度图或 Canny 边缘）到预训练模型内部特征的映射，并在生成过程中加入该引导。其作用类似于 ControlNet，但它是一个更轻量级的即插即用适配器。

加载一个受控适配器，然后使用 from_pretrained() 加载流水线，并通过 adapter= 传入该适配器。

import torch
from diffusers import T2IAdapter, StableDiffusionXLAdapterPipeline, AutoencoderKL

t2i_adapter = T2IAdapter.from_pretrained(
    "TencentARC/t2i-adapter-canny-sdxl-1.0",
    dtype=torch.float16,
)

使用 opencv-python 生成 canny 图像。

import cv2
import numpy as np
from PIL import Image
from diffusers.utils import load_image

original_image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/non-enhanced-prompt.png"
)

image = np.array(original_image)

low_threshold = 100
high_threshold = 200

image = cv2.Canny(image, low_threshold, high_threshold)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

将 canny 图像传入 pipeline 以生成图像。

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", dtype=torch.float16)
pipeline = StableDiffusionXLAdapterPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    adapter=t2i_adapter,
    vae=vae,
    dtype=torch.float16,
).to("cuda")  # or "mps", "xpu", "cpu"

prompt = """
A photorealistic overhead image of a cat reclining sideways in a flamingo pool floatie holding a margarita. 
The cat is floating leisurely in the pool and completely relaxed and happy.
"""

pipeline(
    prompt, 
    image=canny_image,
    num_inference_steps=100, 
    guidance_scale=10,
).images[0]

Generated image (T2I-Adapter + prompt) — 生成的图像

另请参阅 T2I-Adapter 训练。

MultiAdapter

使用 MultiAdapter 组合多个控制（例如 canny 和 depth）。将适配器列表传入 MultiAdapter，然后将对应的控制图像以列表形式传入 pipeline。

import torch
from diffusers.utils import load_image
from diffusers import StableDiffusionXLAdapterPipeline, AutoencoderKL, MultiAdapter, T2IAdapter

canny_image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/canny-cat.png"
)
depth_image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/sdxl_depth_image.png"
)
controls = [canny_image, depth_image]
prompt = ["""
a relaxed rabbit sitting on a striped towel next to a pool with a tropical drink nearby, 
bright sunny day, vacation scene, 35mm photograph, film, professional, 4k, highly detailed
"""]

adapters = MultiAdapter(
    [
        T2IAdapter.from_pretrained("TencentARC/t2i-adapter-canny-sdxl-1.0", dtype=torch.float16),
        T2IAdapter.from_pretrained("TencentARC/t2i-adapter-depth-midas-sdxl-1.0", dtype=torch.float16),
    ]
)

将适配器、提示词和控制图像传入 StableDiffusionXLAdapterPipeline。使用 adapter_conditioning_scale 参数来确定为每个控制分配多少权重。

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", dtype=torch.float16)
pipeline = StableDiffusionXLAdapterPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    dtype=torch.float16,
    vae=vae,
    adapter=adapters,
).to("cuda")  # or "mps", "xpu", "cpu"

pipeline(
    prompt,
    image=controls,
    height=1024,
    width=1024,
    adapter_conditioning_scale=[0.7, 0.7]
).images[0]

Generated image (MultiAdapter + prompt) — 生成的图像

文本反转

Textual Inversion 通过微调绑定到唯一标记（<sks>）的词嵌入，使模型从 3-5 张图片中学习特定概念。之后，你可以在提示词中使用该标记来生成该概念（例如，像素艺术）。

Textual Inversion 权重通常只有几 KB，因为它们仅包含词嵌入。请在调用 from_pretrained() 之后，使用 load_textual_inversion() 加载这些权重，并在提示词中包含该唯一标记以触发生成。

import torch
from diffusers import AutoPipelineForText2Image

pipeline = AutoPipelineForText2Image.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    dtype=torch.float16
).to("cuda")  # or "mps", "xpu", "cpu"

pipeline.load_textual_inversion("sd-concepts-library/gta5-artwork")
prompt = "A cute brown bear eating a slice of pizza, stunning color scheme, masterpiece, illustration, <gta5-artwork> style"
pipeline(prompt).images[0]

要训练嵌入，请参阅训练文本反转。

Textual Inversion 还可以训练以学习负向嵌入，从而引导生成过程避开“模糊”或“丑陋”等不需要的特征，有助于提升图像质量。

EasyNegative 是一种广泛使用的负向嵌入，其中包含多个学习到的负向概念。加载负向嵌入，并指定与负向嵌入关联的文件名和标记。将该标记传入管道中的 negative_prompt 以启用它。

import torch
from diffusers import AutoPipelineForText2Image

pipeline = AutoPipelineForText2Image.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    dtype=torch.float16
).to("cuda")  # or "mps", "xpu", "cpu"
pipeline.load_textual_inversion(
    "EvilEngine/easynegative",
    weight_name="easynegative.safetensors",
    token="easynegative"
)
prompt = "A cute brown bear eating a slice of pizza, stunning color scheme, masterpiece, illustration"
negative_prompt = "easynegative"
pipeline(prompt, negative_prompt=negative_prompt).images[0]

Update on GitHub

Diffusers

旧版适配器

T2I-Adapter

MultiAdapter

文本反转