Hugging Face's logo

Accelerate documentation

FP8

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v1.15.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

FP8

Below are functions and classes relative to the underlying FP8 implementation

TERecipeKwargs

class accelerate.utils.TERecipeKwargs

< >

( use_autocast_during_eval: bool | None = Nonemargin: int | None = Noneinterval: int | None = Nonefp8_format: typing.Literal['HYBRID', 'E4M3', 'E5M2'] = Noneamax_history_len: int | None = Noneamax_compute_algo: typing.Literal['max', 'most_recent'] = Noneoverride_linear_precision: tuple = Noneuse_mxfp8_block_scaling: bool | None = None )

Parameters

  • use_autocast_during_eval (bool, optional, default to False) — Whether to use FP8 autocast during eval mode. Generally better metrics are found when this is False.
  • margin (int, optional, default to 0) — The margin to use for the gradient scaling.
  • interval (int, optional, default to 1) — The interval to use for how often the scaling factor is recomputed.
  • fp8_format (str, optional, default to “HYBRID”) — The format to use for the FP8 recipe. Must be one of HYBRID, E4M3 or E5M2. (Generally HYBRID for training, E4M3 or E5M2 for evaluation)
  • amax_history_len (int, optional, default to 1024) — The length of the history to use for the scaling factor computation
  • amax_compute_algo (str, optional, default to “most_recent”) — The algorithm to use for the scaling factor computation. Must be one of max or most_recent.
  • override_linear_precision (tuple of three bool, optional, default to (False, False, False)) — Whether or not to execute fprop, dgrad, and wgrad GEMMS in higher precision.

Use this object in your Accelerator to customize the initialization of the recipe for FP8 mixed precision training with transformer-engine.

For more information on the args, please refer to the API documentation.

from accelerate import Accelerator
from accelerate.utils import TERecipeKwargs

kwargs = TERecipeKwargs(fp8_format="HYBRID")
accelerator = Accelerator(mixed_precision="fp8", kwargs_handlers=[kwargs])

AORecipeKwargs

class accelerate.utils.AORecipeKwargs

< >

( config: ForwardRef('Float8LinearConfig') | None = Nonemodule_filter_func: typing.Callable | None = Nonepad_inner_dim: bool | None = Noneenable_fsdp_float8_all_gather: bool | None = None )

Parameters

  • config (torchao.float8.Float8LinearConfig, optional, default to None) — The configuration for the FP8 training. If None, a default config will be created with sensible defaults for most use cases:
    • pad_inner_dim=True: Pads matrix dimensions to be divisible by 16, required for torch._scaled_mm operations to prevent runtime errors.
    • enable_fsdp_float8_all_gather=True: Enables FP8 all-gather for FSDP2. This provides memory bandwidth savings by casting parameters before the all-gather operation, saving 50% bandwidth compared to BF16.

    You can override these defaults by providing your own Float8LinearConfig instance.

  • module_filter_func (Callable, optional, default to None) — Optional function that must take in a module and layer name, and returns a boolean indicating whether the module should be converted to FP8. Defaults to accelerate.utils.ao.filter_linear_layers. See it for an example.

Use this object in your Accelerator to customize the initialization of the recipe for FP8 mixed precision training with torchao FP8.

convert_model

accelerate.utils.convert_model

< >

( modelto_transformer_engine = True_convert_linear = True_convert_ln = True )

Recursively converts the linear and layernorm layers of a model to their transformers_engine counterpart.

has_transformer_engine_layers

accelerate.utils.has_transformer_engine_layers

< >

( model )

Returns whether a given model has some transformer_engine layer or not.

contextual_fp8_autocast

accelerate.utils.contextual_fp8_autocast

< >

( model_forwardfp8_recipeuse_during_eval = False )

Wrapper for a model’s forward method to apply FP8 autocast. Is context aware, meaning that by default it will disable FP8 autocast during eval mode, which is generally better for more accurate metrics.

apply_fp8_autowrap

accelerate.utils.apply_fp8_autowrap

< >

( modelfp8_recipe_handler )

Applies FP8 context manager to the model’s forward method

Update on GitHub

←Kwargs handlers Utility functions and classes→