Modality Maturity Index (MMI)

The Modality Maturity Index (MMI) is a benchmark designed to evaluate the multimodal capabilities of LLMs across five modalities and combinations of up to three modalities in both inputs and outputs. It consists of 893 human-written prompts spanning five modalities (text, image, audio, video and document). Each prompt has a set of human-written rubrics for every expected output modality. The evaluation harness for obtaining MMI scores can be found on GitHub: https://github.com/facebookresearch/modality-maturity-index. More information about the benchmark can be found in our paper:

Modality Maturity Index: A benchmark for assessing multimodal capabilities of omni models

Rohit Patel, Dieuwke Hupkes, Sloan Strader - Meta Superintelligence Labs, 2026

📚 Cite this work (BibTeX)

@misc{patel2026modalitymaturityindexbenchmark,
  title         = {Modality Maturity Index: A benchmark for assessing
                   multimodal capabilities of omni models},
  author        = {Rohit Patel and Dieuwke Hupkes and Sloan Strader},
  year          = {2026},
  eprint        = {2608.26317},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2608.26317}
}

Dataset structure

The dataset has a single test split with 893 prompts. Each row contains the following fields:

Field	Description
prompt_id	Unique identifier of the prompt.
prompt_text	The prompt, as it is sent to the model.
input_modalities	The modalities of the prompt's input.
output_modalities	The modalities a correct response should contain (the gold set).
input_files	The input files attached to the prompt.
input_images	Input images, if any.
input_audio	Input audio, if any.
input_video	Input video, if any.
input_document	Input document, if any.
rubrics	The human-written rubrics, with criteria per expected output modality.

Metrics

MMI has two main metrics as well as several complementary metrics:

Metric	What it is
Modality Presence Score (MPS)	Per-prompt F1 between the modalities returned and the gold set, averaged over prompts. Presence only - content is not inspected.
MMI Value	Rubric score: for each expected output modality, an LLM judge grades each human-written criterion in isolation; criteria are averaged within a modality, then modality means are averaged.

Alongside MPS we report precision, recall, a strict and lenient pass rate (binary: 1 only if every expected modality is present), and the error rate, indicating how many prompts were rejected by the model (mostly because it did not support one or more of the input modalities). Undrequested text is not penalised in precision.

A modality can be counted as present in three ways: the system returned actual bytes of the right type (native), the response contains a link to an asset of that modality (URL), or an LLM judge reading the raw response concludes that the modality was in fact produced (judge). The strict standard counts only natively generated assets, the lenient any of the three. For the MPS score, the lenient standard is used.

Leaderboard

This dataset hosts a leaderboard for each of the metrics above. Scores are reported on a 0-100 scale and are read from the .eval_results/ folder of model repositories on the Hub. To add results for a model, add a YAML file such as .eval_results/mmi.yaml to the model repository (or open a pull request on it) with one entry per metric:

- dataset:
    id: facebook/mmi
    task_id: mmi-value
  value: 11.58
- dataset:
    id: facebook/mmi
    task_id: modality-presence-score
  value: 15.54
# ...

The available task IDs are mmi-value, modality-presence-score, error-rate, pass-rate-strict, pass-rate-lenient, recall and precision. See the Hugging Face documentation for all supported fields.

License and citation

The dataset is released under CC BY 4.0. If you use the benchmark, please cite the paper (see the BibTeX above).