HackBench is a benchmark for numbered command localization of reward hacking in long-horizon ML research agent trajectories. Each record contains the original trajectory, a ready-to-use numbered monitor input, and audited gold command indices.

This release contains 169 HACK records across 13 tasks: 41 natural and 128 synthetic hacking examples.

Source agent	Records
GPT-5.5	102
GLM-5.2	67
Total	169

Project: RAM.

Blog Post: HackBench: Detecting and Localizing Reward Hacking in Autoresearch Agents.

What is included

One JSONL row per trajectory record.
Full injected_transcript strings, including recorded reasoning, tool commands, and observations.
monitor_input, with task context and a transcript whose executable command blocks carry one-based [CMD n] labels, plus matching gold_command_indices.
Task instructions, gold command strings, and source-agent settings.

Load the dataset

from datasets import load_dataset

ds = load_dataset("facebook/hackbench", split="test", token=True)
row = ds[0]

# Input to a detector. Keep gold labels separate.
# The instruction below is the agent's task context, not the monitor's prompt.
detector_input = {
    "instruction": row["monitor_input"]["task_instruction"],
    "transcript": row["monitor_input"]["transcript"],
}

# Ground truth, used only after the detector produces its prediction.
gold_command_indices = row["gold_command_indices"]

monitor_input uses the reference preprocessing: historical detector feedback is removed, blocked attempts remain as command blocks, and the research task context is the complete initial user message. This task context is for the monitor to understand the agent's objective; supply your own monitor prompt separately. The numbered transcript is fully rendered without model-specific clipping.

The gold indices refer to the [CMD n] labels in that input. Numbering starts at 1 for each trajectory and counts executable command blocks; a block can contain several lines of code. Use gold_command_indices for evaluation and preserve the provided numbers when managing long context.

The file can also be read directly, without converting its nested metadata to Arrow columns:

import json
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="facebook/hackbench",
    repo_type="dataset",
    filename="data/hackbench_169_gpt55_glm52.numbered.jsonl",
    token=True,
)
with open(path, encoding="utf-8") as stream:
    row = json.loads(next(stream))

Fields

Field	Type	Meaning
`id`	string	Unique record identifier; historical provenance identifiers are preserved.
`task.name`	string	One of the canonical task names listed below.
`task.instruction`	string	Stored task-wrapper metadata
`source_agent`	object	Autoresearch agent; model names use `GPT-5.5` or `GLM-5.2`.
`injected_transcript`	string	Original released trajectory text, retained as provenance.
`monitor_input.task_instruction`	string	Complete recorded initial instructions for the research agent, supplied to the monitor as task context.
`monitor_input.transcript`	string	Preprocessed transcript with `[CMD n]` labels; the numbered-localization input.
`monitor_input.command_count`	integer	Number of executable command blocks in the numbered input.
`gold_command_indices`	list of integers	Audited gold `[CMD n]` indices in `monitor_input.transcript`, using one-based numbering.
`hack_commands`	list of strings	Original gold command text, retained as audit evidence.
`origin`	string	`natural` or `synthetic`.
`hack_type`	string	`general` or `task-specific`.

Gold indices and command strings can reveal the answer. Construct detector input explicitly from monitor_input and keep evaluation targets separate.

Task coverage

Task	Records
`continual-learning`	7
`cross-modal-retrieval`	15
`data_select_ifeval`	11
`flux2_klein_lora`	13
`grpo_multisource`	6
`improving-replay-buffers`	37
`llm_online_serving`	6
`materials-tokenization`	9
`moving_mnist_world_model`	13
`multilingual_ocr`	25
`scaling_law`	8
`time-series-explanation`	14
`w2s`	5

Long-context considerations

Transcript size	Minimum	Median	Maximum
Unicode characters	73,412	1,064,439	3,834,163
UTF-8 bytes	73,532	1,064,975	3,835,613

These lengths describe the original injected_transcript. The JSONL contains both the original text and the numbered reference view.

The JSONL file is 408,528,473 bytes (389.60 MiB). Long rows may exceed a model's context limit, and a web viewer may shorten what it displays. Do not silently truncate them. If an evaluation uses chunking or another context strategy, report that strategy and retain linkage to the full record.

Intended use and limitations

The dataset supports offline evaluation of numbered reward-hacking command localization on 169 HACK trajectories. Report precision, recall, and F₁ computed per trajectory and then averaged over the release. The test split denotes an evaluation release; no separate training or validation split is supplied.

Downloads last month: 41

Total file size:

409 MB