CLI reference

gauntlet run

Fire the adversarial suite at an agent endpoint.

gauntlet run --target $URL [options]

Flag	Default	Description
--target	— (required)	Agent HTTP endpoint (POST, JSON in/out)
--canaries	—	Path to a canaries JSON file
--category	all	Limit to a probe category (repeatable)
--request-field	message	JSON field to put the probe in
--response-field	response	JSON field to read the reply from
--header	—	Extra request header Key: Value (repeatable)
--timeout	20	Per-request timeout (seconds)
--concurrency	8	Concurrent requests
--top	3	How many worst failures to print
--fail-on	HIGH	CI gate severity
--json	—	Write a JSON report to this path
--multiturn	off	Also run multi-turn probes
--history-field	—	For multi-turn: field to send the transcript in (e.g. messages)
--llm	off	Generate fresh adversarial personas via the Anthropic API
--describe	—	Short description of the target agent (with --llm)
--model	claude-haiku-4-5	Model for --llm / judge

Exit code: 0 if nothing meets the gate, 1 if a finding is at/above --fail-on, 2 on a setup error.

gauntlet calibrate

Score the --llm judge against a human-labeled gold set. See Judge calibration.

gauntlet calibrate --gold examples/gold.jsonl --min-kappa 0.6

Flag	Default	Description
--gold	— (required)	JSONL gold set (probe/response/failure per line)
--model	claude-haiku-4-5	Judge model
--min-kappa	0.6	Minimum Cohen's κ to pass (else exit 1)

Environment

Var	Use
ANTHROPIC_API_KEY	required for --llm and calibrate
GAUNTLET_MODEL	default model string