Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

The paper introduces a 21-scenario benchmark for adaptive, multi-round attacks against memoryless LLM defenders. An autonomous attacker observes prior responses and changes tactics for up to 15 rounds. Under fixed first-turn scoring, attack success rates are 0–1%; adaptive attacks raise them to 5.4–14.0%. Pooling three frontier attacker models finds 1.4–2.2 times more unique successful attacks than the best individual attacker. Claude Opus 4.6 and GPT-5.4 tie at 5.4% aggregate ASR, but scenario-level weaknesses differ substantially.