ARB: A Matched Authorship-Rewriting Benchmark Dataset for AI-Text Detector Evaluation
AI Summary
The paper introduces the Authorship-Rewriting Benchmark (ARB), a matched dataset built from 1,800 human texts drawn from XSum, WritingPrompts, and OpenWebText. Each source produces four variants: human-written text, direct LLM generation, human-to-LLM rewriting, and same-generator rewriting of LLM text. At a strict 1% false-positive rate, FastDetectGPT and Binoculars-falcon-7b detected 91.2% and 93.5% of direct LLM text, but only 30.8% and 15.1% of human text rewritten by an LLM. Detection of LLM-originated text remained substantially stronger after rewriting.
Why it's worth reading
It tests the deployment-relevant case of human writing later revised by an LLM, showing that conventional human-versus-machine benchmarks can substantially overestimate detector performance.
Deep Read
1. What happened
Original facts: The paper introduces the Authorship-Rewriting Benchmark (ARB) to measure the gap between conventional AI-text detection tests and realistic LLM-assisted editing. It contains 1,800 human source texts from XSum, WritingPrompts, and OpenWebText. Each source is paired with four variants: HUMAN, Free-LLM, H2L, and LLM2L.
2. Core technology
Original facts: ARB uses four open-weight generators: Llama-3.2-3B, Qwen2.5-7B, Mistral-7B, and Gemma-2-9B. H2L denotes human-authored text rewritten by an LLM. LLM2L denotes LLM-generated text rewritten by the same generator. The study evaluates FastDetectGPT, Binoculars-falcon-7b, RADAR, BERT-Defense, and RoBERTa-Defense.
3. Key evidence and numbers
Original facts: The evaluation uses TPR at a strict 1% false-positive rate. FastDetectGPT detects 91.2% of direct LLM text but only 30.8% of H2L text. Binoculars-falcon-7b falls from 93.5% to 15.1%, a decline of roughly 60 to 78 percentage points for the two detectors. Their LLM2L recall remains 78.3% and 83.0%, respectively, a smaller decline of about 10 to 13 points. RADAR drops from 66.8% to 12.2%, while BERT-Defense and RoBERTa-Defense remain below 3% recall across regimes.
4. Why it matters
Analysis: The results suggest that detectors may rely heavily on distributional signatures of directly generated text rather than reliably identifying whether an LLM participated in editing. Consequently, a standard HUMAN-vs-LLM benchmark may not predict performance on revised human writing. The contrast with LLM2L also indicates that source authorship and rewriting history affect detectability differently.
5. Practical impact
Analysis: Academic-integrity, moderation, and publishing workflows could overestimate detector capability if they evaluate only direct generations. ARB offers a way to test systems against more realistic editing workflows. Product and policy teams should distinguish human-authored text later modified by an LLM from LLM-originated text that is subsequently rewritten. Detector outputs should be treated as risk signals rather than standalone attribution evidence.
6. Limitations and uncertainty
Original facts and uncertainty: The supplied abstract does not specify the rewriting prompts, edit intensity, language coverage, length distributions, data splits, or statistical significance. All four generators are open-weight models, so the findings may not transfer to proprietary models, other languages, human editing, or multi-stage mixed workflows. The low recall of BERT-Defense and RoBERTa-Defense cannot, from the abstract alone, be attributed to implementation, threshold calibration, or general model weakness. Generalizing these results to every detector or real-world writing process remains unverified inference.
7. Original sources
- Paper page: arXiv:2607.29539
- Source type: arXiv paper record
- Publication date: 2026-07-31, according to the supplied source metadata