PANOPTICON: A PII-Based Assemblage of Naturalistic Output Tokens for Investigating Privacy Leakage Within LLM Context Windows

PANOPTICON is a pipeline and dataset for studying privacy leakage in LLM context windows. Using Meta's Llama-3.1-8B-Instruct, the authors generated 67,718 context-window prompts containing PII spans derived from 9,674 publicly available synthetic user profiles. The paper evaluates realism through lexical diversity and S-BERT diversity, then demonstrates a use case involving Prompt Inversion Attacks (PIAs). It addresses the lack of public authentic PII datasets caused by ethical and privacy constraints, and positions PANOPTICON as a benchmark for investigating attacks against private corpora.