Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation

This paper introduces Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR) for measuring how far generated summaries move beyond extractive copying. The formulation uses the harmonic mean of document lengths and a cubic non-overlap factor to produce bounded, non-linear scores. On 100 XSUM documents evaluated with BART-large-cnn, Pegasus-xsum, DistilBart, and MT5-small, SA reportedly separates more extractive systems, around 0.12–0.26, from more abstractive systems, around 0.96–1.77. AR is proposed as a signal for selecting summaries that may require manual hallucination review.