Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries

This study evaluated a human-in-the-loop AI agent that gathers sourced evidence across platforms and drafts one-sentence Translational Science Benefits Model (TSBM) impact summaries. In a real-world workflow at one Clinical and Translational Science Award hub, the system was tested on 10 KL2/K12 scholars and produced 507 findings. Two reviewers accepted or edited 81.7% of findings. Review took a median of 14 minutes per scholar, compared with an estimated 15 hours for manual assembly. Cohen’s kappa was 0.43, while synthesis accuracy and usefulness were rated 4.5 and 4.8 out of 5.