SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation
AI Summary
SKT introduces a verified data-synthesis pipeline for training language-model agents to use reusable skills. Starting from 2,000 public skills, it creates single-skill and multi-skill task configurations, applies rule-based and agent-based verification, repairs failed generations with feedback, and retains trajectories that substantially use every required skill. The resulting dataset contains 4,000 task packages and 27,164 verified trajectories. The authors also build SkillEval from a disjoint test pool. According to the paper’s abstract, supervised fine-tuning on SKT trajectories improves skill-use performance across models, benchmarks, and agent harnesses, with gains linked to verification quality and broader skill coverage.
Why it's worth reading
As agent skill libraries expand, reliably discovering, composing, and executing skills remains a bottleneck. SKT offers a scalable, executable-data approach whose verification and cross-harness claims are directly relevant to agent training pipelines.
Deep Read
1. What happened
Original facts: The paper introduces SKT, a Skill-Use Training pipeline that synthesizes executable tasks and trajectories grounded in agent skills. From 2,000 public skills, it produces 4,000 task packages and 27,164 verified trajectories, plus SkillEval from a disjoint held-out test pool.
2. Core technology
Original facts: SKT selects single-skill and multi-skill configurations, synthesizes tasks and execution trajectories, and verifies them with both rule-based and agent-based procedures. Failed generations receive feedback-guided repair. The pipeline retains trajectories only when they succeed and substantially use every required skill.
3. Key evidence and numbers
Original facts: The reported construction includes 2,000 public skills, 4,000 task packages, and 27,164 verified trajectories. The abstract states that supervised fine-tuning on SKT trajectories consistently improves skill-use performance across models, benchmarks, and agent harnesses. Verification ablations, cross-harness evaluation, and scaling experiments are described as supporting the roles of verification quality and skill coverage. Analysis: If detailed results substantiate these claims, data quality may matter more than simply increasing trajectory volume.
4. Why it matters
Analysis: A skill library provides procedural knowledge, but does not ensure that a model can recognize when to invoke a skill, compose several skills, or complete the resulting execution. Executable verification supplies a clearer success criterion than text-only demonstrations and may reduce shallow or ineffective skill calls in fine-tuning data.
5. Practical impact
Analysis: Agent platforms could connect skill registries, tool interfaces, and execution environments to automatically generate verified training examples. SkillEval may provide a held-out way to compare models or agent harnesses on skill use. In production pipelines, feedback repair and full-skill-coverage filtering could serve as quality-control stages for synthetic data.
6. Limitations and uncertainty
Original facts: The available information is primarily the abstract and does not specify the evaluated models, benchmark names, effect sizes, statistical significance, verification cost, or failure rates. Analysis: Results may depend on skill-definition quality, execution environments, verifier reliability, and task distributions. The criterion for “substantial” skill use is also important. Unverified inference: Cross-harness gains do not by themselves establish comparable effectiveness on long-horizon production tasks, changing tool environments, or safety-sensitive operations.
7. Original sources
- arXiv:2608.02287
- Source listing: Hugging Face Papers