基于验证器引导与强化学习的神经符号可解释推理框架

My Thoughts on a Verifier-Guided Reasoning Framework

As an AI researcher specializing in reasoning and educational question answering, I find the challenge of LLMs providing inconsistent or unverifiable explanations a significant hurdle. My current focus revolves around a novel framework designed to address this: a verifier-guided approach for transparent educational question answering. I'm exploring a combination of techniques here. First, I'm leveraging gold-anchored QLoRA, ensuring that the model, specifically Qwen2.5-3B-Instruct, is initially adapted with field-weighted supervision guided by authoritative answers, strengthening its grounding. Next, I employ a lightweight router to intelligently direct logic problems to an FOL/Z3 verifier and physics problems to a formula- and unit-aware symbolic solver; this task-aware symbolic routing ensures that the right tools are applied. This framework incorporates group-relative RLVR, where verifier feedback actively supports candidate evaluation, self-revision, and reward construction within a reinforcement learning loop. The system evaluates candidate responses along three key dimensions: correctness (P1), evidence/unit consistency (P2), and reasoning depth/explainability (P3). At the inference stage, I utilize a gold-free self-consistency mechanism to aggregate multiple responses, followed by an optional question-only physics verifier to implement conservative system-level corrections. In evaluations on held-out examples, I'm seeing RLVR significantly boost P3, representing a solid increase in reasoning quality. Simultaneously, the hybrid P1 metric remains robust. Furthermore, self-consistency offers a small but noticeable improvement to the model-only P1 score, with additional gains stemming from symbolic verification. This research underscores that RLVR primarily bolsters the explicit reasoning structure, while symbolic verification contributes to system-level answer reliability, offering a valuable complement to the neural policy.

大语言模型（LLM）展现出了强大的推理能力，但其给出的解释往往仍存在不一致、缺乏可靠依据或难以验证的问题。针对透明化教育问答，我们提出了一种验证器引导的可解释推理框架，该框架结合了标准答案锚定的 QLoRA、任务感知的符号路由以及群体相对 RLVR。首先，以权威答案为锚点，采用字段加权的 QLoRA 监督对 Qwen2.5-3B-Instruct 进行微调适配。随后，轻量级路由器将逻辑问题分配给一阶逻辑（FOL）/Z3 验证器，将物理问题分配给公式与单位感知的符号求解器。验证器反馈进一步用于支持 RLVR 过程中的候选评估、自我修正及奖励构建。候选回答从三个互补维度进行评估：用于评估答案正确性的 P1、用于评估证据或单位一致性的 P2，以及用于评估推理深度与可解释性的 P3。在推理阶段，无标准答案的自洽机制先聚合多个候选回答，随后可选的仅输入问题的物理验证器会执行保守的系统级修正。在 438 个留出样本上的实验表明，RLVR 将 P3 从 50.68% 提升至 72.20%，同时混合 P1 大致稳定在 55.94%。自洽机制将纯模型 P1 从 48.86% 提升至 50.23%，其余的混合增益则由符号验证提供。这些结果表明，RLVR 主要强化了显式推理结构，而符号验证则通过在系统层面提高答案可靠性，对神经策略形成了有效互补。