Does Faithfulness-Guided Alignment Hurt Accuracy? Unlocking Accurate and Faithful Post-Retrieval Reasoning
Calibrated Reasoning with Answer-Faithful Traces
Right answers are not enough. CRAFT trains retrieval-augmented LLMs to produce structured, machine-auditable reasoning traces — supervised by deterministic rewards for what can be verified, and an LLM judge for what must be faithful.
A template family v1–v5 that turns auditability into a dial. The chain of faithfulness π → E → ρ → a exposes unsupported or inconsistent content through explicit links.
play with the variants ▶ 2Deterministic rewards enforce what rules can verify — format, citations, answers. A binary-decomposed LLM-judge reward audits what they cannot: semantic faithfulness of open-ended reasoning.
see the rewards ▶ 3The manuscript reports joint answer-quality and Faithfulness gains, with a capacity-dependent preference for trace structure. The result plots reproduce manuscript values; their fitted simulation provenance is noted below.
see the scoreboard ▶Under noisy retrieval, a model can cite irrelevant documents, confuse entities, follow a broken chain — and still land on the gold answer. Exact Match gives it full marks. Nobody notices. In multi-hop QA, where distractors share surface overlap with the query, this failure mode thrives.
Free-form reasoning · shortcut through a distractor · entity confusion —
EM says ✓ 1.0 — the shortcut is invisible to answer-level metrics. ✗
Structured trace · committed evidence · every step checkable —
Four audit checks pass, the reasoning survives scrutiny. EM 1.0 · Faith 1.0 ✓
CRAFT decomposes generation into an XML trace where each stage constrains the next: π → E → ρ → a. Unsupported content is exposed when it breaks one of these machine-auditable links.
Decompose the question into ordered sub-questions before touching any document.
Declare the evidence boundary — e.g. [2, 5] — before reasoning begins.
Step-by-step derivation that follows the plan and may cite only inside the committed boundary.
A short span that must be logically entailed by the reasoning chain — nothing more.
⛓️ Break any link — plan ignored, citation outside the boundary, unsupported claim, non-entailed answer — and the trace is flagged. That is the point.
Auditability is a dial, not a switch. Five template variants ablate the chain component by component, from fully-armored v1 down to the answer-only baseline v5. The template decides which rewards are active and which judge checks apply. Click a card — the live trace below re-assembles itself.
hover a field — its auditors light up on the right
GRPO samples a group of traces per question, normalizes rewards within the group, and updates critic-free. The total reward is the weighted sum of the components active for the chosen variant — deterministic rewards enforce what rules can verify; a judge reward audits what they cannot.
All required tags for the variant, in the correct order. It gates the deterministic citation and answer rewards; the judge scores the submitted trace independently.
A three-level overlap score between the declared <gold_docs> and true supporting documents — exact set, partial overlap, or no overlap.
Normalized exact match of the parsed <answer> against the set of acceptable gold answers.
An LLM judge (Qwen3-30B-A3B, distinct from the Qwen2.5 policies) returns binary verdicts on decomposed criteria; Rfaith is their mean.
Does the reasoning address the plan’s sub-questions, in order?
Are all cited documents inside the committed boundary — and is at least one used?
Is the answer a logical conclusion of the reasoning chain?
Is every claim supported by the cited document text? One unsupported claim fails the check.
Why no direct supervision on plan / reason text? They are open-ended — imitating teacher traces is brittle.
CRAFT supervises function, not form: π is valid iff ρ follows it (A), ρ is valid iff it is grounded (D).
The model discovers its own reasoning paths inside an auditable structure.
In the manuscript's reported values, CRAFT·7B (v1, full chain) exceeds the best API EM on HotpotQA; its lead on 2WikiMHQA is +7.76 EM, while it remains close on MuSiQue.
Data provenance. These numbers match the manuscript's tables, not a fresh evaluation of the released checkpoints. The main-result aggregates are fitted simulations anchored on archived evaluations; ten fitted 1,000-example replicates are not ten independent inference runs. The w/o-judge values are Full-anchored counterfactual estimates and do not establish a measured causal effect of removing the judge reward. Source and checkpoint details.
| Method | MuSiQue | HotpotQA | 2WikiMHQA |
|---|---|---|---|
| Qwen2.5-7B (base) | 35.83 | 56.47 | 57.87 |
| SFT 7B | 31.01 | 57.81 | 62.69 |
| Best API | 53.70 | 63.60 | 70.47 |
| CRAFT 7B | 53.13 | 64.06 | 78.23 |
On MuSiQue, CRAFT improves Faithfulness by 25.40 points over Base and 32.65 points over SFT.
| Method | MuSiQue | HotpotQA | 2WikiMHQA |
|---|---|---|---|
| Qwen2.5-7B (base) | 58.47 | 83.30 | 76.92 |
| SFT 7B | 51.22 | 83.37 | 79.00 |
| Best API | 83.42 | 96.43 | 96.80 |
| CRAFT 7B | 83.87 | 96.31 | 96.79 |
The full trace yields clear gains from 1.5B upward; at 0.5B, performance remains strongly template-dependent.
| Scale | EM | Faithfulness |
|---|---|---|
| 0.5B | 0.00 | 1.11 |
| 1.5B | 26.87 | 28.35 |
| 3B | 41.44 | 58.69 |
| 7B | 53.13 | 83.87 |
Reported counterfactual comparison. On MuSiQue v1, the manuscript lists Faithfulness of 81.96 versus 83.87 and EM of 51.53 versus 53.13 for w/o-judge versus full CRAFT. These are fitted estimates, not an independently measured ablation.
Task-specific capacity is the central hypothesis. The manuscript reports 2WikiMHQA EM of 57.87 for Base and 78.23 for CRAFT at 7B, with consistent gains across the three datasets beginning at 1.5B.
A plan→reason concludes “English” despite finding SpellingB cite→reason cites doc [1] outside the declared boundaryC reason→answer derives “English”, then uses US incomeD grounding “creator is English” — unsupportedA plan→reason creator → nationality → income, in orderB cite→reason every citation inside the boundaryC reason→answer answer follows from the chainD grounding all claims verified in the documentsTwo NLP annotators independently assessed 500 randomly sampled MuSiQue traces with the same four binary audit criteria.