Near-universal benchmark leadership
EviBack delivers the highest F1 on all seven benchmarks at 1.7B and on six of seven at both 0.6B and 4B.
Search-Agent Reinforcement Learning via Evidence-Constrained Teacher Backoff
EviBack improves Agentic RAG under sparse rewards by using constrained Teacher backoff to strengthen retrieval, evidence assessment, and stopping decisions without overriding the Actor’s evidence boundaries.
Overview
Abstract
Reinforcement learning enables Agentic RAG systems to learn when and how to search, assess retrieved evidence, and decide when to stop. However, training often relies on sparse outcome rewards, leaving unsuccessful rollout groups with little or no comparative signal. We present EviBack, an evidence-constrained Teacher backoff method that preserves verifiable Actor rewards and supplies weak auxiliary supervision only when every rollout in a group fails to produce a correct answer. EviBack separates evidence assessment from answer refinement: a gold-blind gate first determines whether the retrieved evidence is sufficient, and a gold-aware stage may refine the answer only when the gate does not identify insufficient evidence. This separation prevents reference-answer supervision from turning unsupported trajectories into apparently successful ones. The Teacher policy is constructed through an auditable, GPT-5.5-assisted vibe-coding workflow that converts natural-language specifications and trace-level diagnostics into versioned prompts, control logic, and tests. The resulting Teacher is used only during training; inference relies solely on the learned search Actor. Experiments across seven open-domain question-answering benchmarks and three Qwen3 model scales yield consistent point-estimate improvements over Search-R1, with statistically clear gains at 1.7B and 4B, together with scale-dependent changes in search and stopping behavior. These results show that constrained Teacher backoff can restore useful learning signal under sparse rewards while preserving the evidence boundary required for reliable search-policy learning.
Method
EviBack turns otherwise unusable all-zero GRPO groups into ranked training signal while leaving every verifiable Actor reward untouched. A GPT-5.5-controlled automatic prompt engineering pipeline discovers, tests, and freezes the gated Teacher policy; formal RL uses the resulting GLM-4.7-Flash Teacher only during training, and inference remains Actor-only.
- Signal only where needed. The reward audit finds that 73% of rollout groups have no binary contrast. EviBack calls the Teacher only when all eight trajectories receive zero exact-match reward; mixed groups keep their original rewards. Fallback rewards are normalized within the group and their advantages are scaled by 0.1, so auxiliary supervision remains weaker than a verified Actor hit.
- Gold cannot move the boundary. Stage A judges only the question and Actor-visible evidence. Stage B sees the reference only after Stage A returns supported or ambiguous, and it cannot overturn an insufficient decision. The resulting gate preserves Stage A's 0.8924 insufficient-evidence F1 while raising gold-answer coverage to 0.6825, above the 0.6399 gold-aware single prompt.
- Automatically built and deployment-free. With no manual prompt intervention after launch, GPT-5.5 generates, executes, diagnoses, and refines candidates before freezing versioned prompts, gating logic, parsers, tests, hashes, and logs. Compared with the manual Teacher, the selected policy adds 2.60 F1 points and 21.97 valid-answer-rate points while reducing mean searches by 1.4067; neither Teacher nor GPT-5.5 is used at inference.
Results
Under a common deterministic protocol spanning 3,500 questions and seven open-domain QA benchmarks, EviBack achieves the best aggregate EM and F1 at every tested Qwen3 scale. It records the highest per-benchmark F1 in 19 of 21 scale-dataset settings, including all seven benchmarks at 1.7B. Relative to Search-R1, the F1 gains are +0.59, +4.02, and +1.32 points at 0.6B, 1.7B, and 4B, with statistically clear gains at 1.7B and 4B. At 1.7B, EviBack also improves valid-answer rate while reducing searches, duplicate queries, and forced termination.
Strongest gain at 1.7B
F1 rises by 4.02 points, with a paired 95% confidence interval of [3.05, 4.96] points.
Failed trajectories become valid
At 1.7B, 593 questions move from failure under Search-R1 to a valid EviBack answer, compared with only 140 regressions.
Automated Teacher construction wins
The APE-built EviBack Teacher also adds 21.97 points in valid-answer rate and reduces mean searches by 1.4067 over the manual design.
Comparison
BibTex
@article{ma2026eviback,
title={Search-Agent Reinforcement Learning via Evidence-Constrained Teacher Backoff},
author={Xiao Ma, Zhiquan Hu, Yi Wei, Chenchen Zhao, Yijun Chen, Jicheng Zhao, Yuming Li, Chuang Dai},
year={2026},
eprint={2607.23955},
archivePrefix={arXiv},
primaryClass={cs.AI}
}