OPERAAAAI 2026
A Reinforcement Learning–Enhanced Orchestrated Planner-Executor Architecture for Reasoning-Oriented Multi-Hop Retrieval.
Existing RAG pipelines retrieve first and reason after. OPERA fuses the two: a hierarchical three-agent architecture reasons while retrieving — every retrieval step is shaped by the current reasoning state, and every reasoning step drives the next retrieval — trained end-to-end with a new multi-agent variant of GRPO.
1Institute of Information Engineering, Chinese Academy of Sciences · 2School of Cyber Security, University of Chinese Academy of Sciences · 3Huazhong University of Science and Technology · 4Tsinghua University · †Corresponding author
Overview
Retrieval and reasoning belong in the same loop
Consider: “What was the previous occupation of the person who succeeded the founder of the company that acquired WhatsApp?” Answering demands a precise sequence of retrievals and inferences, each depending on the last. Conventional retrieve-then-reason pipelines break down here for three reasons: rule-based decomposers produce brittle plans, query reformulation is too shallow to escape failed-retrieval loops, and golden documents drown in noisy top-k results. All three trace back to one root cause — weak coupling between retrieval and reasoning.
Why OPERA
Retrieve-then-reason vs. reason-while-retrieving
- Static plans. Rule-based or planner-first decomposition collapses on out-of-template questions and cannot adapt mid-flight.
- Blind retries. Failed retrievals trigger shallow query tweaks, looping without ever reaching the golden documents.
- Noise wins. Even when the right document is fetched, coarse filtering leaves it buried in noisy top-k context.
- Reasoned plans. A dedicated Plan Agent decomposes the question into ordered sub-goals with explicit placeholder dependencies.
- Reasoned retrieval. A specialised Rewrite Agent reformulates the query only when evidence is judged insufficient — trained on retrieval effectiveness (NDCG) itself.
- Reasoned filtering. The Analysis-Answer Agent assesses sufficiency and extracts the precise answer before the pipeline moves on; a Trajectory Memory Component records the rationale of every action.
Architecture
Two modules, three specialised agents
Plan Agent (Qwen2.5-7B)
Decomposes a multi-hop question into coherent, executable sub-goals with placeholder dependencies — strategy, decided once and revised only when execution demands it.
Analysis-Answer Agent (Qwen2.5-7B)
Judges whether retrieved evidence suffices (φ), and if so extracts a precise answer for the current sub-goal — reasoning-guided filtering instead of blind top-k stuffing.
Rewrite Agent (Qwen2.5-3B)
Activates only on insufficiency: reformulates the query from the reasoning state and feedback, then hands back to retrieval — turning failed lookups into successful ones.
- Plan. The Plan Agent decomposes the question into ordered sub-goals.
- Retrieve. BGE-M3 fetches top-5 evidence from the merged 1.78M-document FAISS index.
- Analyse & answer. The Analysis-Answer Agent answers the sub-goal when evidence is sufficient.
- Rewrite on failure. The Rewrite Agent reformulates the query only after insufficient evidence.
- Synthesise. Executed sub-goal answers are combined into the final answer; the TMC keeps the full audit trail.
MAPGRPO
Train each agent for the job it actually does
Multi-Agents Progressive Group Relative Policy Optimization extends GRPO to a team of heterogeneous agents: agents are optimised sequentially — Plan, then Analysis-Answer, then Rewrite — each conditioned on its already-trained predecessors, with a reward tailored to its role:
Plan quality
Accuracy 30% · coherence 25% · executability 25% · dependency 10% · efficiency 10% — judged against pre-scored golden plans.
Reasoning accuracy
0.65 × answer EM + 0.25 × sufficiency judgement + 0.10 × format — precise credit for precise answers.
Retrieval effectiveness
√NDCG@k against golden documents + format compliance — the reward is the retrieval outcome.
Two ingredients keep training stable: expert injection (each group of G=8 candidates includes the best sample from a DeepSeek-R1 pre-scored pool, provably reducing policy-gradient variance) and group-relative advantages with a KL constraint. MAPGRPO converges to local optima at rate 𝒪(1/√T).
Walkthrough
Watch one question run the loop
- 1PlanGPM · Plan Agent
- 2RetrieveBGE-M3 · top-5
- 3AnalyseREM · sufficiency φ
- 4RewriteREM · query refine
- 5Synthesisefinal answer
The Plan Agent decomposes the question into an ordered, dependency-aware plan:
expected: entity[people from step 1] attend for their undergraduate studies? depends on: 1Strategy is fixed up front — placeholders wire later steps to earlier answers.
BGE-M3 searches the 1.78M-document index for sub-goal 1:
DNA structure discovery Nobel Prize
Top-5 evidence returned to the Reason-Execute Module.
The Analysis-Answer Agent reasons over the evidence before answering:
φ = insufficient — instead of guessing, the sub-goal is handed to the Rewrite Agent with feedback: need laureate names.
The Rewrite Agent reformulates from the reasoning state, not by blind paraphrase:
1962 Nobel Prize DNA Watson Crick winners
Sub-goal 1 answered. The refined query is rewarded by √NDCG@k during MAPGRPO training — retrieval effectiveness is the reward.
Sub-goal 2 executes with the placeholder resolved to Watson, Crick, Wilkins, retrieving each laureate’s undergraduate institution.
Done. The Trajectory Memory Component keeps the full audit trail — every plan, query, verdict and rationale.
Results
State of the art on three multi-hop benchmarks
best baseline 45.7
best baseline 44.3
best baseline 24.3
on the hardest benchmark
| Method | Type | HotpotQA EM / F1 | 2WikiMultiHopQA EM / F1 | MuSiQue EM / F1 | |||
|---|---|---|---|---|---|---|---|
| Qwen2.5-7B (no retrieval) | Single LLM | 18.5 | 26.8 | 16.2 | 23.7 | 4.1 | 9.1 |
| Single-Step RAG | Naive RAG | 31.5 | 44.2 | 25.9 | 37.6 | 14.1 | 18.4 |
| IRCoT | CoT | 42.7 | 54.8 | 43.3 | 56.2 | 18.8 | 23.9 |
| OPERA (CoT only) | CoT | 44.9 | 58.5 | 42.3 | 50.7 | 21.2 | 32.1 |
| Adaptive-RAG | SFT | 45.7 | 56.9 | 30.1 | 39.3 | 24.3 | 35.7 |
| BGM | RL | 41.5 | 53.8 | 44.3 | 55.8 | 19.6 | 26.8 |
| OPERA (MAPGRPO) | RL | 57.3 +11.6 | 69.5 +11.0 | 60.2 +15.9 | 72.7 +16.5 | 39.7 +15.4 | 58.0 +22.3 |
Deltas vs. the best baseline per column. All SFT/RL methods trained on the same mixed dataset; identical Qwen2.5 backbones and FAISS retrieval throughout.
What the ablations say
On MuSiQue, architecture beats training: removing the Plan Agent collapses EM from 39.7 to 17.1 — below the untrained CoT baseline — and removing Plan & Rewrite together lands at 16.7. Training follows a clean staircase: CoT 21.2 → SFT 24.3 → GRPO 34.8 → MAPGRPO 39.7 EM. Out of domain, MAPGRPO keeps its flexibility where SFT overfits: 36.6 EM on single-hop NQ (SFT: 19.5) and 55.7 EM on MultiHopRAG.
Citation
Cite OPERA
@misc{liu2025opera,
title = {{OPERA}: A Reinforcement Learning--Enhanced Orchestrated Planner-Executor
Architecture for Reasoning-Oriented Multi-Hop Retrieval},
author = {Yu Liu and Yanbing Liu and Fangfang Yuan and Cong Cao and Youbang Sun
and Kun Peng and Weizhuo Chen and Jianjun Li and Zhiyuan Ma},
year = {2025},
eprint = {2508.16438},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2508.16438}
}