OPERAOrchestrated Planner-Executor Reasoning Architecture
Ameame1  ›  OPERA  ›  Project Page

OPERAAAAI 2026

Institute of Information Engineering, Chinese Academy of Sciences Chinese Academy of Sciences Huazhong University of Science and Technology Tsinghua University

A Reinforcement Learning–Enhanced Orchestrated Planner-Executor Architecture for Reasoning-Oriented Multi-Hop Retrieval.

Existing RAG pipelines retrieve first and reason after. OPERA fuses the two: a hierarchical three-agent architecture reasons while retrieving — every retrieval step is shaped by the current reasoning state, and every reasoning step drives the next retrieval — trained end-to-end with a new multi-agent variant of GRPO.

Yu Liu1,2 · Yanbing Liu1,2 · Fangfang Yuan1 · Cong Cao1 · Youbang Sun4 · Kun Peng1,2 · Weizhuo Chen1,2 · Jianjun Li3 · Zhiyuan Ma3†

1Institute of Information Engineering, Chinese Academy of Sciences · 2School of Cyber Security, University of Chinese Academy of Sciences · 3Huazhong University of Science and Technology · 4Tsinghua University · †Corresponding author

🎉 News · 2025-11-08 — OPERA has been accepted to the AAAI 2026 Main Track. Also the first PhD work of the first author — thanks to everyone who helped along the way.

Overview

Retrieval and reasoning belong in the same loop

Consider: “What was the previous occupation of the person who succeeded the founder of the company that acquired WhatsApp?” Answering demands a precise sequence of retrievals and inferences, each depending on the last. Conventional retrieve-then-reason pipelines break down here for three reasons: rule-based decomposers produce brittle plans, query reformulation is too shallow to escape failed-retrieval loops, and golden documents drown in noisy top-k results. All three trace back to one root cause — weak coupling between retrieval and reasoning.

OPERA MAPGRPO training framework: role-specific rewards for Rewrite, Analysis-Answer and Plan agents; batch composition with expert injection; benchmark gains on HotpotQA, 2WikiMultiHopQA and MuSiQue.
The complete picture. Top & center: MAPGRPO trains the three agents sequentially with role-specific rewards and judges — a mixed online/offline batch (7/8 generated by the policy, 1/8 best pre-scored sample) stabilises early training. Bottom: the three failure modes of conventional RAG on a real multi-hop question. Right: resulting gains — EM 45.7 → 57.3 on HotpotQA, 44.3 → 60.2 on 2WikiMultiHopQA, 24.3 → 39.7 on MuSiQue.

Why OPERA

Retrieve-then-reason vs. reason-while-retrieving

✕ CONVENTIONAL RAG
  • Static plans. Rule-based or planner-first decomposition collapses on out-of-template questions and cannot adapt mid-flight.
  • Blind retries. Failed retrievals trigger shallow query tweaks, looping without ever reaching the golden documents.
  • Noise wins. Even when the right document is fetched, coarse filtering leaves it buried in noisy top-k context.
✓ OPERA
  • Reasoned plans. A dedicated Plan Agent decomposes the question into ordered sub-goals with explicit placeholder dependencies.
  • Reasoned retrieval. A specialised Rewrite Agent reformulates the query only when evidence is judged insufficient — trained on retrieval effectiveness (NDCG) itself.
  • Reasoned filtering. The Analysis-Answer Agent assesses sufficiency and extracts the precise answer before the pipeline moves on; a Trajectory Memory Component records the rationale of every action.

Architecture

Two modules, three specialised agents

GPM · Goal Planning Module

Plan Agent (Qwen2.5-7B)

Decomposes a multi-hop question into coherent, executable sub-goals with placeholder dependencies — strategy, decided once and revised only when execution demands it.

REM · Reason-Execute Module

Analysis-Answer Agent (Qwen2.5-7B)

Judges whether retrieved evidence suffices (φ), and if so extracts a precise answer for the current sub-goal — reasoning-guided filtering instead of blind top-k stuffing.

REM · Reason-Execute Module

Rewrite Agent (Qwen2.5-3B)

Activates only on insufficiency: reformulates the query from the reasoning state and feedback, then hands back to retrieval — turning failed lookups into successful ones.

  1. Plan. The Plan Agent decomposes the question into ordered sub-goals.
  2. Retrieve. BGE-M3 fetches top-5 evidence from the merged 1.78M-document FAISS index.
  3. Analyse & answer. The Analysis-Answer Agent answers the sub-goal when evidence is sufficient.
  4. Rewrite on failure. The Rewrite Agent reformulates the query only after insufficient evidence.
  5. Synthesise. Executed sub-goal answers are combined into the final answer; the TMC keeps the full audit trail.
OPERA architecture walkthrough on a live example: the Plan Agent emits a JSON plan with dependencies; the Analysis-Answer Agent reasons over retrieved evidence; the Rewrite Agent refines the query after feedback.
One question, end to end. The Plan Agent emits a structured JSON plan (right) whose second sub-goal depends on the first; the REM executes each sub-goal — reasoning, retrieving, and refining the query (“1962 Nobel Prize DNA Watson Crick winners”) when the first attempt lacks laureate names.

MAPGRPO

Train each agent for the job it actually does

Multi-Agents Progressive Group Relative Policy Optimization extends GRPO to a team of heterogeneous agents: agents are optimised sequentially — Plan, then Analysis-Answer, then Rewrite — each conditioned on its already-trained predecessors, with a reward tailored to its role:

Stage 1

Plan quality

Accuracy 30% · coherence 25% · executability 25% · dependency 10% · efficiency 10% — judged against pre-scored golden plans.

Stage 2

Reasoning accuracy

0.65 × answer EM + 0.25 × sufficiency judgement + 0.10 × format — precise credit for precise answers.

Stage 3

Retrieval effectiveness

√NDCG@k against golden documents + format compliance — the reward is the retrieval outcome.

Two ingredients keep training stable: expert injection (each group of G=8 candidates includes the best sample from a DeepSeek-R1 pre-scored pool, provably reducing policy-gradient variance) and group-relative advantages with a KL constraint. MAPGRPO converges to local optima at rate 𝒪(1/√T).

Walkthrough

Watch one question run the loop

QuestionWhat university did the Nobel Prize winner who discovered the structure of DNA attend for their undergraduate studies?
  1. 1PlanGPM · Plan Agent
  2. 2RetrieveBGE-M3 · top-5
  3. 3AnalyseREM · sufficiency φ
  4. 4RewriteREM · query refine
  5. 5Synthesisefinal answer

The Plan Agent decomposes the question into an ordered, dependency-aware plan:

Sub-goal 1. Who discovered the structure of DNA and won the Nobel Prize for it? expected: entity
Sub-goal 2. What university did [people from step 1] attend for their undergraduate studies? depends on: 1

Strategy is fixed up front — placeholders wire later steps to earlier answers.

BGE-M3 searches the 1.78M-document index for sub-goal 1:

DNA structure discovery Nobel Prize

“The 1962 Nobel Prize was awarded for discoveries concerning the molecular structure of DNA…”

Top-5 evidence returned to the Reason-Execute Module.

The Analysis-Answer Agent reasons over the evidence before answering:

Found the 1962 prize — but the passages never name the laureates.

φ = insufficient — instead of guessing, the sub-goal is handed to the Rewrite Agent with feedback: need laureate names.

The Rewrite Agent reformulates from the reasoning state, not by blind paraphrase:

1962 Nobel Prize DNA Watson Crick winners

Retrieval hit → Watson, Crick, Wilkins · 1962 Nobel Prize ✓

Sub-goal 1 answered. The refined query is rewarded by √NDCG@k during MAPGRPO training — retrieval effectiveness is the reward.

Sub-goal 2 executes with the placeholder resolved to Watson, Crick, Wilkins, retrieving each laureate’s undergraduate institution.

Executed sub-goal answers are synthesised into the final answer.

Done. The Trajectory Memory Component keeps the full audit trail — every plan, query, verdict and rationale.

Results

State of the art on three multi-hop benchmarks

57.3%
EM on HotpotQA
best baseline 45.7
60.2%
EM on 2WikiMultiHopQA
best baseline 44.3
39.7%
EM on MuSiQue
best baseline 24.3
+63.4%
relative EM gain
on the hardest benchmark
MethodTypeHotpotQA
EM / F1
2WikiMultiHopQA
EM / F1
MuSiQue
EM / F1
Qwen2.5-7B (no retrieval)Single LLM18.526.816.223.74.19.1
Single-Step RAGNaive RAG31.544.225.937.614.118.4
IRCoTCoT42.754.843.356.218.823.9
OPERA (CoT only)CoT44.958.542.350.721.232.1
Adaptive-RAGSFT45.756.930.139.324.335.7
BGMRL41.553.844.355.819.626.8
OPERA (MAPGRPO)RL57.3 +11.669.5 +11.060.2 +15.972.7 +16.539.7 +15.458.0 +22.3

Deltas vs. the best baseline per column. All SFT/RL methods trained on the same mixed dataset; identical Qwen2.5 backbones and FAISS retrieval throughout.

What the ablations say

On MuSiQue, architecture beats training: removing the Plan Agent collapses EM from 39.7 to 17.1 — below the untrained CoT baseline — and removing Plan & Rewrite together lands at 16.7. Training follows a clean staircase: CoT 21.2 → SFT 24.3 → GRPO 34.8 → MAPGRPO 39.7 EM. Out of domain, MAPGRPO keeps its flexibility where SFT overfits: 36.6 EM on single-hop NQ (SFT: 19.5) and 55.7 EM on MultiHopRAG.

Citation

Cite OPERA

@misc{liu2025opera,
  title         = {{OPERA}: A Reinforcement Learning--Enhanced Orchestrated Planner-Executor
                   Architecture for Reasoning-Oriented Multi-Hop Retrieval},
  author        = {Yu Liu and Yanbing Liu and Fangfang Yuan and Cong Cao and Youbang Sun
                   and Kun Peng and Weizhuo Chen and Jianjun Li and Zhiyuan Ma},
  year          = {2025},
  eprint        = {2508.16438},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2508.16438}
}