Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judge

EMNLP 2026

AgentAuditor: A structure-aware auditing framework that resolves multi-agent reasoning conflicts through localized evidence verification instead of majority voting.

Overview • Datasets • Training Data • Setup • Run • Training • Citation

---

📌 Overview

Multi-agent systems (MAS) have become an effective paradigm for improving LLM reasoning by enabling diverse agents to explore multiple reasoning trajectories. However, existing MAS frameworks commonly aggregate agent outputs through majority voting, which treats agents as independent voters and ignores the underlying reasoning evidence.

This assumption breaks under confabulation consensus, where multiple agents share correlated biases and converge on the same incorrect reasoning path.

We introduce AgentAuditor, a structure-aware aggregation framework that organizes multi-agent reasoning traces into a Reasoning Tree. Instead of selecting answers based on frequency, AgentAuditor identifies Critical Divergence Points (CDPs) and performs localized branch-level evidence auditing.

Furthermore, we propose Anti-Consensus Preference Optimization (ACPO), which trains the Auditor with evidence-verified preference supervision to reduce conformity to misleading majority signals.

This repository provides the official implementation, data-processing pipeline, training code, and evaluation scripts for AgentAuditor.

---

📝 Abstract

Multi-agent systems (MAS) can substantially extend the reasoning capacity of large language models (LLMs). Most MAS frameworks aggregate agent outputs via simple majority voting, discarding the evidential structure of reasoning traces. Majority voting is brittle under confabulation consensus, where agents share correlated biases and converge on the same incorrect rationale. We introduce AGENTAUDITOR, which moves beyond frequency-based aggregation by organizing agent traces into a Reasoning Tree that explicitly represents agreements and divergences in their reasoning. AGENTAUDITOR resolves conflicts by comparing branch-level evidence at critical divergence points, turning global adjudication into efficient, localized verification. We further propose Anti-Consensus Preference Optimization (ACPO), which trains the adjudicator with evidence-verified preference supervision to reduce conformity to misleading majority cues.

---

✨ Key Features

• Reasoning Tree construction that captures shared prefixes and divergent branches across multiple reasoning traces.
• Critical Divergence Point auditing that verifies localized branch evidence instead of judging complete traces blindly.
• Non-voting aggregation that remains robust when multiple agents converge on the same incorrect rationale.
• Anti-Consensus Preference Optimization (ACPO) with evidence-verified, grouped-fork preference supervision.
• Support for QA, mathematical reasoning, and multiple-choice benchmarks.
• Flexible vLLM, Hugging Face LLaMA, and OpenAI-compatible inference backends.
• Scripts for prompt-only inference, LoRA inference, grouped-fork data construction, and multi-GPU training.

---

📂 Datasets

Public Benchmark Datasets

Due to licensing and redistribution restrictions, we do not directly redistribute the original benchmark datasets. Please download them from their official releases, including those hosted on:

• 📦 Hugging Face.

The current implementation provides adapters for MuSiQue, TriviaQA, MMLU, GSM8K, MATH500, and AMC.

Demo Dataset

To quickly verify the complete pipeline, we provide lightweight, self-contained demo bundles. Each JSONL row contains a question and its K agent reasoning traces under responses; the sibling .gt.json file contains evaluation metadata and cached results. Google Drive

After downloading, place the files under data/.

The dataset name passed to the scripts must be one of musique, triviaqa, mmlu, gsm8k, math500, or amc.

---

🧠 Training Data & Checkpoints

We provide a demo grouped-fork ACPO training data used by the training pipeline: Google Drive

Notes on Checkpoint Usage

You can train an adapter with scripts/train.sh, which writes the resulting PEFT adapter to runs/fork_judge_<dataset>/ by default.

For LoRA inference, the adapter directory should contain files such as adapter_config.json and adapter_model.safetensors. Pass the directory through ADAPTER=... when running scripts/run_lora.sh.

---

⚙️ Environment Setup

We provide two environment configuration files: environment.yml and requirements.txt.

Option 1: Conda (Recommended)

conda env create -f environment.yml
conda activate agentauditor

The Conda specification installs the package in editable mode. If needed, run pip install -e . afterward.

Option 2: Manual / Existing Conda Environment

conda create -n agentauditor python=3.10 -y
conda activate agentauditor
pip install -r requirements.txt
pip install -e .

Option 3: Update an Existing Environment

conda env update -f environment.yml --prune
conda activate agentauditor

Suggested Dependency Stack

• PyTorch
• Transformers
• vLLM and Sentence Transformers
• TRL 1.9.2
• PEFT / LoRA and Accelerate
• Hugging Face Datasets
• OpenAI Python SDK (optional backend)

GPU installations are platform-specific. Make sure your NVIDIA driver, CUDA runtime, PyTorch, and vLLM versions are compatible.

---

Practical Setup Tips

• Run every command from the project root.
• Prompt and LoRA inference require a vLLM-compatible GPU environment.
• The default base model is meta-llama/Llama-3.1-8B-Instruct; set MODEL= or AGENTAUDITOR_BASE_MODEL to use a local path or another compatible model.
• ACPO training uses TRL 1.9.2; PYTRAIN= can point to a dedicated training environment if inference and training dependencies conflict.
• The cached demo below runs without loading a model and therefore does not require a GPU.

---

🚀 Running the Code

We provide a lightweight demo and two main inference modes. All commands should be executed from the project root.

Tip: Script settings are environment-variable driven, so experiments can be changed without editing the shell files.

---

① ⚡ Quick Demo (No GPU)

The three-input demo takes dataset name, K reasoning traces per sample, and data path. Its default cache mode prints the cached method ladder (MV → prompt → trained) and McNemar statistics without model inference.

PYTHONPATH=src python -m agentauditor.eval.demo \
  --dataset musique \
  --k 3 \
  --data data/musique_demo_data.jsonl

Other examples:

PYTHONPATH=src python -m agentauditor.eval.demo --dataset triviaqa --k 3 --data data/triviaqa_demo_data.jsonl
PYTHONPATH=src python -m agentauditor.eval.demo --dataset mmlu     --k 3 --data data/mmlu_demo_data.jsonl
PYTHONPATH=src python -m agentauditor.eval.demo --dataset math500  --k 3 --data data/math500_demo_data.jsonl

② 🔍 AgentAuditor Inference

This mode runs the fork judge with the base model and no LoRA adapter. Dataset-specific settings are loaded automatically from src/agentauditor/data/adapters.py.

• 📌 Script: scripts/run.sh

DATASET=musique GPU=0 bash scripts/run.sh

To customize the interpreter, model, input, and output paths:

PY=/path/to/vllm-env/bin/python \
MODEL=meta-llama/Llama-3.1-8B-Instruct \
DATASET=musique \
IN=data/musique_demo_data.jsonl \
OUT=out/musique_prompt.jsonl \
GPU=0 \
bash scripts/run.sh

③ 🧩 AgentAuditor Inference with LoRA

This mode loads a trained LoRA fork judge on top of the base model.

• 📌 Script: scripts/run_lora.sh
• 🔧 Required setting: ADAPTER=<trained-adapter-directory>

DATASET=musique \
ADAPTER=runs/fork_judge_musique_ctx \
GPU=0 \
bash scripts/run_lora.sh

You may also configure a persistent dataset-specific adapter path:

export AGENTAUDITOR_ADAPTER_MUSIQUE=/path/to/fork_judge_musique_ctx
DATASET=musique bash scripts/run_lora.sh

④ 🧪 Live End-to-End Demo

live mode invokes the base and trained fork judges on the provided crowd data. It requires a GPU, a vLLM environment, and a trained adapter.

PYTHONPATH=src python -m agentauditor.eval.demo \
  --dataset musique \
  --k 3 \
  --data data/musique_demo_data.jsonl \
  --mode live \
  --limit 30 \
  --gpu 0 \
  --python /path/to/vllm-env/bin/python \
  --adapter runs/fork_judge_musique_ctx

---

🏋️ Training

Training consists of two stages: constructing grouped-fork ACPO units from a train-split crowd cache, and training the LoRA fork judge.

⑤ 🧱 Build ACPO Training Data

• 📌 Script: scripts/build_traindata.sh

The expected source is a crowd cache such as crowd_cache/musique_aux_ctx_diverse.jsonl. For QA data, the script also expects its sibling .gt.json file.

DATASET=musique USECTX=1 bash scripts/build_traindata.sh

For another crowd source or destination:

DATASET=triviaqa \
USECTX=0 \
GEN=crowd_cache/triviaqa_aux_diverse \
OUTDIR=train_data/fork_units/triviaqa \
DMG_RATIO=1.5 \
bash scripts/build_traindata.sh

The pipeline:

1. extracts answer-divergent forks and creates chosen/rejected ACPO pairs;
2. applies damage-guard downsampling while retaining recovery forks;
3. builds permutation-balanced and all-negative units with a fork-grouped train/validation split.

⑥ 🧠 Train the ACPO LoRA Fork Judge

• 📌 Script: scripts/train.sh

DATASET=musique USECTX=1 NPROC=8 bash scripts/train.sh

To use a dedicated training environment or custom paths:

PYTRAIN=/path/to/train-env/bin/python \
DATASET=musique \
DATA=train_data/fork_units/musique_ctx \
OUT=runs/fork_judge_musique_ctx \
NPROC=8 \
LR=5e-5 \
EP=3 \
bash scripts/train.sh

The trained adapter is written to the directory specified by OUT and can be passed directly to scripts/run_lora.sh.

---

Training Notes

• train_units.jsonl must exist in the directory passed through DATA=.
• Set NPROC to the number of available GPUs; the script sets CUDA_VISIBLE_DEVICES accordingly.
• MAXLEN, LR, EP, and GA control maximum sequence length, learning rate, epochs, and gradient accumulation.
• The base model can be changed with AGENTAUDITOR_BASE_MODEL.
• The downloaded prebuilt training units let you skip scripts/build_traindata.sh and run training directly.

---

📁 Recommended Repository Contents

.
├── README.md
├── environment.yml
├── requirements.txt
├── pyproject.toml
├── data/                         # self-contained demo bundles
├── train_data/                   # grouped-fork ACPO units
├── scripts/
│   ├── run.sh                    # prompt-only inference
│   ├── run_lora.sh               # inference with a trained LoRA
│   ├── build_traindata.sh        # crowd cache -> ACPO units
│   └── train.sh                  # grouped-fork ACPO training
└── src/agentauditor/
    ├── core/                     # reasoning tree, auditing, and adjudication
    ├── data/                     # dataset adapters and normalization
    ├── dataprep/                 # preference-data construction
    ├── eval/                     # demos, scorers, and metrics
    ├── llm/                      # vLLM, LLaMA, and OpenAI backends
    └── train/                    # grouped-fork ACPO trainer

---

🔬 Reproducibility Tips

• Record the dataset, K, data split, base model, and adapter path for every run.
• Keep the base model and LoRA adapter compatible.
• Use the dataset adapters in src/agentauditor/data/adapters.py rather than manually changing method-specific flags.
• Fix random seeds when constructing new preference data.
• Keep generated predictions in JSONL format for later re-evaluation.
• Report both recovery and damage counts, along with McNemar significance, when comparing against majority vote.
• Distinguish cached demo reproduction from live model inference in reported results.

---

📖 Citation

If you use this work in your research, please cite:

@article{yang2026auditing,
  title={Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judge},
  author={Yang, Wei and Li, Shixuan and Ping, Heng and Zhang, Peiyu and Bogdan, Paul and Thomason, Jesse},
  journal={arXiv preprint arXiv:2602.09341},
  year={2026}
}