ReviewArcade

A research framework for investigating Large Language Model (LLM) susceptibility to non-semantic, cosmetic edits in paper reviews. ReviewArcade demonstrates that review decisions can be flipped from "reject" to "accept" through re-framing and re-wording, without changing the underlying research contributions.

Quick Start

# 1. Create and activate a virtual environment
python -m venv venv
source venv/bin/activate  # or `.\venv\Scripts\activate` on Windows

# 2. Install dependencies
uv sync  # or `pip install .`

# 3. Run a sample experiment
python main.py

System Overview

ReviewArcade automates an iterative review-editing loop using a multi-agent system:

┌─────────────────────────────────────────────────────────────┐
│              Iterative Editing Pipeline                      │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  Paper Input → [Judge] → Score & Critique                   │
│                   ↓                                          │
│            [Reached Target?]                                │
│            ↙           ↘                                    │
│          NO             YES → [Final Judge Panel] → Results  │
│          ↓                                                   │
│       [Editor] → Propose Edit (Allowed Action)              │
│          ↓                                                   │
│     [Edit-Critic] → Validate Diff                           │
│          ↓                                                   │
│      Apply Edit → Loop Back                                 │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Key Agents:

• Judge: Evaluates the paper, providing scores (Soundness, Excitement, Overall) and constructive criticism
• Editor: Proposes one Allowed Action (e.g., Clarification, Re-framing) and generates a Git-style diff
• Edit-Critic: Validates that the diff correctly implements the selected action and doesn't violate the taxonomy
• Final Judge Panel: Evaluates papers once they reach the target score or iteration limit

Architecture

The system is modular and extensible:

Component	Purpose
src/main.py	Entry point; orchestrates the pipeline for multiple papers
src/core/	Core editing loop and state management
src/evaluation/	Scoring and evaluation logic
src/data_preprocess/	Dataset loading and preprocessing (HF + PDF conversion)
src/graphics/	Visualizations and plotting utilities
conf/	All Hydra configuration files
visualizer/app.py	Interactive web dashboard for exploring results

Important Files:

• src/main.py — Pipeline orchestration
• src/core/editing.py — Multi-agent iterative loop
• src/models/llm.py — LLM interface (OpenAI-compatible)
• src/taxonomy.py — Allowed and disallowed edit actions
• src/logger.py — Per-paper logging and state tracking
• src/data_preprocess/transform_pdf.py — PDF-to-Markdown conversion

Installation

Requirements

• Python 3.13+
• An LLM endpoint (local vLLM or OpenAI)

Setup Steps

1. Create Virtual Environment

python -m venv venv
source venv/bin/activate

2. Install with uv (recommended)

uv sync

Or with pip:

pip install .

Configuration with Hydra

Everything is configured through Hydra. No hardcoding—override any value from the CLI or YAML files.

Config Structure

conf/
├── config.yaml              # Main config (models, paths, thresholds)
├── models/
│   ├── _default.yaml        # Default model profile
│   ├── seb.yaml             # SEB setup profile
│   ├── hcds.yaml            # HCDS setup profile
│   └── openai.yaml          # OpenAI profile
└── dataset/
    ├── _default.yaml
    └── reviewarcade.yaml    # ReviewArcade benchmark config

Main Configuration

conf/config.yaml defines:

models:
  judge: "meta-llama/Llama-3.1-70b-instruct"
  editor: "meta-llama/Llama-3.1-70b-instruct"
  edit_critic: "meta-llama/Llama-3.1-70b-instruct"
  final_judges:
    - "meta-llama/Llama-3.1-70b-instruct"

openai_api_base: "http://localhost:8000/v1"
openai_api_key: "EMPTY"

max_iterations: 10
acceptance_threshold: 4.0
log_dir: "logs"
output_dir: "results"

conf/dataset/reviewarcade.yaml specifies:

hf_dataset_name: "G4KMU/review_arcade"
hf_dataset_split: "rejected"
num_papers: 5
batch_size: 50
filter_human_scores: true

Usage

Single Run

Run with default configuration:

python main.py

Switch model profiles:

python main.py models=hcds
python main.py models=openai

Override individual settings:

python main.py dataset.num_papers=20 dataset.batch_size=10
python main.py max_iterations=15 acceptance_threshold=3.5
python main.py output_dir=results_exp_01 log_dir=logs_exp_01

Multirun: Sweep Over Configurations

Run multiple configurations in parallel with -m:

# Sweep num_papers: 5, 10, 20
python main.py -m dataset.num_papers=5,10,20

# Sweep models and papers (4 jobs total)
python main.py -m models=seb,hcds dataset.num_papers=5,10

# Sweep splits and acceptance thresholds
python main.py -m dataset.hf_dataset_split=rejected,accepted acceptance_threshold=3.0,4.0

Each multirun job runs independently; Hydra handles job scheduling and output directories.

Run with OpenAI

export OPENAI_API_KEY="sk-..."
python main.py models=openai

OpenAI Batch API (Cost-Efficient Large Runs)

For 20+ papers with OpenAI, use the Batch API (50% discount, higher rate limits):

python main.py models=openai use_openai_batch=True

The pipeline will:

1. Bundle all papers into a batch
2. Upload to OpenAI and poll for completion
3. Continue to the next iteration once done

This avoids rate limiting and significantly reduces costs.

Interactive Results Dashboard

Explore results in a web interface:

cd visualizer
python app.py

Open http://localhost:5000 in your browser to explore:

• Paper scores over iterations
• Edit action timelines
• Judge panel comparisons
• Statistical summaries

Advanced Features

PDF-to-Markdown Conversion

Convert scientific PDFs to Markdown for custom datasets:

python src/data_preprocess/transform_pdf.py /path/to/pdf_directory/

Uses olmocr for OCR-enhanced extraction.

Custom Edit Actions

Extend the taxonomy of allowed edits by modifying src/taxonomy.py:

ALLOWED_ACTIONS = [
    "Clarification",      # Improve clarity without changing meaning
    "Re-framing",         # Present results/methods in different light
    "Emphasis",           # Highlight existing strong points
    "Reorganization",     # Reorder sections for better flow
    # Add custom actions here
]