Enables rigorous review of ML research papers following official ICML guidelines.
Workflow
Step 1: Input Analysis & Mode Selection
Determine input type:
Complete paper
PDF/text with abstract, methodology, experiments, results → Full Review Mode
Incomplete document
Missing major sections, labeled draft/proposal, or user indicates early stage → Early-Stage Feedback Mode
Code repository
User points to folder/repo path → Repository Review Mode
For complete papers
, extract: title, abstract, main claims, methodology, experiments, results. Identify paper type: theoretical, methodological, algorithmic, empirical, bridge paper, or application-driven.
For code repositories
, first explore: read README, scan code structure, find experiment scripts/results, identify the research question and what's implemented.
Step 2: Prior Work Grounding (Critical - All Modes)
This step applies to ALL input types. Grounding in reality is essential for any meaningful feedback.
Generate 3-5 search queries based on the research topic: benchmarks/baselines, same problem, related techniques
Use WebSearch to find recent arXiv papers and published work
Fetch abstracts of 5-10 most relevant papers
Critically synthesize
:
What specific claims in this paper are already addressed by prior work?
What are the actual quantitative improvements over recent baselines?
Are claimed "novelties" actually novel given the literature?
What gaps truly exist vs. what the authors claim exists?
Critical mindset
:
Your job is to verify claims against reality, not accept them at face value
Most papers overclaim—your review should ground their contributions in what the literature actually shows
Default to skepticism: Assume claims are overstated until proven otherwise by evidence
Authors have selection bias toward their own work; you represent the community's interests
Be the critical voice that ensures published work actually advances the field
Then proceed to mode-specific evaluation.
Full Review Mode (Complete Papers)
Step 3: Systematic Evaluation
Evaluate across 7 dimensions (see
references/evaluation-criteria.md
).
Default to skepticism—require strong evidence to score highly.
Dimension
Key Questions (Answer with Literature Evidence)
Originality
Is this truly novel given recent work X, Y, Z? What specific aspects are incremental vs. novel?
Importance
Why does this problem matter? What's the real-world impact? Who will care?
Claims Support
Do experiments actually prove the claims? What alternative explanations exist?
Experimental Soundness
Are baselines from 2023+? Are comparisons fair? What's missing?
Clarity
Can I reproduce this from the paper? Are claims precisely stated?
Community Value
Will this change how people work? Or just add noise?
Prior Work Context
Are comparisons accurate? What recent work (last 2 years) is missing?
Evaluation mindset
:
Start from neutral and require evidence to move up or down
Compare every claim against what you found in the literature search
Most papers are incremental—high originality scores are rare
Weak baselines or missing comparisons are critical flaws, not minor issues
Step 4: Critical Cross-Check Against Literature
Before writing the review, explicitly verify:
Baselines check
List baselines used in paper. List baselines from your literature search of adjacent papers. What's missing?
Methodology check
How do 2-3 adjacent papers approach this problem? Does this paper follow similar methodology? If not, why not?
Claims check
List main claims. For each, cite specific evidence from experiments or proofs. If insufficient, note it.
Citations check
Which papers from your search are cited? Which are missing? Why?
Novelty check
List claimed novelties. For each, cite specific prior work that does or doesn't do this.
This step is not optional. Your review must reference specific findings from your literature search.
Step 5: Generate Review
Follow the ICML review form (see
references/review-template.md
):
Summary
- Neutral, factual (should not be disputed by authors)
Claims and Evidence
- Are claims supported?
Compare to what literature shows
Relation to Prior Work
- Proper context? Missing citations?
List specific missing papers
Strengths
- Specific and substantive,
compared to standards in adjacent work
Weaknesses
- Constructive, explain severity,
cite specific literature for comparison
Questions for Authors
- Numbered, explain impact on evaluation
Minor Issues
- Typos, suggestions
Overall Recommendation
- 1-5 scale with justification
grounded in literature comparison
Confidence Score
- 1-5 scale
Step 6: Quality Check
Verify all claims in review are substantiated
Ensure constructive tone
Check specificity of strengths/weaknesses
Confirm questions are actionable
Key Principles
Be Rigorous AND Constructive
Your primary duty is to the research community—publishing weak papers dilutes the literature.
Be honest
Don't inflate scores to be nice. If baselines are weak, say so clearly.
Be specific
Always cite which literature contradicts or supports claims.
Be fair
Criticism should be substantiated by evidence or literature.
Be actionable
Tell authors exactly what would fix the issues.
"Review the papers of others as you would wish your own to be reviewed"—with rigor, honesty, and specific feedback grounded in the literature.
Be Specific
Bad: "The experiments are weak"
Good: "Experiments compare only against [X] from 2019, but recent baselines [Y] (2024) and [Z] (2024) should be included."
Fair Novelty Assessment
Originality may arise from: creative combinations, new domains, removing restrictive assumptions, novel datasets, new problem formulations.
But
Most claimed novelty is actually incremental. Verify against literature before accepting novelty claims.
Score Calibration
Use this reference frame:
5s are rare
Reserve for papers that will clearly influence the field
4s are uncommon
Solid papers with rigorous execution and clear contributions
3s are common
Papers with merit but significant limitations
2s are common
Incremental work or work with major methodological issues
1s indicate fundamental problems
Wrong results, no contribution, or severe ethical issues
If you find yourself giving mostly 4s and 5s, you're likely being too generous. Re-calibrate against what the literature shows is standard.
Application-Driven Papers
For application-driven ML: methods should fit real-world constraints, non-standard datasets acceptable if documented, compare against domain baselines.
Rating Scales
Overall (1-5):
Use the full range. Most papers should be 2-3.
5 (Strong Accept)
Significant contribution, will be influential, no major flaws
4 (Accept)
Solid contribution, rigorous execution, minor issues only
3 (Weak Accept)
Contribution exists but limited; or good idea with execution flaws
2 (Weak Reject)
Incremental contribution insufficient for venue; or significant methodological issues
1 (Reject)
Fundamental flaws, not ready, or no meaningful contribution
Red flags that should lower scores
:
Baselines older than 2 years (unless explicitly justified)
Missing comparisons to obvious related work from literature search
Claims not directly supported by presented experiments
Novelty claims contradicted by prior work
Confidence (1-5):
5=Expert/certain, 4=Confident, 3=Fairly confident, 2=Uncertain, 1=Not in area
Early-Stage Feedback Mode
Use this mode for incomplete drafts, research proposals, or code repositories. Focus shifts from "accept/reject evaluation" to "constructive guidance on how to make this publishable."
After completing Steps 1-2 (input analysis and prior work grounding), proceed here.
Step 3: Generate Formative Feedback
Use the Early-Stage Feedback Template (see
references/review-template.md
). No numerical scores—focus on constructive guidance.
For code repositories
, additionally address:
Code quality and organization
Experiment design and reproducibility
What's missing for a paper (baselines, ablations, analysis)
References
references/evaluation-criteria.md
- Detailed criteria for each dimension
references/review-template.md
- Full template with examples
references/common-issues.md
- Common paper issues to identify