Unjournal Evaluation Form

Judge the quality of the research heuristically. Consider all aspects of quality, credibility, importance to future impactful applied research, and practical relevance and usefulness, importance to knowledge production, and importance to practice.

Benchmark: serious research in the same area encountered in the last three years.

Do the authors do a good job of:

• Stating their main questions and claims clearly?
• Providing strong evidence and powerful approaches to inform these?
• Correctly characterizing the nature of their evidence?

Consider the following:

• Are the methods well-justified and explained?
• Are they a reasonable approach to answering the question(s) in this context?
• Are the underlying assumptions reasonable?
• Are the results and methods likely to be robust to reasonable changes in assumptions?
• Does the author demonstrate robustness?
• Did the authors take steps to reduce bias from opportunistic reporting and questionable research practices?

To what extent does the project contribute to the field or to practice, particularly in ways relevant to global priorities and impactful interventions?

• Focus on "improvements that are actually helpful" (applied stream)
• Originality and cleverness should be weighted less than typical journals — we focus on impact
• More weight on "contribution to global priorities" than "contribution to academic field"
• Do the paper's insights inform beliefs about important parameters and intervention effectiveness?
• Does the project add useful value to other impactful research?
• Sound, well-presented null results can also be valuable

• Are goals and questions clearly expressed?
• Are concepts clearly defined and referenced?
• Is the reasoning transparent? Assumptions explicit?
• Are all logical steps clear and correct?
• Does the writing make arguments easy to follow?
• Are conclusions consistent with the evidence presented?
• Do authors accurately characterize evidence and its support for main claims?
• Are data and analysis relevant to the arguments?
• Are tables, graphs, diagrams easy to understand (no major labeling errors)?

This covers several considerations:

Replicability, reproducibility, data integrity: Would another researcher be able to perform the same analysis and get the same results? Are methods explained clearly enough for credible replication? Is code provided? Is data source clear and as available as reasonably possible?

Consistency: Do numbers in the paper and code output make sense? Are they internally consistent throughout?

Useful building blocks: Do authors provide tools, resources, data, and outputs that might enable future work and meta-analysis?

Reference: COS TOP Guidelines — a framework for evaluating transparency across 8 dimensions including data, code, materials, and preregistration.

• Is the topic and approach useful to global priorities, cause prioritization, and high-impact interventions?
• Does the paper consider real-world relevance, policy, and implementation questions?
• Are the setup, assumptions, and focus realistic?
• Do authors report results relevant to practitioners?
• Do they provide useful quantified estimates (costs, benefits) for impact quantification?
• Do they communicate in ways policymakers can understand without misleading oversimplification?