Potential evaluation tools and guidelines for Unjournal evaluators
As of January 2025, this is an incomplete list of tools we've tried or are considering. We aim to make this a more carefully curated and vetted list.
← Back to Evaluation FormThe Unjournal encourages the responsible use of AI tools to enhance evaluation quality. Key principles:
See the full policy: Unjournal's AI/LLM policy proposal
Automated regression and statistical analysis checking for research papers.
Open RegCheck →Checks statistical reporting consistency: verifies that p-values, test statistics, and degrees of freedom in papers are internally consistent.
It checks the arithmetic of reported tests, not whether the test was the right one.
Treat AI output as leads to check, not as judgments. In our comparison of AI and human evaluations of Unjournal papers, a frontier model came close to the agreement between human evaluators on several criteria, but compressed its ratings, covered the criteria unevenly, and caught expert-flagged problems inconsistently. Chat assistants can also give different answers when asked the same question twice.
AI agent-based and LLM-based tool including fact checking and reasoning checking.
Useful for surfacing candidate issues; treat each point as a lead to check.
Pro model shows substantial insight and potential; see our piloting here.
Good for summaries, restating arguments and drafting; check every claim it makes about the paper against the paper.
Open ChatGPT →Deep-dive analysis of individual papers. Upload a PDF and ask targeted questions about methodology, findings, and limitations.
Its answers are grounded in the files you upload: good for finding and recalling passages, not for judging them.
AI research paper analysis providing structured feedback and summaries.
Open Paper-Wizard →Runs many model passes over a manuscript and returns a ranked list of technical errors, such as gaps in derivations, numerical mistakes, and inconsistencies between text and tables, usually in under an hour. First document free.
Its benchmark results are its own. Treat each flagged issue as a lead to check.
Open refine.ink →AI assistant; Opus model useful for deep analysis of research papers and evaluation assistance.
As with other chat assistants, verify what it says about the paper, and don't use it for ratings.
Open Claude →AI-powered search with cited sources; useful for quick literature checks and fact verification.
Check that each cited source says what it is cited for.
Open Perplexity →AI-powered research assistant for finding related papers, extracting key findings, and answering research questions.
Search engine for research findings. Quickly see what the scientific consensus is on specific claims.
AI-powered academic search engine for finding and exploring research papers and citation networks.
Smart citations showing whether papers support or contradict cited claims. Helps verify how a paper's references and citing works relate to its key claims.
Vector-space approach for finding related concepts and research. Exploring integration with Unjournal evaluation data and RePEc.
The Transparency and Openness Promotion (TOP) guidelines from the Center for Open Science. A framework for evaluating transparency standards across 8 dimensions: citation standards, data transparency, analytic methods, research materials, design and analysis, study preregistration, analysis plan preregistration, and replication.