Issue Match Audit

Check the AI judges' matches between human evaluators' concerns and model-raised issues. Match from scratch instead

Not saved

Keys: Y same P partly N different U unsure · J/K next/previous

Add other reviewers

Load JSON exports from this page (another reviewer's verdicts) or from the from-scratch annotator. Nothing is uploaded; files are read in your browser.

    What you are checking

    For each paper there are two lists of concerns: one from the human evaluators and one from a model that read only the paper. An AI judge then said which items raise the same concern. This page shows each pair the judge proposed and asks you whether it is right.

    • Same concern: the same problem, mechanism and implication.
    • Partly: a shared substantive core, but materially different scope or specifics.
    • Different: different problems, even if they touch the same topic or table.
    • Unsure: you cannot tell from the text shown. Unsure verdicts are left out of all metrics.

    The fastest useful order is: Contested first (where the judges disagree or the score is borderline), then Human issues no judge matched (to catch missed matches), then the rest. When you finish a human issue in the misses view, press No match among these; its remaining pairs then count as "different" when computing the judges' recall.

    Saving and sharing

    Your verdicts are kept in this browser only. Download the JSON when you are done and save it to llm-uj-research-eval/data/human_issue_match_annotations/audit/; python scripts/match_audit.py score then computes the same metrics as the Agreement tab.

    Where the data comes from

    The page loads the latest results/match_audit/audit_data.json from the public llm-uj-research-eval repository, which scripts/match_audit.py export regenerates whenever a judged run is summarised. If that cannot be reached, it uses the copy bundled with this page. The badge at the top says which one you are seeing.

    Human issues are taken from published Unjournal evaluations or from evaluation managers' curated summaries. Evaluators are referred to as E1, E2; names are not shown.

    Metrics on the Agreement tab

    • Judge vs judge: whether two judges agree that a pair, or a human issue, is matched. Needs no human input.
    • Precision: of the judge's proposed matches you reviewed, the share you marked same or partly.
    • Recall: of the matches you confirmed or added, the share the judge proposed.
    • Cohen's kappa: agreement beyond chance on reviewed pairs (1 = perfect, 0 = chance).