# Issue Annotation Tools

Tools for comparing human expert critiques with LLM-generated research issue assessments.

> **[Open the Annotation UI](https://daaronr.github.io/unjournal_tools_interfaces/annotation/ui/)** — Interactive browser-based annotation tool

## Overview

This directory contains a pipeline for systematically evaluating how well LLM issue detection aligns with human expert critiques in research paper evaluation.

## Tools

### `build_issue_annotation_data.py`
Data pipeline that prepares data for manual annotation:
- Reads human critiques and LLM assessment reports
- Parses text to extract individual issues
- Generates severity labels (necessary/optional/unsure)
- Outputs combined JSON dataset for the annotation UI

### `match_issues_embeddings.py`
Automated issue matching using sentence embeddings:
- Uses `sentence-transformers` (MiniLM-L6-v2 model)
- Calculates cosine similarity between human and LLM issues
- Similarity threshold: 0.35
- Outputs `results/key_issues_matched.json`

### `compare_issues_llm.py`
Advanced semantic matching using GPT API:
- Provides detailed semantic analysis
- Generates match explanations and comparative analysis
- Outputs coverage metrics and quality scores

### `ui/` - Browser-based Annotation Tool
Interactive UI for human annotators to manually score matches:
- 3-panel layout: Human critiques | LLM issues | Detailed matching
- Match scoring (0-1 slider)
- Confidence ratings
- Export as JSON or CSV

## Usage

```bash
# Build annotation data
python build_issue_annotation_data.py

# Run embedding-based matching
python match_issues_embeddings.py

# Run LLM-based matching (requires OpenAI API key in key/openai_key.txt)
python compare_issues_llm.py

# Use the UI
open ui/index.html
```

## Dependencies

```bash
pip install sentence-transformers numpy openai
```

## Origin

These tools were originally developed in the [`llm-uj-research-eval`](https://github.com/valentinklotzbuecher/llm-uj-research-eval) repository, which tests LLM evaluation quality against human Unjournal evaluations. See the [live demo](https://llm-uj-research-eval.netlify.app/).