HF4ATS Data

The checkpoints used to generate these preference pairs were 7-8B parameter LLMs tuned using HF4ATS-SFT data (these checkpoints as well as the SFT data are also available on SWISSUbase). The preference pair subsets are as follows: 1. preferences_ie.jsonl: all preferences on ATS pairs with information equality according to pair creators. 2. preferences_iter.jsonl: all preferences from the four target annotators or two expert annotators with the highest inter-annotation agreement. 3. preferences_itra.jsonl: all preferences from the four target annotators or two expert annotators with the highest intra-annotation agreement. 4. preferences_<model_name>.jsonl: all preferences on ATS pairs generated by the <model_name> checkpoint (these checkpoints result from supervised fine-tuning on HF4ATS-SFT).

Additionally, we include all raw annotation files collected during the course of this project. These files include duplicated ATS pairs and pairs that had not been annotated at time of submission. They are aggregated and de-duplicated at the annotator-day level inside preferences_raw.jsonl. preferences_raw.jsonl also contains a field indicating a given annotation's reason for exclusion from preferences_all.jsonl (as well as all subsequent subsets).