Tweet Annotation Sensitivity Experiment 2

Annotators could select one or both of hate speech, offensive language, or indicate that neither applied. Conditions B and C split the annotation of a single tweet across two screens. For Condition B , the first screen prompted the annotator to indicate whether the tweet contained hate speech.

On the following screen, they were shown the tweet again and asked whether it contained offensive language. Condition C was similar to Condition B, but flipped the order of hate speech and offensive language for each tweet. In Conditions D and E, the two tasks are treated independently with annotators being asked to first annotate all tweets for one task, followed by annotating all tweets again for the second task.

Annotators assigned Condition D were first asked to annotate hate speech for all their assigned tweets, and then asked to annotate offensive language for the same set of tweets. Condition E worked the same way, but started with the offensive language annotation task followed by the hate speech annotation task. We recruited US-based annotators from the crowdsourcing platform Prolific during November and December 2022.

Each annotator annotated up to 50 tweets. The dataset also contains demographic information about the annotators. Annotators received a fixed hourly wage in excess of the US federal minimum wage after completing the task.