Assessing construct reliability through open-ended survey response analysis

We used the software Turnitin™ to assess the originality of open-ended responses in four mixed-method online surveys that included validated multi-item rating scales. Across surveys, 18-35% of participants were identified as having copied responses from online sources.

We assessed indicator reliability and internal consistency reliability and found that both were lower for participants identified as using copied text versus those who wrote more original responses. Those who provided more original responses also provided more consistent responses to the validated scales, suggesting that these participants were more attentive. We conclude that this process can be used to screen qualitative responses from online surveys.