Self-Reported LLM Usage and Outcomes on a Data Science Project: Evidence from Two Undergraduate Courses

We find some mild evidence from this observational study that LLM usage may be associated with better scores, especially for students who do not natively speak English. Additionally, comparing self-reported usage, there was a considerable increase between the class that occurred in January-April 2024 where 41% of students self-reported extensive LLM usage and the class that occurred in September-December 2024 where 69% reported extensive LLM usage.

Despite the classroom setting used for evaluation, the task of interest is similar to the work done by professional data scientists. Our finding suggests the need for more extensive work evaluating how LLMs can be integrated into the data science workflow in a way that provides value in both the classroom and the workplace.</p>