Can large language model-based AI Assistants support development of data science workflows through step-specific and alternative recommendations?

We conducted one of the first in-depth empirical studies on how data scientists interact with AI coding assistants, specifically GPT-4, across descriptive and predictive tasks. To support this research, I developed a model-agnostic Jupyter plugin (CATSci) that provides alternative code recommendations throughout the data science workflow. The study showed that including explicit workflow step information in prompts significantly improved the acceptance of recommendations, while offering alternative suggestions did not statistically increase task success but still helped users discover new methods. The study also uncovered notable differences in how recommendations were accepted and used between descriptive and predictive tasks, with descriptive tasks presenting unique challenges. Overall, participants expressed positive sentiments toward AI assistance, highlighting the potential of AI tools in data science and underscoring the need for better support for exploratory work and thoughtful interface design.

Related Publications

[1] Ramasamy et. al, 𝗔𝗜 𝘀𝘂𝗽𝗽𝗼𝗿𝘁 𝗳𝗼𝗿 𝗱𝗮𝘁𝗮 𝘀𝗰𝗶𝗲𝗻𝘁𝗶𝘀𝘁𝘀: 𝗔𝗻 𝗲𝗺𝗽𝗶𝗿𝗶𝗰𝗮𝗹 𝘀𝘁𝘂𝗱𝘆 𝗼𝗻 𝘄𝗼𝗿𝗸𝗳𝗹𝗼𝘄 𝗮𝗻𝗱 𝗮𝗹𝘁𝗲𝗿𝗻𝗮𝘁𝗶𝘃𝗲 𝗰𝗼𝗱𝗲 𝗿𝗲𝗰𝗼𝗺𝗺𝗲𝗻𝗱𝗮𝘁𝗶𝗼𝗻𝘀, Empirical Software Engineering Journal 30, Article number: 133 (2025). Read more in the blog.