Title: Using AI in Research Measuring Instructional Practices
Abstract:
A learning sciences research team at Digital Promise is examining how instructional practices vary across introductory statistics courses offered in blended modalities across broad-access 2- and 4-year colleges. This study aims to understand how instructor practice changes when digital courseware—specialized software that delivers content, practice, and assessment for an entire course—is introduced. It is part of a foundation-funded initiative designed to improve learning and course success for racially minoritized students and those from low-income backgrounds enrolled in college gateway courses through a combination of courseware and instructor supports.
Drawing on multiple data sources, including classroom observations, instructor and student surveys, interviews, and AI-generated classroom audio data, the research investigates the use of generative AI (GenAI) to systematically classify instructional approaches. Over the past two years, data from more than 90 introductory statistics classes across 28 colleges have been analyzed using AI algorithms that capture talk-time percentages and speaker-labeled transcripts to identify proportions of instructor and student speech.
Over the last two years, we have amassed data from more than 90 introductory statistics classes taught in 28 colleges through student surveys and interviews, instructor surveys and interviews, classroom observations, and AI-enhanced classroom audio data. We have employed both GenAI tools and traditional quantitative and qualitative techniques to identify contrasting instructional approaches.
The resulting classification of instructional approaches revealed distinct patterns that varied in courseware use, student engagement, and the likelihood that focal group students achieved a grade of C or higher. The findings highlight both the potential and current limitations of applying GenAI tools in large-scale educational research.
Bio:
Barbara Means is Senior Principal Learning Sciences Researcher at Digital Promise, where she studies the implementation and effectiveness of educational innovations supported by technology. Currently, she leads large-scale courseware implementation and efficacy studies in the domains of college statistics and chemistry. A fellow of the American Educational Research Association, Dr. Means has advised the U.S. Department of Education on national educational technology plans for both K-12 and postsecondary education. She has authored or edited a half dozen books related to learning and technology research. She also has served on many study panels related to science education for the National Academies of Sciences, Engineering and Medicine, including the panels that produced How People Learn I and How People Learn II.
