Classroom teaching with AI-based educational software involves complex tasks spanning across three phases: before class, during class, and after class. 1) Before class, teachers need to do adequate lesson planning and preparation. It is important to ensure that the instructional materials and technology to leverage are customized to the classroom context and aligned with the curriculum. 2) During class, teachers frequently improvise and constantly differentiate their actions to serve the needs of their different students. 3) After class ends, post-class reflection can help teachers to think critically about what happened, to flexibly adapt their instructions. It is important to provide teacher support in all phases. The advance of educational technology and multi-modal learning analytics (MMLA) (Cukurova et al., 2020; Escueta et al., 2017; Holstein et al., 2019b; Holstein & Aleven, 2022) holds promise to support teachers and students to achieve more effective teaching and learning, by empowering teachers to focus on instructional tasks that they do the best and that they enjoy doing. My PhD research aims to create synergy between humans and AI as we design, develop, and evaluate educational technology that fosters more customized, differentiated, reflective learning environments.

Customization of Instructional Materials and Technology. While there are many ways to customize learning environments, my work focuses on instructional materials and technology. Prior literature has noted the need and importance of aligning instructional materials and technology with teachers' practice and preferences to achieve sustained use and adoption.

Customization of instructional materials. For instructional materials, I studied how to achieve customized instructional materials (mathematical hints) while saving time for domain experts through a crowdsourcing pipeline. An important question left open by past work is how to support crowds with minimal expert guidance in revising and customizing instructional materials. This work found a clear need for customizing hints to individual teachers' preferences and highlights the need for more elaborate scaffolds so the crowd has specific knowledge of the teachers' requirements for hints.

Customization of technology. For technology, I studied how to ensure feasibility (in the sense of student coverage) for dynamic pairing policies (which support pairing up students in a personalized way), in a teacher orchestration tool. Prior work has provided little insight into the feasibility of pairing policies, especially in the ITS context. I studied how to leverage data simulation of historical transaction log data (SimPairing) to assess the feasibility of dynamic learning group formation policies in different classroom contexts. I found that dynamically pairing students based on their in-the-moment wheel-spinning status yields good feasibility for struggling students, even under moderate restrictions on allowed pairings. Importantly, any fixed instantiation of a policy does not fully account for class variability and extreme cases–policies perform differently across classrooms and sessions, suggesting that policy feasibility may be optimized by gradually loosening parameters or through classroom-levelcustomization. This study contributes to how different dynamic pairing policies have different feasibility results in different classrooms and bridges the literature gap in its investigation of the feasibility of user-centered dynamic pairing policies.

Dynamic Differentiation between Individual and Collaborative Learning. There are many ways to make learning environments support differentiated learning, and my work focuses on dynamically combining different learning modes for students. Prior research found that combining individual and collaborative learning can be more beneficial than either way alone. However, prior work has not created or studied a technology ecosystem to achieve dynamically differentiated learning between individual and collaborative learning and systematically evaluate the value of the dynamic combination of individual and collaborative learning. Grounded in the context of real-time, analytic-based intelligent tutoring systems, my colleagues and I created a technology ecosystem to achieve the dynamic combination of individual and collaborative learning. I then conducted a series of studies to understand teachers' preferences and boundaries in co-orchestrating with AI systems in such dynamic pairing, evaluated in an authentic classroom prototyping study, the feasibility of such dynamic combinations in classrooms, and conducted a controlled study to evaluate how dynamic transitions are compared with traditional transitions.

I discovered teachers' preferences about the respective roles of teachers, students, and AI in a co-orchestration system of dynamic pairing. I found that dynamic transitions between different activities have pedagogical value in the eyes of teachers and, to a lesser degree, students. Supporting dynamic transitions in actual classrooms is worthwhile from a pedagogical perspective, yet not without challenges, such as finding the optimal timing, content, and partner during such transitions. I found that teachers and students demonstrated a subjective preference for the dynamic condition, which stemmed from its flexibility and individualization. However, for some of the test items, the standard condition had more learning gains than the dynamic condition.

Multi-modal Based Teacher Reflection Tool. My work focuses on supporting teacher reflection through multimodal learning analytics (MMLA) in classrooms where students use ITS. Teacher reflection can be an important component in advancing the quality of instruction. Teacher reflection holds potential to scaffold teachers' critical thinking (Korthagen, 2004), support knowledge construction in teaching (Conway, 2001), and promote self-regulation in teachers (Boud & Falchikov, 2007; Singh, 2008). However, more work is needed to understand how a reflection tool can be designed to fit into teachers' routines and keep in mind their preferences for data sharing and privacy considerations. With the help of colleagues, I studied teachers' preferences about multi-modal data analytics collection and analytics use in reflection tools. I found teachers showed the strongest interest in data about motivating students and interactions with students (teacher analytics) and students' learning and progress (student analytics). I also learned about specific teachers’ data collection and sharing preferences in MMLA.

In my final study related to Reflecto, a tool that leveraged multi-modal data analytics (including location sensing and transaction log data) to support teacher reflection, I started with refining and pilot testing this teacher reflection tool, and a small-scale prototyping activity. Then I led an exploratory classroom study with actual teachers and students, which addressed the following research questions: 1) RQ1: How do teachers use the MMLA-based teacher reflection tool during individual and collaborative reflection sessions? 2) RQ2: How does Reflecto influence teachers' understanding of their teaching practices and future intentions for interacting with students? 3) RQ3: How does the use of an MMLA-based reflection tool impact teachers' actual teaching practices? 4) RQ4: What are teachers' perceptions of the MMLA-based reflection tool, and what design feedback do they offer?

My PhD research leverages various methodologies, including pipeline and tool design, data simulation and mining, user-centered research, as well as both prototyping study and controlled classroom experiments. My research focuses on aiding teachers in customizing instructional materials and technology (i.e., pairing algorithms) and facilitating dynamically-differentiated classrooms, as well as reflective instructional practices using Multimodal Learning Analytics (MMLA). This dissertation advances knowledge on customization of instructional materials, orchestration technology, and teacher reflection in AI-based learning environments. I demonstrate the promise and value of leveraging minimal expert guidance and a crowdworker pipeline to save teachers' time in instructional materials customization. My work also highlights the pedagogical value of dynamically combining individual and collaborative learning through a technology ecosystem, as well as the nuances and design insights related to human–AI shared control in this context. In addition, I show that an MMLA-based reflection tool (leveraging teacher location and student log data from ITSs) can help teachers validate existing understandings of classroom practice, challenge assumptions about their students and classrooms, and generate actionable insights.

211 pages

Thesis Committee:
Vincent Aleven (Co-Chair, CMU)
Nikol Rummel(Co-Chair, Ruhr University Bochum/CMU)
Kenneth Holstein
Xu Wang (University of Michigan)

Brad A. Myers, Head, Human-Computer Interaction Institute
Martial Hebert, Dean, School of Computer Science

Creative Commons: CC-BY (Attribution)

Return to: SCS Technical Report Collection
School of Computer Science homepage

This page maintained by reports@cs.cmu.edu