An Interpretable Approach to Identify Performance Indicators Within Unstructured Learning Environments

As educational technology becomes increasingly integrated into curricula, more students are engaging with online learning platforms, interactive simulations, and MOOCs. These unstructured environments record students’ behaviors, providing a rich source of data on their learning processes. Researchers can model this data to gain insights into which behaviors and interactions most significantly affect students’ performance. However, most current methods for interpreting these models rely on manually designed features, which may not generalize across different scenarios or research questions. Conversely, post-hoc explainability methods for more complex models exist, but they lack consensus. To overcome these challenges, this study introduces the Transformer-based Identification of Performance Indicators (TIPI), a novel approach for identifying student behavior patterns that influence performance, emphasizing interpretability by design. TIPI employs a transformer-based deep learning model to convert students’ click-stream data into action tokens, periodically predicts students’ overall learning outcomes, and identifies the actions most indicative of performance. We conducted a comprehensive analysis using data from two versions of an interactive simulation for trade-school students to demonstrate TIPI’s effectiveness. The results reveal that the actions TIPI identifies as performance indicators align with subject matter experts’ insights. Furthermore, actions deemed similar by TIPI correspond to their context and location within the interactive simulation. Looking forward, TIPI has the potential to enhance personalized learning experiences in online environments by guiding students towards behaviors that most positively impact their predicted performance.