Multi-Modal Learning Analytics for Adaptive Instruction Using Behavioral, Cognitive, and Affective Signals

This paper investigates an adaptive instruction framework driven by multi-modal learning analytics that integrates behavioral logs, cognitive proxies, and opportunistic affective signals under realistic missingness. The study analyzes 286 learners across an 8-week course, comprising 1,942 learning sessions windowed at 10-second resolution. Modality retention reflected deployment constraints, with behavioral coverage at 100%, cognitive coverage at 88%, and affective coverage at 73%, alongside reliability indices of 0.92, 0.84, and 0.71, respectively. Predictive modeling results show that reliability-aware multi-modal fusion outperformed behavioral-only baselines for mastery and next-step correctness, improving AUC from 0.812 to 0.872 and Macro-F1 from 0.741 to 0.793. Gains increased with content difficulty, with hard units improving from 0.794 to 0.887 AUC and from 0.712 to 0.801 Macro-F1. In instructional impact evaluation, adaptive sequencing raised end-of-course mastery rate from 0.70 to 0.78 (absolute +0.08, relative +11.4%) and increased near-transfer performance from 71.6 to 76.9 (+5.3 points), while total learning time rose modestly from 212.4 to 219.7 minutes (+3.4%). Efficiency outcomes improved across devices, reducing median time-to-mastery from 26.5 to 23.1 minutes on desktop and from 28.7 to 26.6 minutes on mobile, with no subgroup exhibiting degraded outcomes. Attention diagnostics indicated state-conditional modality reliance, with affective weighting rising most in frustration states (0.24), and policy behavior aligned with pedagogical intent through increased worked-example selection under confusion and restrained pacing interventions. Overall, the results demonstrate that tri-modal evidence can improve both inference and learning outcomes while remaining robust to missingness and device heterogeneity.