Code-Aware LLM Prompting in Deductive Qualitative Analysis: A Study in Multi-framework Analysis of Learning Designs

Large Language Models (LLMs) have demonstrated potential for (semi-)automating the qualitative analysis of unstructured data, particularly in deductive qualitative coding using codebooks. While prior research has shown the feasibility of this technology, model performance seems to vary depending on the nature of the constructs being coded. However, existing approaches typically apply a single LLM prompting strategy across entire datasets, often of discourse transcripts or questionnaire data, using a single coding scheme or theoretical frame. This paper introduces an adaptive prompting approach where LLM prompts are customized based on researcher-defined or data-driven rules for specific codes. We apply this approach to analyze the description of 35 multimedia learning designs (comprising 758 items/activities) created by teachers in an inquiry-based learning digital platform. Two human coders and an open-weights LLM (Llama 3.3) coded the dataset attending to three different pedagogical frameworks. Our results indicate that data-driven adaptive prompting outperforms uniform prompting approaches (namely, zero-shot, few-shot with/without context) in terms of agreement with human coders and cost. While the improvement is not statistically significant, the approach offers potential advantages for large datasets (especially, considering the costs), highlighting opportunities for human-AI collaboration in educational data analysis.