Improving Islamic Boarding School Teacher’ Competence in Designing AI-Based Assessment of Critical Thinking Skills

Abstract

This study reports a structured capacity-building program designed to improve Islamic boarding school teachers’ competence in designing AI-based assessments for critical thinking skills. The intervention was carried out with 91 teachers across 14 school subjects at a large pesantren —a traditional Islamic educational institution that integrates religious and national curricula—in East Java, Indonesia. The program integrated three components: (i) foundational understanding of generative AI, (ii) explicit mapping of higher-order thinking indicators (analysis, inference, evaluation), and (iii) co-construction of assessment tasks and analytic rubrics through guided prompt engineering. A one-group pretest–posttest design was used to evaluate teacher learning gains. Results demonstrate a very large improvement in teachers’ assessment design knowledge and skills, with mean scores increasing from 37.75 (pretest) to 90.49 (posttest), and a statistically significant t-value (t = 29.49, p < .01). Survey findings also indicate strong positive perceptions of AI usefulness, relevance, and feasibility. Teachers were able to generate authentic HOTS tasks contextualized to Islamic and local cultural values, accompanied by clear rubrics for evaluating evidence-based reasoning. The study concludes that AI-assisted design can meaningfully accelerate teacher capacity to design higher-order assessments in faith-based educational settings, provided that AI is used as a scaffold for human reasoning, not as an answer generator. Implications for scaling professional development and institutionalizing AI-embedded assessment innovation in pesantren contexts are discussed