Emekli E., Soylu R., Emekli E., Kıyak Y. S., Hosgören Alıcı Y., Coşkun Ö., ...Daha Fazla
Tıp Eğitimi Dünyası, cilt.24, sa.74, ss.209-216, 2025 (TRDizin)
Özet
Background: Multiple-choice questions (MCQs) are widely used in medical education due to their objectivity, efficiency, and ability to cover a broad knowledge base. Case-based MCQs provide additional benefits by evaluating students’ clinical reasoning and decision-making skills. In psychiatry education, unique challenges arise from overlapping symptoms, reliance on subjective reports, and the absence of objective diagnostic tools. The aim of this study was to administer MCQs generated through template-based automatic item generation (AIG) in psychiatry to medical students and to evaluate their psychometric properties (difficulty and discrimination indices).
Methods: Following ethical approval from XXX University Ethics Committee, the study included 138 volunteer students (61.6%) from a total of 224 who completed psychiatry clerkship during the 2023–2024 and 2024–2025 academic years. From a pool of 1189 template-based automatically generated questions, 22 were randomly selected to form the exam. The test was administered face-to-face under supervision, and students were not informed of the origin of the items. Difficulty indices were calculated as the proportion of correct answers, while discrimination indices were computed by comparing the performance of the top 27% and bottom 27% groups.
Results: The mean exam score was 15.21 ± 3.55 out of 22. The average difficulty index was 0.69, classifying the exam as “easy.” Of the items, 63.6% were very easy, 9.1% easy, and 27.3% moderate. The most difficult item concerned somatization (0.33), whereas the easiest was related to bipolar disorder (0.92). Discrimination indices ranged from 0.19 to 0.70, with an average of 0.37. Ten items (45.6%) demonstrated excellent discrimination, eleven (50%) acceptable, and one (4.5%) poor. The highest discrimination was observed in the schizophreniform disorder item (0.70), while the lowest was in the postpartum psychosis item (0.19).
Conclusions: This study represents the first direct implementation of template-based AIG in Turkish psychiatry education. The findings demonstrated that automatically generated MCQs achieved acceptable psychometric standards in terms of both difficulty and discrimination. Template-based AIG may reduce faculty workload while ensuring consistent and high-quality question development. However, further refinement is needed to generate items assessing higher-order cognitive processes. Multicenter comparative studies could provide stronger evidence for the integration of AIG into medical education assessments.