On the search sensitivity of a meta-analysis of AI-generated MCQs

Creative Commons License

Kıyak Y. S., Kaya A. B., Emekli E.

MEDICAL TEACHER, cilt.2026, ss.1, 2026 (SCI-Expanded, Scopus)

  • Yayın Türü: Makale / Editöre Mektup
  • Cilt numarası: 2026
  • Basım Tarihi: 2026
  • Doi Numarası: 10.1080/0142159x.2026.2708991
  • Dergi Adı: MEDICAL TEACHER
  • Derginin Tarandığı İndeksler: Academic Search Ultimate (EBSCO), Social Science Premium Collection (ProQuest), Biomedical Reference Collection: Corporate Edition (EBSCO), Education Collection (ProQuest), Education Source Ultimate (EBSCO), Health Research Premium Collection (ProQuest), Scopus, Sociology Database (ProQuest), Agricultural & Environmental Science Database, Science Citation Index Expanded (SCI-EXPANDED), CINAHL, Educational research abstracts (ERA), EMBASE, EBSCO Education Source, MEDLINE, Public Affairs Index
  • Sayfa Sayıları: ss.1
  • Açık Arşiv Koleksiyonu: AVESİS Açık Erişim Koleksiyonu
  • Eskişehir Osmangazi Üniversitesi Adresli: Evet

Özet

Dear Editor

We read with interest Alani et al.’s article [Citation1]. This timely synthesis usefully quantifies an important question, but its conclusions require caution.

Alani et al. searched PubMed, Google Scholar, Web of Science and the Cochrane Library, identifying 71 records and including 12 studies. In our contemporaneous review, we searched PubMed, Web of Science, Scopus and ERIC, identified 2347 records and included 71 studies [Citation2]. While differences in scope may explain variation in included studies, a total yield of 71 records appears unexpectedly low. Re-running the published PubMed strategy retrieves more records, suggesting that additional limits or field restrictions may need clarification. Greater transparency in database-specific search reporting would improve reproducibility.

This is consequential because several direct AI-versus-human comparisons with exam-based outcomes (e.g. difficulty and discrimination) appear absent from the review; for specific examples, see supplementary materials in our review [Citation2]. Not all would necessarily be poolable; some may require transformation, author contact, narrative synthesis or transparent exclusion. However, a systematic review should allow readers to distinguish studies excluded for specific reasons.

We suggest a more cautious interpretation: among the limited number of included studies, no statistically significant pooled difference was detected for difficulty or discrimination. Future reviews would be strengthened by explicit validity frameworks and fully reproducible search reporting.