Validation of a ChatGPT Usage Instrument for English Language Learning Among Indonesian Senior High School Students Using the Rasch Model
DOI:
https://doi.org/10.61255/jupiter.v4i5.1872Keywords:
ChatGPT, English Language Learning, High School Students, Instrument Validation, Rasch ModelAbstract
Purpose: This study aimed to examine the psychometric properties of an adapted ChatGPT usage instrument for English language learning among Indonesian senior high school students using Rasch measurement and to evaluate the measurement characteristics of a proposed nine-item revision. Methods: A quantitative cross-sectional instrument-validation design was employed involving 585 senior high school students in Melawi Regency, West Kalimantan, Indonesia. The initial instrument consisted of ten items adapted from the ChatGPT Usage Scale for Foreign Language Learners, comprising three Usability items and seven Learning items. Instrument adaptation included translation, expert validation, and preliminary readability testing. Rasch analyses using Winsteps version 3.73 examined dimensionality, item fit, reliability, separation, rating-scale functioning, local dependence, item-person targeting, person fit, and gender-related Differential Item Functioning (DIF). Following the identification of Item 8 for further evaluation, an alternative nine-item version was re-estimated using the same dataset. Findings: Both instrument versions demonstrated a dominant Rasch measurement dimension, explaining 54.2% and 53.8% of the raw variance, respectively. Person reliability decreased slightly from 0.87 to 0.85 following the removal of Item 8, while item reliability remained unchanged at 0.97. The initial ten-item version identified Item 8 as exhibiting statistically significant but small gender-related DIF under the unadjusted criterion (p = .0323; DIF contrast = −0.26 logits). Re-analysis of the nine-item version identified a similarly small, nominally significant DIF finding for Item 1 (p = .0364; DIF contrast = −0.26 logits). Neither finding remained statistically significant under exploratory Bonferroni adjustments. The comparative analysis indicated that removing Item 8 produced relatively modest changes in overall measurement characteristics. Research Implications: The findings provide preliminary psychometric evidence supporting the proposed nine-item version as an alternative instrument specification. However, because both versions were evaluated using the same sample, independent cross-validation and further investigation of potentially problematic items are necessary before broader implementation. Originality: This study contributes context-specific psychometric evidence concerning ChatGPT usage measurement among Indonesian senior high school students by incorporating comprehensive Rasch diagnostics and comparing the initial ten-item instrument with a proposed nine-item revision.
Abstract views: 0
, 
PDF downloads: 1
Downloads
References
Agala, C. B., Fried, B. J., Thomas, J. C., Reynolds, H. W., Lich, K. H., Whetten, K., Zimmer, C., & Morrissey, J. P. (2020). Reliability, validity and measurement invariance of the Simplified Medication Adherence Questionnaire (SMAQ) among HIV-positive women in Ethiopia: A quasi-experimental study. BMC Public Health, 20(1), 567. https://doi.org/10.1186/s12889-020-08585-w
Aryadoust, V., Ng, L. Y., & Sayama, H. (2021). A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research. Language Testing, 38(1), 6–40. https://doi.org/10.1177/0265532220927487
Barrot, J. S. (2023). Using ChatGPT for second language writing: Pitfalls and potentials. Assessing Writing, 57, 100745. https://doi.org/10.1016/j.asw.2023.100745
Berrío, Á. I., Gómez-Benito, J., & Arias-Patiño, E. M. (2020). Developments and trends in research on methods of detecting differential item functioning. Educational Research Review, 31, 100340. https://doi.org/10.1016/j.edurev.2020.100340
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149
Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed). Lawrence Erlbaum Associates Publishers.
Çobanoğullari, F., & Özbek, Ö. (2025). AI-powered language learning: Developing the ChatGPT usage scale for foreign language learners. Education and Information Technologies, 30(9), 12517–12534. https://doi.org/10.1007/s10639-025-13342-w
Du, J., & Daniel, B. K. (2024). Transforming language education: A systematic review of AI-powered chatbots for English as a foreign language speaking practice. Computers and Education: Artificial Intelligence, 6, 100230. https://doi.org/10.1016/j.caeai.2024.100230
El-Hamamsy, L., Zapata-Cáceres, M., Martín-Barroso, E., Mondada, F., Zufferey, J. D., Bruno, B., & Román-González, M. (2025). The competent computational thinking test (cctt): A valid, reliable and gender-fair test for longitudinal CT studies in grades 3–6. Technology, Knowledge and Learning, 30(3), 1607–1661. https://doi.org/10.1007/s10758-024-09777-8
Ghafouri, M., Hassaskhah, J., & Mahdavi-Zafarghandi, A. (2024). From virtual assistant to writing mentor: Exploring the impact of a ChatGPT-based writing instruction protocol on EFL teachers’ self-efficacy and learners’ writing skill. Language Teaching Research, 13621688241239764. https://doi.org/10.1177/13621688241239764
Hapsari, I. P., & Wu, T.-T. (2022). AI Chatbots Learning Model in English Speaking Skill: Alleviating Speaking Anxiety, Boosting Enjoyment, and Fostering Critical Thinking. In Y.-M. Huang, S.-C. Cheng, J. Barroso, & F. E. Sandnes (Eds.), Innovative Technologies and Learning (pp. 444–453). Springer International Publishing. https://doi.org/10.1007/978-3-031-15273-3_49
Karataş, F., Abedi, F. Y., Ozek Gunyel, F., Karadeniz, D., & Kuzgun, Y. (2024). Incorporating AI in foreign language education: An investigation into ChatGPT’s effect on foreign language learners. Education and Information Technologies, 29, 19343–19366. https://doi.org/10.1007/s10639-024-12574-6
Khalaf, M. A., & Omara, E. M. N. (2022). Rasch analysis and differential item functioning of English language anxiety scale (ELAS) across sex in Egyptian context. BMC Psychology, 10, 242. https://doi.org/10.1186/s40359-022-00955-w
Kocak, O., Çoban, M., Aydın, A., & Çakmak, N. (2021). The mediating role of critical thinking and cooperativity in the 21st century skills of higher education students. Thinking Skills and Creativity, 42, 100967. https://doi.org/10.1016/j.tsc.2021.100967
Kohnke, L., Moorhouse, B. L., & Zou, D. (2023). ChatGPT for language teaching and learning. RELC Journal, 54(2), 537–550. https://doi.org/10.1177/00336882231162868
Liu, G., & Ma, C. (2024). Measuring EFL learners’ use of ChatGPT in informal digital learning of English based on the technology acceptance model. Innovation in Language Learning and Teaching, 18(2), 125–138. https://doi.org/10.1080/17501229.2023.2240316
Liu, Z. (2024). Second language writing anxiety and ChatGPT adoption as an automated writing evaluation tool. Journal of Applied Research in Higher Education. https://doi.org/10.1108/JARHE-06-2024-0260
Mahapatra, S. (2024). Impact of ChatGPT on ESL students’ academic writing skills: A mixed methods intervention study. Smart Learning Environments, 11, 9. https://doi.org/10.1186/s40561-024-00295-9
Marengo, A., Karaoglan-Yilmaz, F. G., Yılmaz, R., & Ceylan, M. (2025). Development and validation of generative artificial intelligence attitude scale for students. Frontiers in Computer Science, 7, 1528455. https://doi.org/10.3389/fcomp.2025.1528455
Meniado, J. C., Huyen, D. T. T., Panyadilokpong, N., & Lertkomolwit, P. (2024). Using ChatGPT for second language writing: Experiences and perceptions of EFL learners in Thailand and Vietnam. Computers and Education: Artificial Intelligence, 7, 100313. https://doi.org/10.1016/j.caeai.2024.100313
Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 100050. https://doi.org/10.1016/j.rmal.2023.100050
Orhan, A., Aydın Yıldız, T., & Çınar Yağcı, Ş. (2026). Assessing EFL learners’ attitudes on generative artificial intelligence: Development and validation of generative artificial intelligence attitude scale for EFL learners (GenAIAS). Journal of Research on Technology in Education, 58(3), 494–514. https://doi.org/10.1080/15391523.2024.2437744
Qian, P., & Xie, X. (2026). Task–technology fit leads to conflict: The double-edged-sword effect of generative artificial intelligence on scientific creative performance in humanities and social sciences research. International Journal of Human–Computer Interaction, 42(3), 1867–1891. https://doi.org/10.1080/10447318.2025.2526580
Qu, K., & Wu, X. (2024). ChatGPT as a CALL tool in language education: A study of hedonic motivation adoption models in English learning environments. Education and Information Technologies, 29(15), 19471–19503. https://doi.org/10.1007/s10639-024-12598-y
Salah, M., Abdelfattah, F., Al Halbusi, H., Jassem, S., Mohammed, M., Ismail, M. M., & Al Balghouni, A. (2025). Can generative AI craft scale items? A mixed-method study on AI’s capability to adapt and create new scales with recommendations for best practices. Social Sciences & Humanities Open, 12, 101698. https://doi.org/10.1016/j.ssaho.2025.101698
Slamet, J. (2024). Potential of ChatGPT as a digital language learning assistant: EFL teachers’ and students’ perceptions. Discover Artificial Intelligence, 4(1), 46. https://doi.org/10.1007/s44163-024-00143-2
Song, C., & Song, Y. (2023). Enhancing academic writing skills and motivation: Assessing the efficacy of ChatGPT in AI-assisted language learning for EFL students. Frontiers in Psychology, 14, 1260843. https://doi.org/10.3389/fpsyg.2023.1260843
Sumintono, B., & Widhiarso, W. (2015). Aplikasi pemodelan Rasch pada assessment pendidikan. Research Publications (2011 to 2015). https://knova.um.edu.my/research_publications_2011_2015/5938
Tram, N. H. M., Nguyen, T. T., & Tran, C. D. (2024). ChatGPT as a tool for self-learning English among EFL learners: A multi-methods study. System, 127, 103528. https://doi.org/10.1016/j.system.2024.103528
Wang, J. (2025). Textbook authors’ perspectives on selecting and instructing Chinese characters and words in beginner and intermediate textbooks. Language Teaching Research, 29(4), 1400–1425. https://doi.org/10.1177/13621688221096197
Wei, L. (2023). Artificial intelligence in language instruction: Impact on English learning achievement, L2 motivation, and self-regulated learning. Frontiers in Psychology, 14, 1261955. https://doi.org/10.3389/fpsyg.2023.1261955
Wind, S. A., & Peterson, M. E. (2018). A systematic review of methods for evaluating rating quality in language assessment. Language Testing, 35(2), 161–192. https://doi.org/10.1177/0265532216686999
Yan, D. (2023). Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation. Education and Information Technologies, 28(11), 13943–13967. https://doi.org/10.1007/s10639-023-11742-4
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Dwi Sudarti, Urai Salam, Venny Karolina, Aunurrahman, Yanti Sri Rezeki, Norhaslinda Hassan
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
