Indonesian–Arabic Academic Translation Quality: A Comparative Content Analysis of ChatGPT and Google Translate
DOI:
https://doi.org/10.15575/jta.v9i2.51510Keywords:
Arabic language, ChatGPT, Google Translate, Indonesian-Arabic Translation, Machine translation, Translation quality evaluationAbstract
This study compares the quality of Indonesian–Arabic academic translations produced by Google Translate and ChatGPT (GPT-4), a topic rarely examined despite widespread MT use in Arabic Language Education programs. Using a qualitative descriptive design with content analysis, the data comprised ten purposively selected undergraduate thesis titles from Indonesian university repositories, including UIN Sunan Kalijaga, UIN Suska Riau, UIN Imam Bonjol Padang, and Universitas Muhammadiyah Makassar. Titles were selected for their academic rigor and need for conceptual precision and formal Arabic register. Translations were analyzed at the phrase level using a Hybrid MQM–Nababan–Baker rubric encompassing seven dimensions: accuracy, acceptability, readability, lexical equivalence, grammatical equivalence, cohesion and coherence, and academic fluency, each rated on a three-point scale. Validity was ensured through alignment with Arabic translation theory, equivalence frameworks, and MQM standards, while iterative consistency checks supported reliability. Results reveal clear differences. ChatGPT achieved an average score of 20.0 out of 21 points (93%), which is classified as Very Good and indicates strong suitability for academic publication. Google Translate scored 13.3 out of 21 points (63%), classified as Good, but requiring post-editing. ChatGPT excelled in contextual meaning, syntactic restructuring, accurate idhafah, case governance, consistent terminology, and scholarly style, whereas Google Translate showed literal transfer. This study enriches AI-assisted translation discourse by grounding evaluation in Arabic translation theory and pedagogy, emphasizing generative AI’s pedagogical potential as a complementary tool while reaffirming the indispensable role of human expertise in maintaining linguistic accuracy, rigor, and academic standards.
References
Abidah, N. K. K., Hikmawati, R., & Erawanto, V. (2024). ChatGPT in Indonesia-Arabic Translation: A Quality Analysis. Al-Arabi: Journal of teaching Arabic As A Foreign Language, 8(1), 33–52. http://dx.doi.org/10.17977/um056v8i1p33-52
Aeni, A., Baharuddin, B., Putera, L. J., & Melani, B. Z. (2024). The Accuracy of Chatgpt in Translating Linguistics Text in. Didaktik: Jurnal Ilmiah PGSD STKIP Subang, 10(1), 59–68. https://doi.org/10.36989/didaktik.v10i1.2559
Aghai, M. (2024). ChatGPT vs. Google Translate: Comparative Analysis of Translation Quality. Iranian Journal of Translation Studies, 22(85), 85–100. https://dorl.net/dor/20.1001.1.17350212.1403.22.1.9.2
Ahmed, O. F., Dewi, I. K., & Anis, M. Y. (2025). Translating Arabic Literary Metaphors: A Comparative Evaluation of ChatGPT-4.0 and Google Translate 2024. Ideas: Journal on English Language Teaching and Learning, Linguistics and Literature, 13(2), 3177–3199. https://doi.org/10.24256/ideas.v13i2.7439
Al-Darabee, M. A., Farghal, M., & Haider, A. S. (2025). Netflix Versus Google Translate: A Case Study of the English-Arabic Translation of Scatological Terms. Studies in Linguistics, Culture and FLT, 13(2), 77–95. https://doi.org/10.46687/BCAY5747
Alayba, A. M. (2025). Arabic Natural Language Processing (NLP): A Comprehensive Review of Challenges, Techniques, and Emerging Trends. Computers 14.11, 14(11), 1-32. https://doi.org/10.3390/computers14110497
Alenezi, A. M. (2024). Error Analysis of Neural Machine Translation in Technical Texts: Google Translate as A Case Study. Journal of the North for Humanities. 8(2), 167–181. https://doi.org/10.12816/0061799
Aljanah, D. S., Koderi, K., & Akmansyah, M. (2025). The Mind-Machine Duality in Carving Meaning: ChatGPT and Self Regulated Learning in Arabic Research. Al-irfan: Journal of Arabic Literature and Islamic Studies. 8(1), 428–448. https://doi.org/10.58223/al-irfan.v8i1.470
Alzain, E., Nagi, K. A., & Algobaei, F. (2024). The Quality of Google Translate and ChatGPT English to Arabic Translation: The Case of Scientific Text Translation. Forum for Linguistic Studies. 6(3), 837–849. https://doi.org/10.30564/fls.v6i3.6799
Ameur, M. S. H., Meziane, F., & Guessoum, A. (2020). Arabic Machine Translation: A Survey of The Latest Trends and Challenges. Computer Science Review, 38(100305), 1-29. https://doi.org/10.1016/j.cosrev.2020.100305
Arifianto, M. L., Kholisin, K., Ridwan, N. A., Izzudin, I. F., Mujahidah, Z. A., & Muhtar, S. N. (2022). Designing Research Result-Based Arabic Pragmatic Learning Module On Politeness Principles And Strategies. Ta’lim Al-’Arabiyyah: Jurnal Pendidikan Bahasa Arab & Kebahasaaraban, 6(2), 134–154. https://doi.org/10.15575/jpba.v6i2.20235
Ataman, D., Birch, A., Habash, N., Federico, M., & Koehn, P. (2025). Machine Translation in the Era of Large Language Models: A Survey of Historical and Emerging Problems. Information 16.9, 16(9), 1–36. https://doi.org/10.3390/info16090723
Brahmana, C. R. P. S., Sofyan, R., & Mono, U. (2025). Evaluating The Strengths and Limitations of Chat Gpt-Generated Translations in Academic Post-Editing Workflows : A Case Study of English-Indonesian Academic Texts. Language Literacy: Journal of Linguistics, Literature, and Language Teaching, 9(1), 135–151. https://doi.org/10.30743/ll.v9i1.10551
Chandra, R., Chaudhari, A., & Rayavarapu, Y. (2025). An Evaluation of LLMs and Google Translate for Translation of Selected Indian Languages via Sentiment and Semantic Analyses. IEEE Access, 13, 122386–122407. https://doi.org/10.1109/ACCESS.2025.3585629
El-Farahaty, H. (2025). Integrating Corpora and AI Tools in the Teaching, Learning, and Assessing of Arabic/English Legal Translation. International Journal for the Semiotics of Law, 38(6), 2031–2060. https://doi.org/10.1007/s11196-025-10281-0
Faris, A., & Abdurrahman, M. (2023). Analisis Penerjemahan Bahasa Arab ke Bahasa Indonesia Melalui Artificial Intelligence ChatGPT. Al-Af’idah: Jurnal Pendidikan Bahasa Arab dan Pengajarannya, 7(2), 222–233. https://doi.org/10.52266/al-afidah.v7i2.1967
Guba, M. N. A., & Quba'a, A. A. (2025). AI Translation : Evaluating ChatGPT ’s Reliability in Translating Arabic to English. Jait: Journal of Artificial Intelligence and Technology, 5, 551-556. https://doi.org/10.37965/jait.2025.0831
Gupta, D., Malik, S., & Rana, A. (2022). Adopting Semantic Interoperability for Improved Healthcare. Proceedings of the International Conference on Innovative Computing & Communication (ICICC). 1-11. http://dx.doi.org/10.2139/ssrn.4096399
Hakiki, A. S., Sulthoni, S., & Devi, N. (2023). Analisis Hasil Terjemah Google Translate dan ChatGPT Bahasa Arab-Indonesia: Study Komparatif. Lanedunc: Journal Of Arabic, English, and Indonesian Language Teaching, Linguistic, and Literature, 1(1), 23–41. https://ejournal.aksaracendikia.com/index.php/Lanedunc/article/view/3
Jauhar, A. F. A., & Setiyawan, A. (2025). Comparative Analysis of Arabic Translation Results Between ChatGPT and Deepl. Tadris Al-‘Arabiyyah: Jurnal Pendidikan Bahasa Arab dan Kebahasaaraban, 4(2), 287–302. https://doi.org/10.15575/ta.v4i2.46575
Jooste, W., Haque, R., & Way, A. (2021). Philipp Koehn: Neural Machine Translation. Machine Translation, 35(2), 289–299. https://doi.org/10.1007/s10590-021-09277-x
Kusuma, E. D., & Yulia, F. (2023). Analysis of Arabic-Indonesian Translation Errors on Google Translate. Scaffolding: Jurnal Pendidikan Islam dan Multikulturalisme, 5(2), 1–13. https://doi.org/10.37680/scaffolding.v5i1.2722
Kusumadewi, J. S., Anis, M. Y., & Mohammed, C. A. (2024). Translation Analysis of Negative Sentences in the Zakat Chapter of the Book Al-Fiqh Al-Muyassar. Ta’lim Al-’Arabiyyah: Jurnal Pendidikan Bahasa Arab & Kebahasaaraban, 8(1), 34–55. https://doi.org/10.15575/jpba.v8i1.33848
Lommel, A., Uszkoreit, H., & Burchardt, A. (2014). Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics. Tradumatica Tecnologies de la Traduccio, 12, 455–463. https://doi.org/10.5565/rev/tradumatica.77
Mohsen, M. (2024). Artificial Intelligence in Academic Translation: A Comparative Study of Large Language Models and Google Translate. Psycholinguistics, 35(2), 134–156. https://doi.org/10.31470/2309-1797-2024-35-2-134-156
Nagoudi, E. B., Elmadany, A. R., & Mageed, M. A. (2022). Turjuman: A Public Toolkit for Neural Arabic Machine Translation. Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (Osact5), 1–11. https://doi.org/10.48550/arXiv.2206.03933
Nababan, M., Nuraeni, A., & Sumardiono, S. (2012). Pengembangan Model Penilaian Kualiats Terjemahan. Kajian Linguistik dan Sastra, 24(1), 39–57. Retrieved from https://journals.ums.ac.id/index.php/KLS/article/view/101
Nasaruddin, N. (2024). Using ChatGPT in Teaching Arabic as a Foreign Language. Arabiyatuna: Jurnal Bahasa Arab, 8(1), 1-24. https://doi.org/10.29240/jba.v8i1.9413
Nurfaiza, N. (2024). Pengaruh Penggunaan ChatGpt dalam Pembelajaran Terjemah Indonesia-Arab. Ibtidaiyah: Jurnal Pendidikan Guru Madrasah Ibtidaiyah, 2(2), 70-89. https://doi.org/10.52491/ibtidaiyah.v2i2.228
Popovic, M. (2020). Informative Manual Evaluation of Machine Translation Output. COLING 2020-28th International Conference on Computational Linguistics, Proceedings of the Conference, 5059–5069. https://doi.org/10.18653/v1/2020.coling-main.444
Sadikhova, S., & Babayev, J. (2025). Challenges Encountered in Translation of Culture-bound and Subject-specific Abstract Terminology While Using Google Translate. Euro Global Journal of Linguistics and Language Education, 2(3), 119–126. https://doi.org/10.69760/egjlle.2500203
Sadiq, S. (2025). Evaluating English-Arabic translation: Human translators vs Google Translate and ChatGPT Saudi Sadiq. Journal of Languages and Translation, 12(1), 67–95. https://doi.org/10.21608/jltmin.2025.423147
Skowron, D., & Baczkowska, A. (2023). Assessment of Various GPT Models Versus the Human Text: A Quantitative Analysis of Lexis and Cohesion. Forum Filologiczne Ateneum, 1(11), 139–156. https://doi.org/10.36575/2353-2912/1(11)2023-09
Suharsono, S., Ashadi, A., & Sudartinah, T. (2024). Comparative Study of Thematic Choice and Progression on Text Written by Humans and AI Machine. Joll: Journal of Language and Literature, 24(1), 153-174. https://doi.org/10.24071/joll.v24i1.6538
Syafanah, D. N., Permatasari, I., & Rivyal, M. (2025). Google Translate dan Penerjemahan: Penggunaan Google Translate pada Penerjemahan Teks Bahasa Arab. Blaze: Jurnal Bahasa dan Sastra dalam Pendidikan Linguistik dan Pengembangan, 3(3), 255–267. https://doi.org/10.59841/blaze.v3i3.2902
Syam, M. N., Isnaini, R. L., Rohmah, L., & Sa'adah, S. N. (2023). The Analysis of Google Translate Translation Error From Indonesian to Arabic and Tips for Using It. Ijaz Arabi: Journal of Arabic Learning, 6(1), 247–257. https://doi.org/10.18860/ijazarabi.v6i1.16299
Wahyudi, D., & Gina, A. R. (2023). Analysis of Arabic Textbooks by H. Darsono and T. Ibrahim Based on Mackey’s Theory. Ta’lim Al-’Arabiyyah: Jurnal Pendidikan Bahasa Arab & Kebahasaaraban, 7(1), 110–127. https://doi.org/10.15575/jpba.v7i1.24847
Wang, L., Lyu, C., Ji, T., Zhang, Z., Yu, D., Shi, S., & Tu, Z. (2023). Document-Level Machine Translation with Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 16646–16661. https://doi.org/10.18653/v1/2023.emnlp-main.1036
Widowati, D., Mubarak, M. Z., Saehudin, A., Abshar, U., & Suparno, D. (2025). Enhancing the Translation Quality of ChatGPT in Educational News: A Technique-Based Study on Adengad.Net. Langkawi: Journal of the Association for Arabic and English, 11(1), 190–204. https://doi.org/10.31332/lkw.v11i1.10684
Youssef, H., & Ismail, S. (2023). Cohesion and Coherence in Essays Generated by ChatGPT : A Comparative Analysis to University Students’ Writing ChatGPT. CDELT Occasional Papers in the Development of English Education, 83(1), 143–165. https://doi.org/10.21608/OPDE.2023.325331
Downloads
Additional Files
Published
How to Cite
Issue
Section
Citation Check
License
Copyright (c) 2025 Alfitri Alfitri, Gamal Abdul Nasir Zakaria, Misran Misran, Talqis Nurdianto
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish in Ta'lim al-'Arabiyyah: Jurnal Pendidikan Bahasa Arab dan Kebahasaaraban agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
Ta'lim al-'Arabiyyah: Jurnal Pendidikan Bahasa Arab dan Kebahasaaraban is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Based on a work at https://journal.uinsgd.ac.id/index.php/Talim
