A Review of Cultural Literacy Within Course Outlines at an Australian University: An Innovative Use of Generative AI (ChatGPT) to Develop a Framework to Evaluate and Critique University Course Outlines
DOI:
https://doi.org/10.70232/jrep.v3i2.157Keywords:
Artificial Intelligence, ChatGPT, Course Outlines, Framework, Cultural Literacy, UniversityAbstract
This study explores the integration of artificial intelligence (AI), specifically large language models (LLMs) like ChatGPT, in evaluating cultural literacy within higher education (HE) curricula. Recognizing the increasing importance of cultural literacy in preparing globally competent graduates, the research investigates the extent to which university course outlines incorporate these competencies across disciplines. The primary objectives are to develop an effective framework for assessing cultural literacy levels in course content and to evaluate the utility of ChatGPT as an analytical tool in this process. Using a mixed-methods approach, the study first employed ChatGPT to generate a four-level classification framework for cultural literacy, which distinguishes between courses with no evidence, potential for embedding, implicit presence, and explicit integration of cultural literacy. Subsequently, a dataset of 584 publicly available course outlines from a university’s website was analyzed. ChatGPT applied the framework to categorize each course, with human coders independently validating a subset of the assessments to ensure reliability. Inter-rater reliability metrics confirmed high agreement between AI-generated and human evaluations, underscoring the robustness of the methodology. The findings reveal that many courses, especially in disciplines with high international student populations such as Technology, Engineering, and Health, lack explicit cultural literacy content. However, the framework demonstrated effectiveness in systematically assessing these competencies and offered actionable insights for curriculum development. The study concludes that AI-powered tools like ChatGPT present a promising, efficient means for continuous curriculum auditing and enhancement to foster inclusive and globally aware graduates. It also highlights the potential for broader application across institutions and disciplines, emphasizing the importance of transparent validation processes to ensure the credibility and reproducibility of AI-driven assessments in higher education.
References
Atlas, S. (2023). ChatGPT for higher education and professional development: A guide to conversational AI. The University of Rhode Island, College of Business Faculty Publications, 458. Retrieved from https://digitalcommons.uri.edu/cba_facpubs/548
Baidoo-Anu, D., & Ansah, L. O. (2023). Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Journal of AI, 7(1), 52-62. https://doi.org/10.61969/jai.1337500
Barrette, C. M., & Paesani, K. (2018). Conceptualizing cultural literacy through student learning outcomes assessment. Foreign Language Annals, 51(2), 331-343. https://doi.org/10.1111/flan.12337
Bennett, D., Knight, E., Dockery, A. M., & Bawad, S. (2020). Pedagogies for employability: Understanding the needs of STEM students through a new approach to employability development. Higher Education Pedagogies, 5(1), 340–359. https://doi.org/10.1080/23752696.2020.1847162
Caratozzolo, P., Cukierman, U., Nørgaard, B., Schrey-Niemenmaa, K., Azofeifa, J. D., & Rueda-Castro, V. (2024) Future skills forecasting: ensuring quality learning for every segment of the workfrce, 2024 IEEE Global Engineering Education Conference (EDUCON) (pp. 1-5), Kos Island, Greece, 2024. https://doi.org/10.1109/EDUCON60312.2024.10578916
De Wit, H. (2020). Internationalisation in higher education:: a western paradigm or a global, intentional and inclusive concept?. International Journal of African Higher Education, 7(2), 31-36. https://doi.org/10.6017/ijahe.v7i2.12891
Elias, A., & Mansouri, F. (2020). A systematic review of studies on interculturalism and intercultural dialogue. Journal of Intercultural Studies, 41(4), 490–523. https://doi.org/10.1080/07256868.2020.1782861
García Ochoa, G., & McDonald, S. (2019). Destabilisation and cultural literacy. Intercultural Education, 30(4), 351-367. https://doi.org/10.1080/14675986.2018.1540112
García Ochoa, G., McDonald, S. & Monk, N. (2016). Embedding cultural literacy in higher education: A new approach. Intercultural Education 27(6): 546–559. https://doi.org/10.1080/14675986.2016.1241551
Ge (Rochelle) (葛贇), Y. (2022). Internationalisation of higher education: new players in a changing scene. Educational Research and Evaluation, 27(3-4), 229-238. https://doi.org/10.1080/13803611.2022.2041850
Govender, N., Moheeput, K., Singh-Pillay, A., Cyparsade, M., James, A., Mabaso, B., Ngema, S., & Kolobe, L. (2025). Cultural decolonization and implications for pedagogy in integrating STEM-IKS education in the Global South. Cultural Studies of Science Education. https://doi.org/10.1007/s11422-025-10270-6
Guillén-Yparrea, N., & Ramírez-Montoya, M. S. (2023). A review of collaboration through intercultural competencies in higher education. Cogent Education, 10(1). https://doi.org/10.1080/2331186X.2023.2281845
Kelleher, J. (2002). Cultural literacy and health. Epidemiology 13(5), 497-500. Retrieved from https://journals.lww.com/epidem/fulltext/2002/09000/Cultural_Literacy_and_Health.2.aspx
Kern, R. (2015). Language, Literacy, and Technology. Cambridge University Press.
Khawaja, R. (2023, September 1). Fine-tuning LLMs 101. Retrieved from https://datasciencedojo.com/blog/fine-tuning-llms/
Kidman, G., & Tan, H. (2025). Disciplinary thinking in STEM: Balancing epistemologies for integrated education. In G. Kidman & H. Tan (Eds.), Knowledge generation in STEM and STEAM education (pp. 29–60). Springer. https://doi.org/10.1007/978-3-032-03247-8_2
Kucer, S. B. (2009). Dimensions of literacy: A conceptual base for teaching reading and writing in school settings (3rd ed.). Routledge.
Lee, M. (2024). Intercultural understanding: Implications for multicultural education. Multicultural Education Review, 16(2), 89–93. https://doi.org/10.1080/2005615X.2024.2376305
Marangell, S. & D’Orazzi, G. (2023). Students’ changing conceptualizations of university internationalization in Australia. Higher Education Research & Development, 42(5), pp. 1230-1246, https://doi.org/10.1080/07294360.2023.2193725
Meyer, J., & Land, R. (2005). Overcoming Barriers to Student Understanding. Taylor & Francis Limited.
Omodan, B. I. (2025). The roles of epistemology and decoloniality in addressing power dynamics in university education. Globalisation, Societies and Education, 23(5), 1226–1240. https://doi.org/10.1080/14767724.2024.2335661
Onal, S., & Kulavuz-Onal, D. (2023). A cross-disciplinary examination of the instructional uses of ChatGPT in higher education. Journal of Educational Technology Systems, 52(3) 301–324 https://doi.org/10.1177/00472395231196532
Paesani, K., Allen, H. W., & Dupuy, B. (2016). A multiliteracies framework for collegiate foreign language teaching. Pearson.
Pandya, B., Ruhi, U., & Patterson, L. (2023). Preparing the future workforce for 2030: the role of higher education institutions. Frontiers in Education, 8, 1295249. https://doi.org/10.3389/feduc.2023.1295249
Parker, L., Carter, C., Karakas, A., Loper, A. J., & Sokkar, A. (2024). Graduate instructors navigating the AI frontier: The role of ChatGPT in higher education. Computers and Education Open, 6, 100166. https://doi.org/10.1016/j.caeo.2024.100166
Qadir, J. (2022). Engineering education in the era of ChatGPT: Promise and pitfalls of generative AI for education. techrxiv.org. https://doi.org/10.36227/techrxiv.21789434.v1
Qadir, J., Islam, M. Q., & Al-Fuqaha, A. (2022). Toward accountable human-centered AI: Rationale and promising directions. Journal of Information, Communication and Ethics in Society, 20(2), 329–342. https://doi.org/10.1108/JICES-06-2021-0059
Romero-Rodríguez, J. M., Ramírez-Montoya, M. S., Buenestado-Fernández, M., & Lara-Lara, F. (2023). Use of ChatGPT at university as a tool for complex thinking: Students’ perceived usefulness. Journal of New Approaches in Educational Research, 12(2), 323-339. https://doi.org/10.7821/naer.2023.7.1458
Scrimshaw, S. C. (2019). Science, health, and cultural literacy in a rapidly changing communications landscape, Proceedings of the National Academy of Sciences, 116(16), 7650-7655. https://doi.org/10.1073/pnas.1807218116
Tran, L. T., Jung, J., Unangst, L., & Marshall, S. (2023). New developments in internationalisation of higher education. Higher Education Research & Development, 42(5), 1033-1041. https://doi.org/10.1080/07294360.2023.2216062
United Nations Department of Economics and Social Affairs. (2024). Goal 4: Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all. Retrieved from https://sdgs.un.org/goals/goal4
Universities Australia (2019). Higher education: facts and figures. Retrieved from https://universitiesaustralia.edu.au/wp-content/uploads/2019/08/190716-Facts-and-Figures-2019-Final-v2.pdf
UofA. (2023). 2024 pocket statistics. The University of Adelaide.
Van Mol, C., & Perez-Encinas, A. (2022). Inclusive internationalisation: Do different (social) groups of students need different internationalisation activities?. Studies in Higher Education, 47(12), 2523-2538. https://doi.org/10.1080/03075079.2022.2083102
Wu, H., Lee, D., Zerner, T., Court-Kowalski, S., Devitt, P., & Palmer, E. (2025). A comparison of the psychometric properties of GPT-4 versus human novice and expert authors of clinically complex MCQs in a mock examination of Australian medical students. Medical Teacher, 48(1), 74-84. https://doi.org/10.1080/0142159X.2025.2513418
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Daniel Lee, Tin Nguyen
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
