Questionnaire Design and Validation to Measure the Effectiveness of the Outcome-Based Teaching-Learning Process
DOI:
https://doi.org/10.70232/jesds.v3i1.54Keywords:
Active Learning, Collaborative Learning, Confirmatory Factor Analysis, Exploratory Factor Analysis, Higher-Order Thinking, Questionnaire Design, SatisfactionAbstract
In outcome-based education, the learners know what is expected of them after the course completion. The learners here are responsible for their learning process, which increases their active learning, self-learning, collaborative learning, higher-order skills, and, in turn, life-long learning skills. This paper is designed to develop a questionnaire that measures domain-dependent and domain-independent skills like self-learning, active and collaborative learning, higher-order thinking, and the learners’ satisfaction and continuous updating of the curriculum for the proposed outcome-based teaching-learning process. It also serves as a template for the steps needed to design and validate a questionnaire. During the design of the questionnaire, the content validity was established by five experts, and the CVI value of the items considered was ≥ 0.80. After designing the questionnaire, the proposed methodology was implemented and validated on the undergraduate BTech learners of “Seemanta Engineering College” as a pilot study. The preliminary and factor analysis results were applied to 220 learners, and the results were significant. The confirmatory results (RMSEA = 0.049, CFI = 0.963, TLI = 0.957, and SRMR = 0.036), considering 436 learners, also validate that the designed questionnaire is reliable and valid to measure the effectiveness of the teaching-learning process. This paper will guide the researcher step by step to design and validate the questionnaire, thus improving the quality of research.
References
Abrahim, S., Mir, B. A., Suhara, H., Mohamed, F. A., & Sato, M. (2019). Structural equation modeling and confirmatory factor analysis of social media use and education. International Journal of Educational Technology in Higher Education, 16, 1–25. https://doi.org/10.1186/s41239-019-0157-y
Aleandri, G., & Refrigeri, L. (2013). Lifelong learning, training and education in globalized economic systems : Analysis and Perspectives. Procedia - Social and Behavioral Sciences, 93, 1242–1248. https://doi.org/10.1016/j.sbspro.2013.10.022
Aljandali, A. (2016). Quantitative analysis and IBM® SPSS® statistics a guide for business and finance. Springer. https://doi.org/10.1007/978-3-319-45528-0
Almanasreh, E., Moles, R., & Chen, T. F. (2019). Evaluation of methods used for estimating content validity. Research in Social and Administrative Pharmacy, 15(2), 214–221. https://doi.org/10.1016/j.sapharm.2018.03.066
Artino, A. R., La Rochelle, J. S., Dezee, K. J., & Gehlbach, H. (2014). Developing questionnaires for educational research: AMEE Guide No. 87. Medical Teacher, 36(6), 463–474. https://doi.org/10.3109/0142159X.2014.889814
Ayyildiz, Y., & Tarhan, L. (2015). Development of the self-directed learning skills scale. International Journal of Lifelong Education, 34(6), 663–679. https://doi.org/10.1080/02601370.2015.1091393
Barari, N., RezaeiZadeh, M., Khorasani, A., & Alami, F. (2022). Designing and validating educational standards for E-teaching in virtual learning environments (VLEs), based on revised Bloom’s taxonomy. Interactive Learning Environments, 30(9), 1640–1652. https://doi.org/10.1080/10494820.2020.1739078
Camelia, F., & Ferris, T. L. J. (2018). Validation studies of a questionnaire developed to measure students’ engagement with systems thinking. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 48(4), 574–585. https://doi.org/10.1109/TSMC.2016.2607224
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://doi.org/10.1037/h0046016
Carini, R. M., Kuh, G. D., & Klein, S. P. (2006). Student engagement and student learning: Testing the linkages. Research in Higher Education, 47(1), 1–32. https://doi.org/10.1007/s11162-005-8150-9
Carr, R., Palmer, S., & Hagel, P. (2015). Active learning: The importance of developing a comprehensive measure. Active Learning in Higher Education, 16(3), 173–186. https://doi.org/10.1177/1469787415589529
Castaldi, P., & Mimmo, N. (2018). An experience of project based learning in aerospace engineering. IFAC-PapersOnLine, 52(12), 484-489. https://doi.org/10.1016/j.ifacol.2019.11.290
Cinici, A. (2016). Balancing the pros and cons of GMOs : Socio- scientific argumentation in pre-service teacher education. International Journal of Science Education, 38(11), 1841–1866. https://doi.org/10.1080/09500693.2016.1220033
Coleman, T. E., & Money, A. G. (2020). Student-centred digital game – based learning : a conceptual framework and survey of the state of the art. Higher Education, 79, 415–457. https://doi.org/doi.org/10.1007/s10734-019-00417-0
Dai, H., Wei, W., & Wang, H. (2017). Impact of outcome-based education on software engineering teaching : a case study. In 2017 IEEE Conference on Teaching, Assessment, and Learning for Engineering (pp. 261–264). IEEE. https://doi.org/10.1109/TALE.2017.8252344
Dassanayake, D. K. M. P. M. M., Wijesinghe, S. N., Jayasiri, T. L. C., Keenawinna, K. A. R. T., Rankothge, W. H., & Gamage, N. D. V. (2020). AwareME: Public awareness through game-based learning. IEEE Region 10 Annual International Conference, Proceedings/TENCON (pp. 662–666). IEEE. https://doi.org/10.1109/TENCON50793.2020.9293720
Divjak, B., Rienties, B., Iniesto, F., Vondra, P., & Žižak, M. (2022). Flipped classrooms in higher education during the COVID-19 pandemic: findings and future research recommendations. International Journal of Educational Technology in Higher Education, 19 (1), 1–24. https://doi.org/10.1186/s41239-021-00316-4
Douglas, J., Douglas, A., & Barnes, B. (2006). Measuring student satisfaction at a UK university. Quality Assurance in Education, 14(3), 251–267. https://doi.org/10.1108/09684880610678568
Drini, M. (2018). Using new methodologies in teaching computer programming. In 2018 IEEE Integrated STEM Education Conference (ISEC) (pp. 120–124). IEEE. https://doi.org/10.1109/ISECon.2018.8340461
Elangovan, N., & Sundaravel, E. (2021). Method of preparing a document for survey instrument validation by experts. MethodsX, 8, 1–9. https://doi.org/10.1016/j.mex.2021.101326
Fonseca, M. J., Costa, P., Lencastre, L., & Tavares, F. (2013). A statistical approach to quantitative data validation focused on the assessment of students’ perceptions about biotechnology. SpringerPlus, 2(1), 1–13. https://doi.org/10.1186/2193-1801-2-496
Geng, S., Law, K. M. Y., & Niu, B. (2019). Investigating self-directed learning and technology readiness in blending learning environment. International Journal of Educational Technology in Higher Education, 16(1), 1–22. https://doi.org/10.1186/s41239-019-0147-0
Goh, T. T., & Yang, B. (2021). The role of e-engagement and flow on the continuance with a learning management system in a blended learning environment. International Journal of Educational Technology in Higher Education, 18(1). https://doi.org/10.1186/s41239-021-00285-8
Gopal, R., Singh, V., & Aggarwal, A. (2021). Impact of online classes on the satisfaction and performance of students during the pandemic period of COVID 19. Education and Information Technologies, 26(6), 6923–6947. https://doi.org/10.1007/s10639-021-10523-1
Governor, D., Lombardi, D., & Duffield, C. (2021). Negotiations in scientific argumentation: An interpersonal analysis. Journal of Research in Science Teaching, 58(9), 1389–1424. https://doi.org/10.1002/tea.21713
Griffioen, D. M. E. (2022). A questionnaire to compare lecturers’ and students’ higher education research integration experiences. Teaching in Higher Education, 27(2), 185–200. https://doi.org/10.1080/13562517.2019.1706162
Hair, J. F., Black, W. C., Babin, B. J. & Anderson, R. E. (2014). Multivariate data analysis (7th ed). Pearson Education.
Hair, J. F., Joe Risher, J., Sarstedt, M., & Ringle, C. M. (2019). When to use and how to report the results of PLS-SEM. European Business Review, 31(1), 2–24. https://doi.org/10.1108/EBR-11-2018-0203
Haukås, Å., Storto, A., & Tiurikova, I. (2021). Developing and validating a questionnaire on young learners’ multilingualism and multilingual identity. Language Learning Journal, 49(4), 404–419. https://doi.org/10.1080/09571736.2021.1915367
Kazemi, S. S., Tavafian, S. S., Hidarnia, A., & Montazeri, A. (2020). Development and validation of an instrument of occupational low back pain prevention behaviours of nurse. Journal of Advanced Nursing, 76(10), 2747–2756. https://doi.org/10.1111/jan.14459
Kember, D., & Leung, D. Y. P. (2008). Establishing the validity and reliability of course evaluation questionnaires. Assessment and Evaluation in Higher Education, 33(4), 341–353. https://doi.org/10.1080/02602930701563070
Khanna, P., Singh, K., & Proctor, C. (2021). Exploring the impact of a character strengths intervention on well-being in Indian classrooms. School Mental Health, 2017, 1–14. https://doi.org/10.1007/s12310-021-09450-w
Kothari, C. R. (2004). Research methodology methods and techniques. New Age International Publishers.
Krouska, A., Troussas, C., & Sgouropoulou, C. (2022). Mobile game-based learning as a solution in COVID-19 era: Modeling the pedagogical affordance and student interactions. Education and Information Technologies, 27(1), 229–241. https://doi.org/10.1007/s10639-021-10672-3
Kulkarni, U. P., Kulkarni, S. B., Shindhe, K. C., & Joshi, A. N. (2016). A case study on an unrealized truth of an outcome based education and corrective strategies. In 2016 IEEE 4th International Conference on MOOCs, Innovation and Technology in Education (pp. 285–289). IEEE. https://doi.org/10.1109/MITE.2016.063
Li, Y., Liu, Y., & Shu, P. (2015). Teaching research and practice of blended leaning model based on computational thinking. In Proceedings - Frontiers in Education Conference (pp. 1–8). IEEE. https://doi.org/10.1109/FIE.2015.7344415
Lobczowski, N. G., Allen, E. M., Firetto, C. M., Greene, J. A., & Murphy, P. K. (2020). An exploration of social regulation of learning during scientific argumentation discourse. Contemporary Educational Psychology, 63, 1–17. https://doi.org/10.1016/j.cedpsych.2020.101925
Lu, K., Yang, H. H., Shi, Y., & Wang, X. (2021). Examining the key influencing factors on college students’ higher-order thinking skills in the smart classroom environment. International Journal of Educational Technology in Higher Education, 18(1), 1–13. https://doi.org/10.1186/s41239-020-00238-7
Mat Roni, S., Merga, M. K., & Morris, J. E. (2020). Conducting quantitative research in education. Springer Nature Singapore. https://doi.org/10.1007/978-981-13-9132-3
Mendonca, M., Papageorgiou, K., Mello, D. E. De, Papageorgiou, E. I., Fabri, J. A., Souza, L. B. De, & Palacios, R. H. C. (2020). Digital game-based learning in a robotics course. In 11th International Conference on Information, Intelligence, Systems and Applications (pp. 1–7). IEEE. https://doi.org/10.1109/IISA50023.2020.9284366
Mertens, W., Pugliese, A., & Recker, J. (2017). Quantitative data analysis: A companion for accounting and information systems research. Springer International Publishing. https://doi.org/10.1007/978-3-319-42700-3
Michalopoulou, C. (2017). Likert scales require validation before application - another cautionary tale. BMS Bulletin of Sociological Methodology/ Bulletin de Methodologie Sociologique, 134(1), 5–23. https://doi.org/10.1177/0759106317693786
Mohanta, J., & Mandal, S. K. Das. (2019). The effectiveness of the outcome-based curriculum towards improving educational quality for technical education. In Proceedings - IEEE 10th International Conference on Technology for Education (pp. 242–243) IEEE. https://doi.org/10.1109/T4E.2019.00-13
Napitupulu, D., Rahim, R., Abdullah, D., Setiawan, M. I., Abdillah, L. A., Ahmar, A. S., Simarmata, J., Hidayat, R., Nurdiyanto, H., & Pranolo, A. (2018). Analysis of student satisfaction toward quality of service facility. Journal of Physics: Conference Series, 954(1). https://doi.org/10.1088/1742-6596/954/1/012019
NSSE. (2017). Engagement indicators. Retrieved from https://nsse.indiana.edu/2017_Institutional_Report/pdf/NSSE17EngagementIndicators(NSSEvilleState).pdf
Oliveira, L., & Cardoso, E. L. (2021). A project-based learning approach to promote innovation and academic entrepreneurship in a master’s degree in food engineering. Journal of Food Science Education, 20(4), 120-129. https://doi.org/10.1111/1541-4329.12230
Rao, N. J. (2020). Outcome-based education: an outline. Higher Education for the Future, 7(1), 5–21. https://doi.org/10.1177/2347631119886418
Ray, A. K. (2013). Developing suitable pedagogical methods for various classes, intellectual calibers and research in e-learning. Retrieved from http://www.ide.iitkgp.ac.in/Pedagogy1/pedagogy_main.jsp
Silber-Varod, V., Eshet-Alkalai, Y., & Geri, N. (2019). Tracing research trends of 21st-century learning skills. British Journal of Educational Technology, 50(6), 3099–3118. https://doi.org/10.1111/bjet.12753
Singh, A., Rocke, S., Pooransingh, A., & Ramlal, C. J. (2019). Improving student engagement in teaching electric machines through blended learning. IEEE Transactions on Education, 62(4), 297-304. https://doi.org/10.1109/TE.2019.2918097
Singh, K., Junnarkar, M., & Kaur, J. (2016). Measures of positive psychology. Springer. https://doi.org/10.1007/978-81-322-3631-3
Stockemer, D. (2019). Quantitative methods for the social sciences. A practical introduction with examples in SPSS and Stata. Springer International Publishing.
Sukla, D., & Dungsungneon, A. P. (2016). Students perceived level and teachers teaching strategies of higher order thinking skills; A study on higher educational institutions in Thailand. Journal of Education and Practice, 7(12), 211–219.
Tekkol, I. A., & Demirel, M. (2018). An investigation of self-directed learning skills of undergraduate students. Frontiers in Psychology, 9, 1–14. https://doi.org/10.3389/fpsyg.2018.02324
Tomas, L., Evans, N. (Snowy), Doyle, T., & Skamp, K. (2019). Are first year students ready for a flipped classroom? A case for a flipped learning continuum. International Journal of Educational Technology in Higher Education, 16(1), 1–22. https://doi.org/10.1186/s41239-019-0135-4
Tsai, F. H., Hsiao, H. S., Yu, K. C., & Lin, K. Y. (2022). Development and effectiveness evaluation of a STEM-based game-design project for preservice primary teacher education. International Journal of Technology and Design Education, 32(5), 2403–2424. https://doi.org/10.1007/s10798-021-09702-5
UNESCO. (2018). Issues and trends in education for sustainable development issues and trends in education. UNESCO Publishing.
Wei, Y., Wang, J., Yang, H. H., Wang, X., & Cheng, J. (2019). An investigation of academic self-efficacy, intrinsic motivation and connected classroom climate on college students ‘ engagement in blended learning. In 2019 International Symposium on Educational Technology (pp. 160–164). IEEE. https://doi.org/10.1109/ISET.2019.00041
Wijnen, F., Walma van der Molen, J., & Voogt, J. (2021). Measuring primary school teachers’ attitudes towards stimulating higher-order thinking (SHOT) in students: Development and validation of the SHOT questionnaire. Thinking Skills and Creativity, 42, 100954. https://doi.org/10.1016/j.tsc.2021.100954
Yazzie-Mintz Ethan, M. K. (2012). Handbook of research on student engagement. Springer.
Yelamarthi, K., & Drake, E. (2015). A flipped first-year digital circuits course for engineering and technology students. IEEE Transactions on Education, 58(3), 179–186. https://doi.org/10.1109/TE.2014.2356174
Zarrinfard, S., Rahimi, M., & Mohseny, A. (2021). Flipping an on-campus general English course: a focus on technology complexity of instruction and learners’ levels of impulsivity. International Journal of Educational Technology in Higher Education, 18(1). https://doi.org/10.1186/s41239-021-00280-z
Zeynal, H., Zakaria, Z., Anisseh, M., & Mansoorzadeh, S. (2017). Strategic implementation of outcome-based education system in Buein-Zahra Technical University of Iran. In 2017 IEEE 9th International Conference on Engineering Education (pp. 122–127). IEEE. https://doi.org/10.1109/ICEED.2017.8251178
Zhao, D., Muntean, C. H., Chis, A. E., Rozinaj, G., & Muntean, G. M. (2022). Game-based learning: enhancing student experience, knowledge gain, and usability in higher education programming courses. IEEE Transactions on Education, 65(4), 1–12. https://doi.org/10.1109/TE.2021.3136914
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Jyotiprava Mohanta, Shyamal Kumar Das Mandal
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
