Measuring And Improving The Effectiveness Of Question Choices In Medical Assessments
Main Article Content
Abstract
The purpose of this study was to examine the relationships between item difficulty, ability of the items to distinguish certain performance levels and the strength of the distractors in medical MCQs. The aim was to decide if selected questions needed to be kept, updated or disposed of. In addition, the research focused on deciding how many options to include per multiple-choice question for better quality and reliability. An examination of the data was done using a cross-sectional approach in the pediatric section of a teaching hospital. Post hoc evaluation of the study period showed 800 MCQs and 4,000 connecting distractors were used. The analyzed factors for each exam item were difficulty index, discrimination index and distractor performance. Most questions had a difficulty level between 36.70% and 73.14%, with discrimination averaging between 0.20 and 0.34. Distractor efficiency was above 66.50% for most cases. About 48.4% of the examined items contained no unwanted extras, 35.3% had one NFD, 11.4% had two, 3.9% had three and 1.1% had four. When the number of answer options was three or four, NFDs were found less often than when the number was five. The stronger the discrimination and the more difficulty an item required, the better the efficiency of its distractors became (P < 0.005). Using the Kuder-Richardson 20 calculation, the mean reliability was 0.76. Indeed, many MCQs were acceptable, but a small number required changes or should be replaced. If you have three or four answer options, there are less irrelevant choices and the reliability of the test rises.
Article Details
References
Case SM, Swanson DB. Extended-matching items: A practical alternative to free-response questions. Tech Learn Med. 1993;5:107–15.
Cronbach LJ, Shavelson RJ. My current thoughts on coefficient alpha and successor procedures. Educ Psycol Meas. 2004;64:391–418.
Bloom BS, Hastings JT, Madaus GF. Handbook on Formative and Summative Evaluation of Student Learning. New York, USA: McGraw-Hill; 1971. p. 103.
Skakun EN, Nanson EM, Kling S, Taylor WC. A preliminary investigation of three types of multiple choice questions. Med Educ. 1979;13:91–6.
Skakun EN, Nanson EM, Taylor WC, Kling S. An investigation of three types of multiple choice questions. Annu Conf Res Med Educ. 1977;16:111–16.
Hingorjo MR, Jaleel F. Analysis of one-best MCQs: The difficulty index, discrimination index and distractor efficiency. J Pak Med Assoc. 2012;62:142–7.
Tarrant M, Ware J, Mohammed AM. An assessment of functioning and non-functioning distractors in multiple-choice questions: A descriptive analysis. BMC Med Educ. 2009;9:40.
Kelley TL. The selection of upper and lower groups for validation of test items. J Educ Psychol. 1939;30:17–24.
Mehta G, Mokhasi V. Item analysis of multiple choice questions: An assessment of the assessment tool. Int J Health Sci Res. 2014;4:197–202.
Bland JM, Altman DG. Cronbach’s alpha. BMJ. 1997;314:572. doi: 10.1136/bmj.314.7080.572.
Nunnally JC, Bernstein IH. Psychometric Theory. 3rd ed. New York, USA: McGraw-Hill; 1994.
Mitra NK, Nagaraja HS, Ponnudurai G, Judson JP. The levels of difficulty and discrimination indices in type A multiple choice questions of pre-clinical semester 1 multidisciplinary summative tests. Int EJ Sci Med Educ. 2009;3:2–7.
Gajjar S, Sharma R, Kumar P, Rana M. Item and test analysis to identify quality multiple choice questions (MCQs) from an assessment of medical students of Ahemdabad, Gujarat. Indian J Community Med. 2014;39:17–20.
Karelia BN, Pillai A, Vegada BN. The levels of difficulty and discrimination indices and relationship between them in four-response type multiple choice questions of pharmacology summative tests of year II M.B.B.S students. Int EJ Sci Med Educ. 2013;7:41–6.
Sharif M, Rahimi SM, Rajabi M, Sayyah M. Computer software application in item analysis of exams in a college of medicine. ARPN J Sci Tech. 2014;4:565–9.
Lin LC, Tseng HM, Wu SC. Item analysis of the registered nurse license exam by nursing candidates from vocational nursing high schools in Taiwan. Proc Natl Sci CouncRepub China D. 1999;9:24–31.
Kaur M, Singla S, Mahajan R. Item analysis of in use multiple choice questions in pharmacology. Int J Appl Basic Med Res. 2016;6:170–3.
Mackenzie J. Vague and ambiguous questions on multiple-choice exercises: The case for. Educ Philos Theory. 1994;26:23–33. doi: 10.1111/j.1469-5812.1994.tb00198.x. [DOI] [Google Scholar]
Rodriguez MC. Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educ Meas Issues Pract. 2005;24:3–13. doi: 10.1111/j.1745-3992.2005.00006.x. [DOI] [Google Scholar]
Tomak L, Bek Y. Item analysis and evaluation in the examinations in the Faculty of Medicine at OndokuzMayis University. Niger J Clin Pract. 2015;18:387–94. doi: 10.4103/1119-3077.151720. [DOI] [PubMed] [Google Scholar]
Mukherjee P, Lahiri SK. Analysis of multiple choice questions (MCQs): Item and test statistics from an assessment in a medical college of Kolkata, West Bengal. IOSR J Dent Med Sci. 2015;14:47–52. doi: 10.9790/0853-141264752. [DOI] [Google Scholar]
Nwadinigwe PI, Naibi L. The number of options in a multiple-choice test item and the psychometric characteristics. J Educ Pract. 2013;4:189–96. [Google Scholar]
Vegada B, Shukla A, Khilnani A, Charan J, Desai C. Comparison between three option, four option and five option multiple choice question tests for quality parameters: A randomized study. Indian J Pharmacol. 2016;48:571–5. doi: 10.4103/0253-7613.190757. [DOI] [PMC free article] [PubMed] [Google Scholar]
Tarrant M, Ware J. A comparison of the psychometric properties of three- and four-option multiple-choice questions in nursing assessments. Nurse Educ Today. 2010;30:539–43. doi: 10.1016/j.nedt.2009.11.002. [DOI] [PubMed] [Google Scholar]
Sim SM, Rasiah RI. Relationship between item difficulty and discrimination indices in true/false-type multiple choice questions of a para-clinical multidisciplinary paper. Ann Acad Med Singapore. 2006;35:67–71. [PubMed] [Google Scholar]