Assessment of Cognitive Level and Item-Writing Flaws in Multiple-Choice Questions in High Stakes Nursing Examinations

2.50
Hdl Handle:
http://hdl.handle.net/10755/153307
Type:
Presentation
Title:
Assessment of Cognitive Level and Item-Writing Flaws in Multiple-Choice Questions in High Stakes Nursing Examinations
Abstract:
Assessment of Cognitive Level and Item-Writing Flaws in Multiple-Choice Questions in High Stakes Nursing Examinations
Conference Sponsor:Sigma Theta Tau International
Conference Year:2006
Author:Tarrant, Marie, RN, MPH, PhD
P.I. Institution Name:University of Hong Kong
Title:Assistant Professor
Co-Authors:Sasha Hayes, RN, MN; Aimee Knierim, RN, BSN; James Ware, MA, MB, FRCS, DMSc
Multiple Choice Questions (MCQs) are a frequently used assessment method in nursing examinations.  Few nurse educators, however, have formal preparation in constructing MCQs.  An unfortunate consequence is that many MCQ items on baccalaureate nursing tests and examinations have item writing flaws (IWFs).  All nursing assessments should have validity and a common threat to validity is the presence of IWFs.  An ongoing project shared by three Hong Kong nursing schools seeks to improve the assessments in high stakes nursing examinations. At one of the schools over 2800 MCQs have been collected, and are currently being classified and entered into an item test bank. At present, 824 of these items have been chosen randomly and further classified according to the cognitive level tested and the frequency of 19 accepted IWFs. Two cognitive levels were used for item classification: testing only simple recall of isolated fact (K1) and test of knowledge application, reasoning and problem solving (K2).  Item discrimination properties and difficulty index were also assessed for each question and compared across IWFs and cognitive level using chi-squares and ANOVAs.  Over 80% of the items were K1 and over 40% of questions contained IWFs. The frequency of IWFs among K1 items was 43.7% as compared to 12% for the K2 items (p<0.001). Common IWFs included the use of negatives or except in the question stem, the presence of logical clues, gratuitous information, ambiguous or unclear questions and failure to pass the hand-cover test. Questions containing IWFs were significantly more likely to have lower discrimination ability and more likely to negatively affect the difficulty index (p<0.05). Further analysis is currently underway to classify and examine the IWFs in all 2800 questions and to examine their impact upon question performance..
Repository Posting Date:
26-Oct-2011
Date of Publication:
17-Oct-2011
Sponsors:
Sigma Theta Tau International

Full metadata record

DC FieldValue Language
dc.typePresentationen_GB
dc.titleAssessment of Cognitive Level and Item-Writing Flaws in Multiple-Choice Questions in High Stakes Nursing Examinationsen_GB
dc.identifier.urihttp://hdl.handle.net/10755/153307-
dc.description.abstract<table><tr><td colspan="2" class="item-title">Assessment of Cognitive Level and Item-Writing Flaws in Multiple-Choice Questions in High Stakes Nursing Examinations</td></tr><tr class="item-sponsor"><td class="label">Conference Sponsor:</td><td class="value">Sigma Theta Tau International</td></tr><tr class="item-year"><td class="label">Conference Year:</td><td class="value">2006</td></tr><tr class="item-author"><td class="label">Author:</td><td class="value">Tarrant, Marie, RN, MPH, PhD</td></tr><tr class="item-institute"><td class="label">P.I. Institution Name:</td><td class="value">University of Hong Kong</td></tr><tr class="item-author-title"><td class="label">Title:</td><td class="value">Assistant Professor</td></tr><tr class="item-email"><td class="label">Email:</td><td class="value">tarrantm@hku.hk</td></tr><tr class="item-co-authors"><td class="label">Co-Authors:</td><td class="value">Sasha Hayes, RN, MN; Aimee Knierim, RN, BSN; James Ware, MA, MB, FRCS, DMSc</td></tr><tr><td colspan="2" class="item-abstract">Multiple Choice Questions (MCQs) are a frequently used assessment method in nursing examinations.&nbsp; Few nurse educators, however, have formal preparation in constructing MCQs.&nbsp; An unfortunate consequence is that many MCQ items on baccalaureate nursing tests and examinations have item writing flaws (IWFs).&nbsp; All nursing assessments should have validity and a common threat to validity is the presence of IWFs.&nbsp; An ongoing project shared by three Hong Kong nursing schools seeks to improve the assessments in high stakes nursing examinations. At one of the schools over 2800 MCQs have been collected, and are currently being classified and entered into an item test bank. At present, 824 of these items have been chosen randomly and further classified according to the cognitive level tested and the frequency of 19 accepted IWFs. Two cognitive levels were used for item classification: testing only simple recall of isolated fact (K1) and test of knowledge application, reasoning and problem solving (K2).&nbsp; Item discrimination properties and difficulty index were also assessed for each question and compared across IWFs and cognitive level using chi-squares and ANOVAs.&nbsp; Over 80% of the items were K1 and over 40% of questions contained IWFs. The frequency of IWFs among K1 items was 43.7% as compared to 12% for the K2 items (p&lt;0.001). Common IWFs included the use of negatives or except in the question stem, the presence of logical clues, gratuitous information, ambiguous or unclear questions and failure to pass the hand-cover test. Questions containing IWFs were significantly more likely to have lower discrimination ability and more likely to negatively affect the difficulty index (p&lt;0.05). Further analysis is currently underway to classify and examine the IWFs in all 2800 questions and to examine their impact upon question performance..</td></tr></table>en_GB
dc.date.available2011-10-26T12:11:08Z-
dc.date.issued2011-10-17en_GB
dc.date.accessioned2011-10-26T12:11:08Z-
dc.description.sponsorshipSigma Theta Tau Internationalen_GB
All Items in this repository are protected by copyright, with all rights reserved, unless otherwise indicated.