ABN MULTIPLE CHOICE EXAMINATION DEVELOPMENT

Questions Item Selection:
The ABN written examination is intended to assess general knowledge in clinical neuropsychology.  Previously ABN had utilized an essay written examination but with changes in certification of health care professionals it was decided to develop a multiple choice format examination to replace the essay examination. Work on the ABN multiple choice test conceptually began in 2003 and data collection started in 2004.  Test questions were submitted in multiple-choice format by board certified neuropsychologists who were also board examiners.  The authors of specific multiple choice questions provided citations supporting their questions and the citations were checked to assure accuracy.  A subgroup of five board certified examiners were also trained in question writing based on Measurement and Assessment in Education (Reynolds, Livingston, and Willson, 2008).  A body of questions was examined, and there was discussion, debate, re-wording, and clarification of the questions submitted based on the expert judgment of the reviewers.  Some multiple choice questions were dropped as being inappropriate to the purpose of the assessment.

Initial Standardization of Question Pool:
The set of multiple choice questions was then administered to a group of 30 ABN board certified neuropsychologists.  Multiple choice questions that failed to discriminate among levels of neuropsychology knowledge were eliminated. For example, if all of the ABN board certified neuropsychologists answered a given question correctly, it was eliminated as being too easy. Conversely, if a majority of the ABN board certified neuropsychologists failed to answer a multiple choice question correctly, then the multiple choice question was eliminated as being too difficult.  The multiple choice questions remaining after elimination of non-discriminating items were then revised, when necessary, to improve clarity of communication using the Measurement and Assessment in Education item writing training, and a final pool of 317 multiple choice questions was kept.  

Survey of Important Training and Clinical Knowledge Areas:
A survey instrument was developed and pilot tested on the ABN board and then sent to the membership of ABN. The survey assessed the areas of clinical importance in clinical neuropsychology based on the Houston Training Guidelines conference.  The survey involved ratings typically used in developing certification examinations (Henderson, 1996).  Respondents were asked to rate the importance of particular knowledge areas in the practice of clinical neuropsychology, the frequency in which these areas are used, and the potential harm that was possible if these knowledge areas were inappropriately applied.  From summarization of these ratings, the relative weighting of the number of questions needed in each area was determined.  Ratings were as follows:
1.    Neuroanatomy 11%.
2.    Neurological disorders 12%.
3.    Psychiatric disorders 12%.
4.    Neuroimaging 9%.
5.    Neuropathology 11%.
6.    Neuroscience 10%.
7.    Neurochemistry 7%.
8.    Psychometrics 13%.
9.    Brain behavior relationships 15%.

Using these weightings, questions from the remaining item pool were then assigned to each of the nine assessment areas.  Questions that did not fit clearly into any of the assessment areas were not assigned.  A pool of 200 questions were then kept using these weightings.  

Assignment to Form A or Form B
The 200 items were then administered to 25 practicing board certified neuropsychologists.  These 25 neuropsychologists were administered all 200 questions, and the passing frequency for each multiple choice question was calculated. These data were used to divide the test into two 100 item forms (A and B) where each of the individual knowledge areas (1 through 9) were approximately equally weighted across both forms of the exam and with the overall pass rate between the two forms was generally equivalent.  Thus, two 100-item exams were established.  Next the pass frequency of each question within each of the Knowledge domains was calculated.

Knowledge Area    Form A    Form B
Neuroanatomy     198    192
Neurological disorders    182    189
Psychiatric disorders    207    211
Neuroimaging    165    179
Neuropathology    162    167
Neuroscience     197    206
Neurochemistry    110    113
Psychometrics    236    235
Brain Behavior Relationships    255    252

A paired samples t test revealed that none of the individual knowledge areas were significantly different (p. = >.05) in their scores on Form A vs. Form B.  Performance overall on Form A vs. Form B showed a correlation of .99 (p= <.001) between the two forms.  Therefore the two forms are considered to tap the same basic knowledge areas with about the same level of item difficulty in each knowledge area.  

Following this step, both forms of the multiple choice exam were then administered to an additional 50 board certified neuropsychologists who took the multiple choice exam for normative purposes.  The performance of each question was carefully scrutinized to ensure that each question was in fact representative of basic neuropsychological knowledge consistent with the level of knowledge generally found at the board certification level.  A passing score on either Form A or Form B was established as a correctly answering 70% of the questions on the test.
Updating the Multiple Choice Exams:
Additionally, multiple choice questions that were submitted and completed the screening process were kept in a separate database so that additional questions can be added to the exam process as needed or as changes in the field of clinical neuropsychology warrant. Each additional multiple choice item will be subject to the same statistical procedures as the items currently in Forms A and B. Psychometric item weighting procedures will be used to ensure equivalent difficulty levels with previous forms of the examination.  At each ABN board meeting reports are made that include new data on the validation of the exam, and any issues related to questions that are no longer relevant are reviewed by the ABN board.  The written examination committee assesses the statistical properties of each of the exam questions on a yearly basis to identify whether any questions need to be changed, updated, dropped, or replaced. In addition, periodic surveys (every 3-5 years) of board certified neuropsychologists will be conducted to determine the proper weighting of content areas in clinical neuropsychology.  

References:
 
Henderson, J, P. (1996) Job Analysis.  In Certification: A NOCA Handbook
Edited by A. H. Browning, A. C. Bugbee and M. A. Mullins, Washington, DC: The National Organization for competency Assurance (NOCA)

Reynolds, C.R., Livingston, R. & Willson, W. (2008) (2nd Ed.) Measurement and Assessment in Education   Boston: Allyn & Bacon/Merrill