Violating Conventional Wisdom in Multiple Choice Test Construction.
This research examined 2 elements of multiple-choice test construction, balancing the key and optimal number of options. In Experiment 1 the 3 conditions included a balanced key, overrepresentation of a and b responses, and overrepresentation of c and d responses. The results showed that error-patterns were independent of the key, reflecting selection of the most plausible foil. Experiment 2 examined the optimal number of options. The comparison of a 3-option to a 4-option test showed that the 3-option test retained reliability, while allowing for a greater sampling of information. These findings suggest channeling of effort to the content of items, rather than focusing on the convention wisdom in mechanics of test construction. ********** Although many sources address the issues of how to write individual multiple-choice test items, there are few research-based guidelines for overall construction of multiple-choice tests. The present research examined two issues involved in the mechanics of overall test construction, reflecting tasks that take up much added time and energy. The first issue involves 'balancing the key'--assuring that each option is chosen as 'correct' an approximately equal number of times per exam. The second issue involves choosing the optimal number of options per item. Experiment 1: Balancing the Key for Multiple Choice Tests Balancing the key is a procedure in which each option should be correct an approximately equal proportion of the time. Conventional wisdom suggests that a balanced key is desirable in that it minimizes the effects of guessing. For example, Haladyna (1994) suggested that any serious departure from this formula would lead to higher-performing students deducing the pattern of correct answers and selecting guesses accordingly. Lower-performing students would be more disadvantaged, consistently selecting a single option for guesses, regardless of pattern. In addition, Haladyna, Downing and Rodriguez (2002) strongly recommend varying the location of the correct answer according to the number of choices, a procedure that would balance the key. In their review of 19 studies published since 1990, 52% explicitly recommended this procedure, 48% failed to address the issue altogether, and none spoke against it. Sechrest, Kihlstrom and Bootzin (1993) further advocated balancing the key, based on the premise that students may use a response set for guessing, either biasing their tests towards more or fewer correct answers based on guessing. Ory and Ryan (1993) also encouraged use of a balanced key; however, they did not provide a rationale for their suggestion. Ellsworth, Dunnell, and Duell (1990), in their critique of test banks that accompany textbooks, also found that most test banks do not provide a balanced key. Option c is used most often and option a is used least often. They used these findings to support their conclusions regarding the poor quality of test banks. The implication is that a balanced key is a desirable quality. Unfortunately, no studies empirically examining the balancing of the key are found in the literature. The widespread promotion of this conventional wisdom, which seems intuitively valid, comes from recommendations with no accompanying evidence. This process, however, often results in the expenditure of extra time and effort in test construction--time better allocated to the quality of individual items. The present study empirically examined whether a balanced key would make a difference in test performance. Method Participants Eighty introductory psychology students, primarily traditional freshmen participated in this study. There were two sections of 40 students each, with participants randomly assigned to conditions individually at each test. Thus, any student across both sections may or may not have participated in all three conditions based on random assignment across three tests. …