Week 7 DISCUSSION
Response Format
Selecting response formats (i.e., the choices test-takers have to respond to test items) for test items is another crucial consideration when building a test. The first major response format consideration typically encountered is whether to require open-ended or closed-ended responses. The Rorschach inkblot test is an example of a test that elicits open-ended responses. There are practically no limits to what an examinee can say when responding to this test. At the other extreme are tests that require the examinee to choose between a very limited set of responses, such as true/false, yes/no, or multiple-choice tests, in which only one of several options is correct.
There are advantages and disadvantages to open-ended response formats. For instance, open-ended items can provide rich information, but developing a reliable scoring system is often difficult. Scoring systems for the Rorschach have developed over many decades, are difficult to learn, and their utility is controversial. Individually administered IQ tests, such as the Wechsler tests, contain open-ended items in which examinees are asked to define vocabulary words. Creating scoring criteria for each vocabulary word requires collection of typical responses, expert judgment, and assessment of scoring criteria for inter-rater reliability and accuracy. Test administrators need to have extensive training to correctly administer and score this test.
A very common and useful closed-ended response format is Likert scaling. This format asks for each item to be rated on a multipoint scale, typically, 1: Strongly disagree, 2: Disagree, 3: Neutral, 4: Agree, 5: Strongly agree. Variations include more or fewer points on the scale, different scale labels, rating of frequency vs. intensity, and inclusion or elimination of a neutral scale midpoint. Another consideration is whether to include reverse-scored items. For instance, an anxiety scale might include the item “I am nervous almost all of the time.” Giving the item a high rating, such as 4 (Agree) would indicate a higher anxiety level. You might also, however, include the item “I am usually calm” on your anxiety scale, so that a 4 would actually indicate a lower anxiety level. Reverse items are adjusted before statistically analyzing the scale by subtracting scores from one more than the highest rating on the scale. With a 5-point scale, ratings of a reverse-scored item would be subtracted from 6, so that 5 becomes 1, 4 become 2, etc.
Other response formats, such as ipsative (also called forced-choice) items are less common. These items ask the examinee to choose between options or to rank the desirability of various choices.
To prepare for this Discussion, consider the appropriateness of various response formats for the test you are proposing for your Final Project. Select two different response formats to use for this Discussion.
With these thoughts in mind:
Readings
- Kline, T.
(2005). Psychological testing:
A practical approach to design and evaluation. Thousand Oaks, CA: Sage.
Kline, T., Psychological testing: A practical approach to design and evaluation. Copyright 2005 Sage Publications Inc. Books. Used with permission from Sage Publications via the Copyright Clearance Center.- Chapter 3, “Designing and Scoring Responses”
- Fisher, S. T., Weiss,
D. J., & Dawis, R. V. (1968). A comparison of Likert and pair comparisons techniques in
multivariate attitude scaling. Educational and Psychological Measurement, 28(1),
81–94.
Fisher , S. T., Weiss, D. J., & Dawis, R. V., A comparison of Likert and pair comparisons techniques in multivariate attitude scaling, in Educational and Psychological Measurement. Copyright 1968 Sage Publications Inc. Journals. Used with permission from Sage Publications, Inc. via the Copyright Clearance Center. - Lissitz, R. W., &
Green, S. B. (1975). Effect of the number of scale points on reliability: A
Monte Carlo approach. Journal of Applied Psychology, 60(1),
10–13.
Retrieved from the Library databases. - McCrae, R. R.
(1994). The counterpoint of
personality assessment: Self-reports and observer ratings. Assessment, 1(2),
159–172.
McCrae, R. R., The counterpoint of personality assessment: Self-reports and observer ratings, in Assessment. Copyright 1994 Sage Publications Inc. Journals. Used with permission from Sage Publications, Inc. via the Copyright Clearance Center. - Shrout, P. E., &
Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater
reliability. Psychological Bulletin, 86(2), 420–428.
Retrieved from the Library databases.
