Recent research studies on cloze tests have focused on the question of construct validity. Oller (1973) (1976) argues that doze tests measure some sort of global language proficiency. Chihara et al. (1977) and Chavez-Oller (1985) maintain that doze items are sensitive to constraints across sentences. Alderson (1983) and Porter (1983), however, provide some evidence to suggest that doze is essentially sentence-bound. These conflicting conclusions seem to have failed to take into account important factors in the test-making process.
The study reported here was designed to examine whether construct validity is affected by the type of passage used and familiarity with the topic. One narrative passage with 395 words in length was taken from a short story. Two expository passages of approximately 360 words each were taken from two encyclopedias. One of these was on a familiar topic concerning a baseball game and the other described a cricket game. Every 8th word was deleted from the narrative passage and every 6th word from the expository passages. The sentences of these doze tests were scrambled to produce one more test over each passage. Six doze tests were administered to 156 Japanese undergraduate students who consisted of three different English proficiency groups.
To determine whether there were significant differences between the mean scores for the narrative doze tests across the groups, a two-way ANOVA was performed. Two main effects were significant at p<.001; context (sequential vs. scrambled), F(1,150) = 9.43; level of English proficiency, F(2,150) = 23.55. However, the interaction was not significant. A three-way ANOVA was also employed for the scores of the expository doze tests. The main effects for level of English proficiency, F(2,300) = 99.35, and familiarity (baseball vs. cricket), F(1,300) = 185.20, were significant at p<.001 but the context effect was found not significant. No interactions were found significant except that between familiarity and level of proficiency, F(2,300) = 6.57. Item analysis performed on each doze test by means of item discrimination and facility indices showed that the sequential version of the narrative doze test included more items whose scores were consistent with the total ones.
The conclusion to be drawn from these experimental results is that different passages selected for doze tests and different degree of the topic familiarity have effects on the construct validity which cannot be neglected.