How could we be so WRONG?
A student's "wrong" answers delineate the upper boundaries of the extent of their content, process and linguistic structure mastery (proficiency) of a discipline content area.
We discard this important information at their peril and our future peril.
Patterns of response selection tell a more accurate story about achievement than the frequency of selection of the answers we expect to observe. The answers we expect are not necessarily "right," if there are legitimate alternative considerations we have missed or deliberately ignored. Our tests measure interpretation skills and not knowledge.
Ignoring this intervening variable (item interpretation) has contaminated much of our thinking about effective education. This error is invisible when we assume that selecting "wrong" answers comes from "blind guessing" and not from thoughtful alternative interpretations (some correct and some ill conceived.) The nature of the misconceptions and valid options provide important diagnostic information, as the research into concept inventories has shown.
Shame on us for taking too narrow a view of performance accomplishment. Our team has cracked this shell with an alternative scoring procedure, which includes all answer selections. It has been in the literature for more than 20 years (Powell and Shklov, 1992; EPM, Winter). Our habit of thinking that "being 'right'" is all that matters is as out of date as Ptolemy's Cosmology.
is a "submarine" a boat, a sandwich, a dwelling on the ocean floor or a US Marine of low professional rank?
Another example:
Not all birds have; wings, feet, bills or nests. There is no "right" answer to this question. Penguins have flippers; hawks have talons; owls have beaks and mocking birds use the nests of other birds.
Our research shows that we have misrepresented proficiency levels for more than two thirds of our students when changes in total scores are our sole criterion for scholastic success.
We make life's-future determinations on data of this poor a quality level, then we wonder why our schooling system is generally so inferior to other industrialized countries on international measures of proficiency.
Is it not time for a change?
Jay Powell
------------------------------
James Powell
CEO
Better Schooling Systems
------------------------------