More on Rasch and again, here’s my original statement, then John’s, then my latest response…
7 Dick: To use these ordered series, we can then put every response into a matrix ordered by the difficulty level of the items and the ability level of students. As a result, we should see a Guttman scaling in which students tend to get all items correct that are below their ability level and no items correct that are above their ability level. From the matrix rows and columns, we also have the percentage of items correct for each student and percentage of students answering correctly for each item.
7. John: I have not seen a Guttman scale, but I think I basically understand this paragraph. It seems to me that the premise of this method is contradictory to my statement in paragraph 4 above—
"But if we use 10 problems, and Smart kid gets 9 right and Smarter kid gets 10 right, we don’t know which one Smart got wrong. Maybe he got a medium-difficulty problem wrong, not the toughest one."
The premise of your methodology here is that most likely we will in fact know which one problem Smart kid got wrong, and it will most likely be the #1 hardest problem.
Remember to distinguish between the calibration process of the test items with the derivation of the scale, and the subsequent use of the items and scale for assessing students after we have the scale and items analyzed. When we start out and first give students some unknown items we think might measure some underlying dimension of student growth, we don’t have any way of knowing anything about whether an underlying scale actually exists or what the difficulty level of the items is or their position and separation along the scale we think we might find.
When we use the Rasch model to analyze test items and students, we must use exact knowledge of which items each student answers correctly and which students answered an item correctly. The Rasch analysis depends on having this information. In other words, it is not a count of correct items we use, it is a huge matrix of each item in their order of correct responses and each student in their order or correct responses that we use. The cells in this (ordered item by ordered student) matrix shows whether each item was answered correctly or not. If the response data show a consistent pattern from no correct to suddenly all correct when we move along either a column or a row, then both the students and the items are following an order of difficulty or ability and we know we can find the underlying scale. It is from that matrix that we can derive probabilities for each cell and the conversion of the probabilities to a logarithmic scale.
