The following was emailed to Oregon’s Superintendent of Public Instruction, Susan Castillo, on 11/07/2011:
Hi Susan – I know you’ve reviewed the most recent NAEP results as have I. The distribution of reading achievement scores for grades four and eight remained essentially unchanged as they have for roughly the last two decades. How can this be? For the last decade, in particular, on a nationwide basis we have spent billions of dollars trying to improve reading achievement. We have spent lavishly on special education, the latest curriculum programs, response to intervention strategies, early childhood literacy programs, staff development programs, technology-based remedial programs – and yet achievement has not improved. Again, how can this be?
The answer is surprisingly straightforward.
In the NEAP results we are seeing the intersection of two controlling variables, differences in cognitive ability among students and the standardization of access to learning.
If you administered a high quality cognitive ability assessment to the same students who took the NAEP reading exam, you would see that the results map to each other to a very high degree. Lower ability students present lower reading achievement and higher ability students present just the opposite.
But if you also overlaid the time provided for learning to these same students you would find it almost identical for all levels of ability – about 6 hours per day for about 180 days per year.
Ability varies (as it always has), yet instruction time is about the same (as it has been for decades). More than three quarters of the variance in test scores can be explained by these factors alone.
The primary effects of ability differences are seen in variations in the rate, quality and retention of learning experienced by different students. Lower ability students need more explicit instruction and practice to master concepts and skills, are more prone to misconceptions needing detection and correction, and don’t retain as much in memory needing more structured review over time. The consequence is that successful, long term learning for lower ability students requires more time.
Even with exceptional instruction for all, student achievement within a group will diverge over time as a function of cognitive ability. So when you give an examination to all students of a given age at the same time, differences in achievement will inevitably appear.
So the NAEP reading results are unsurprising, nor are they likely to change much going forward – unless you change the ability profile of the population (not possible) or change access to instruction (possible, but not likely with the current economy).
Thus the “snapshot” view of achievement provided by NAEP is generally unhelpful as is the manner in which the current standards associated with it are (inaccurately) interpreted.
What we really need is an assessment system that can effectively measure year-over-year individual student achievement growth and disaggregate group data by various cognitive ability levels. This system would require new assessment tools and a new framework for interpreting results.
I urge you to use your leadership position to re-frame the accountability mindset currently in vogue and to move the state system of assessments in a more productive direction.
To this end I invite you to answer three questions:
(1) What is the psychological rationale for the current standards-based assessment system?
(2) Why aren’t the measurement and reporting of individual student growth the center point of the current state assessments?
(3) Why is cognitive ability not systematically measured and used to predict achievement?
With achievement growth and ability data in hand, a meaningful program of instructional improvement could actually have an empirical foundation. And the reporting of student progress would be meaningful. And the accountability focus for public education could actually be anchored in a sensible context.
This is a big challenge. But it might provide a memorable capstone to your long and sincere intent to improve public education.
All the best.