Loading ...
The following was emailed to Oregon’s Superintendent of Public Instruction, Susan Castillo, on 11/07/2011:
Hi Susan – I know you’ve reviewed the most recent NAEP results as have I. The distribution of reading achievement scores for grades four and eight remained essentially unchanged as they have for roughly the last two decades. How can this be? For the last decade, in particular, on a nationwide basis we have spent billions of dollars trying to improve reading achievement. We have spent lavishly on special education, the latest curriculum programs, response to intervention strategies, early childhood literacy programs, staff development programs, technology-based remedial programs – and yet achievement has not improved. Again, how can this be?
The answer is surprisingly straightforward.
In the NEAP results we are seeing the intersection of two controlling variables, differences in cognitive ability among students and the standardization of access to learning.
If you administered a high quality cognitive ability assessment to the same students who took the NAEP reading exam, you would see that the results map to each other to a very high degree. Lower ability students present lower reading achievement and higher ability students present just the opposite.
But if you also overlaid the time provided for learning to these same students you would find it almost identical for all levels of ability – about 6 hours per day for about 180 days per year.
Ability varies (as it always has), yet instruction time is about the same (as it has been for decades). More than three quarters of the variance in test scores can be explained by these factors alone.
The primary effects of ability differences are seen in variations in the rate, quality and retention of learning experienced by different students. Lower ability students need more explicit instruction and practice to master concepts and skills, are more prone to misconceptions needing detection and correction, and don’t retain as much in memory needing more structured review over time. The consequence is that successful, long term learning for lower ability students requires more time.
Even with exceptional instruction for all, student achievement within a group will diverge over time as a function of cognitive ability. So when you give an examination to all students of a given age at the same time, differences in achievement will inevitably appear.
So the NAEP reading results are unsurprising, nor are they likely to change much going forward – unless you change the ability profile of the population (not possible) or change access to instruction (possible, but not likely with the current economy).
Thus the “snapshot” view of achievement provided by NAEP is generally unhelpful as is the manner in which the current standards associated with it are (inaccurately) interpreted.
What we really need is an assessment system that can effectively measure year-over-year individual student achievement growth and disaggregate group data by various cognitive ability levels. This system would require new assessment tools and a new framework for interpreting results.
I urge you to use your leadership position to re-frame the accountability mindset currently in vogue and to move the state system of assessments in a more productive direction.
To this end I invite you to answer three questions:
(1) What is the psychological rationale for the current standards-based assessment system?
(2) Why aren’t the measurement and reporting of individual student growth the center point of the current state assessments?
(3) Why is cognitive ability not systematically measured and used to predict achievement?
With achievement growth and ability data in hand, a meaningful program of instructional improvement could actually have an empirical foundation. And the reporting of student progress would be meaningful. And the accountability focus for public education could actually be anchored in a sensible context.
This is a big challenge. But it might provide a memorable capstone to your long and sincere intent to improve public education.
All the best.
Ron
Hello Ron,
Can you expand on your conclusion, “What we really need is an assessment system that can effectively measure year-over-year individual student achievement growth and disaggregate group data by various cognitive ability levels. This system would require new assessment tools and a new framework for interpreting results”?
At first blush, that seems like a statistical/analytical nightmare to administer. Are you suggesting more individual testing/evaluation?
Hi Kona – In Oregon, the transition to a growth-based system would be relatively straight forward. The Oregon Assessment of Knowledge and Skills (OAKS) already produces scale scores for students – a necessary ingredient for calculating growth. Since the grade level assessments are all linked, year-to-year growth calculations would be relatively simple.
However, the state assessment would need to add a cognitive ability measure. The good news is that there are several reliable and valid standardized assessments available. These tests generally take about one hour to administer, so the burden on schools and students would be minimal.
The state assessment folks would need to apply some new statistical tools to establish the linkages between ability and achievement. But these tools already exist and are well-tested.
The big challenge is to change the mindset of policy makers. Technically, the system could be put in place in a year or less – once the initiative is approved.
For a good, current discussion of growth measures you might want to review Doug Harris’ new book, Value-Added Measures in Education.
Ron,
Are you referring to a system that educates the student without group grade distinction? A system in which each student’s progression is determined by their individual abilities rather than their age?
You said, “The state assessment folks would need to apply some new statistical tools to establish the linkages between ability and achievement.
Hi Kona – No, I’m not refering specifically to ungraded instructional programs, though such programs might be a workable alternative in some settings. I’m making a more general point that we need to take students where they are, being as fully aware of their learning needs as possible, and help them move forward in their learning as much as possible. In this framework, learning growth is the central focus guiding instructional decision-making regardless of the instructional configuration.
As for the statistical tools, we need a way of recognizing “good” growth when we see it, knowing that growth rates vary across ability levels. So growth targets will not be the same for all students. The data analysis tools for establishing these types of relationships are well known and ready to be applied.
You are right on target especially with the concept that kids learn at different rates. They also learn in different ways and demonstrate learning in different ways.
To respond to your questions,
1. There is no psychological rationale for the current testing system other than assure the same students go into the subclass for the rest of their lives.
2. Those in charge have no concept of human growth and development. Kids are supposed to move ahead in robotic sameness like plastic wivots on an assembly line. The reality is kids blossomn at different times.
3. Schools are consistanty with the 18th century design, they don’t want kids moving at a slower pace to succeed.
To make the necessary changes, many dominoes will fall and that is scary to those comfortable with failure. Go to http://www.WholeChildReform.com for my books detailing the plan for change
“Cognitive ability” can change in response to physical and environmental factors. If you measure a child who hasn’t had a meal since the day before or one who is living in a violent household or whose mom is out of work, they’re not going to do as well. Why are we focusing on testing cognitive ability rather than changing the social and environmental factors that negatively effect not only cognitive but also emotional, physical and social health? Put the money where it’s really needed. Children are human beings, not data points.
Hi Meredith – The factors you cite may well influence a child’s performance on an assessment on any given day. And on school performance in general. About 25% of the variance in tests scores arises from factors like the adverse effects of poverty, child neglect, poor instruction, emotional trauma, etc. I agree that we, as a society, should devote more resources to improving these conditions.
But a relatively small investment in measuring learning, the central focus of public education, is also important. By current estimates we spend less than 2% of our education dollars on all assessment activities. This is evidence of underinvestment in an essential educational activity.
I argue for measuring cognitive ability systematically because it is at the heart of giving students fair and accurate information about learning progress. The current snapshot system condemns many students to the perpetual status of “not meeting standards” for no justifiable reason. I’m with you, lets focus on individual students and their growth and work to optimize each person’s potential.
Kids who are economically disadvantaged show reading growth along with middle class peers but lose ground in summer while middle class doesn’t. Over time the gap widens. It’s true in US and true in Australia.
http://www.eduweb.vic.gov.au/edulibrary/public/publ/research/publ/researcharticle_the_summer_achievement_gap.pdf
We need to stop applying rocket science and focus on common sense solutions.
Hi Pam – You are quite right that summer learning loss is a problem for many (but not all) economically disadvantaged students. But it is also a problem for lower ability students, regardless of economic background, and for some special needs students, including those with learning disabilities.
Elsewhere I have argued for a longer school year with a much shorter summer recess, which would, in part, address this problem.
In very challenging circumstances, we may need to offer some students even longer (extended) school years with more review, re-teaching and applied practice.
But of course in the current economic climate we are struggling to maintain even the traditional 180 day school year.
And I would have to say that given the complexity of the human brain and the billions of cognitive transactions performed during learning, the study of education is pretty close to rocket science. Causes and effects don’t make themselves easily known. If there were simple, common sense solutions, they would have been applied long ago and the problems of underachievement would only be a distant memory. I’m afraid that there’s still lots of hard, rocket science work to be done.
Ron, I agree.
You said, “Elsewhere I have argued for a longer school year with a much shorter summer recess, which would, in part, address this problem”.
I would extend that to seven six-week terms with a week off every six weeks. This would produce about a two or three week summer vacation.
What are the problems with the above suggestions? Any benefits? Would the unions object to this?
Your idea seems like a good one, Kona. I see two benefits: (1) no summer learning loss and (2) less time needed to re-establish classroom routines in the fall.
There will be compensation issues as teacher associations generally think of employment from a daily rate perspective. I think the increased days of instruction can be managed through good faith negotiations or legislation or some combination of both.
In any case, increasing the length of the school year as we are discussing it is a high leverage improvement that yields lots of bang for the buck. We would get more achievement per dollar invested than we get from most interventions.
Yes, the 210-215 day contract would be a difficult expectation. There was a programmed effort in Oregon about two decades ago to have large class sizes and short school years while having relatively very high teacher compensation (the K-12 Oregon teacher compensation was the sixth highest of all states a decade ago). So my guess is that extending the school year and maintaining compensation equal to the median state would be difficult in our political climate.