When Testing Takes Over

Daniel Koretz has spent a career studying educational assessment and testing policy, weighing the consequences of high-stakes accountability tests. In a bracing new book that might be seen as a capstone to that work, Koretz excoriates our current reliance on high-stakes testing as a fraud — an expensive and harmful intervention that does little to improve the practices it purports to measure, instead feeding a vicious cycle of pointless test prep. The book’s title, The Testing Charade, captures his point; excessive high-stakes testing undermines the goals of instruction and meaningful learning.

For parents, teachers, school leaders, and advocates who want to understand how we got here, the book is an accessible exploration, charting a path toward more sensible assessment practices. We asked Koretz, a professor at the Harvard Graduate School of Education, to reflect on how current testing policies touch the lives of parents and teachers — and how they can advocate for change.

From a parent’s point of view, the public conversation around testing can seem quite binary. There’s a pro-rigor and achievement camp, and there’s an anti-testing, opt-out camp. Can you offer a balanced framing of this for parents?  

As I stress in The Testing Charade, standardized tests themselves are not the problem; the problem is the misuse and sometimes outright abuse of testing. Testing done right can be valuable, sometimes irreplaceable. For example, how do we know that the performance gap between African-American and white students is slowly narrowing, or the gap between poor and well-off students has been growing at the same time? Standardized tests.

And standardized tests, designed and used appropriately, can help teachers improve instruction. Indeed, the main use of standardized tests many years ago, when I was in school, was to improve instruction, not to hold teachers accountable.

Ironically, one of the many harms inflicted by excessive high-stakes testing is that it has undermined the main benefits of good standardized testing. In many places, it has led to severe score inflation — gains in scores far larger than real improvements in learning. In some cases, score gains have been three to six times as large as real gains in achievement. These inflated scores don’t provide an honest and useful indication of student performance. And the pressure to raise test scores has become so strong that testing often degrades instruction rather than improving it. Many parents have encountered this — for example, large amounts of teaching time lost to test prep that is boring, or worse.

It’s time to curtail the inappropriate uses of tests, but let’s use tests appropriately when they can help us help kids.

In the current testing landscape, teachers have to spend a good deal of energy rallying the troops every year. Bad test prep, as you call it, can derail a class and block real learning. How can teachers reframe this narrative, in the near term?

To undo the problems created by test-based accountability, teachers must refocus instruction on teaching the underlying knowledge and skills that any good test should reflect, rather than spending time preparing kids for the specific test used for accountability.

You can think of this as similar to a political poll. To estimate how the electorate will vote, you sample a very small number of people. To estimate how much mathematics students have learned over a long period of time, you ask them a small number of questions, and you use those few questions as a guide. However, you don’t win an election by trying to persuade the few people contacted in a poll; you win it by persuading the electorate. And you don’t improve students’ mathematical knowledge and skills by focusing on the small sample of specific material sampled in one test. You do it by teaching them math.

However, while this sounds straightforward, it is very hard for teachers to do. First, many are trapped. Better teaching of content will raise scores on a good test, of course — but not nearly as fast as the bogus gains that can often be achieved by means of bad test prep. Under the current system, many teachers face performance targets that make it a real risk to choose the real but slower gains that can be achieved by improving teaching. Second, many younger teachers have been trained to believe that test prep is good instruction.

So let’s be fair: if we want teachers to abandon bad test prep, we have to change the incentives and constraints that confront them.

What is an accountability system that works? What should parents and teachers be advocating for?

1. To start, we have to measure what matters — what we consider most important for judging the quality of a school. Ask yourself: what do I want to see when I walk into a classroom? The accountability system should give teachers credit for these. There are more, but I would start with what I called in Charade the Big Three: student achievement (which goes well beyond what can easily be measured with standardized tests), the quality of educators’ practices, and school climate. If you want to see classrooms that are engaging — I certainly do — then we should give teachers credit for creating them.
   
   The test-based accountability system was based on the idea that if you measure just a few of the important things — primarily, testable performance in math and reading — the rest would follow along. That was naïve. We know from decades of research in many fields, not just in educational testing, that if you pressure people to improve on just a few aspects of their job, some of the other important aspects of the job will stay the same, and some will deteriorate.
    
2. We can’t just demand that teachers do better. Many will need support to help them reach the goals we set for them.
    
3. We have to use tests wisely. Used sensibly, they can provide valuable information to help improve instruction. However, treating them as the main goal of education and exerting too much pressure to raise scores as an end in itself will backfire, undermining instruction and giving us misleading information about students’ performance.
    
4. We have to set reasonable targets. In recent years, performance targets were often set without any regard for the gains that teachers can make by legitimate means. Often, they were unrealistically demanding. We’ve seen what happened. Faced with targets they couldn’t meet by legitimate means, some teachers faced a stark choice: fail, cut corners, or cheat. We’ve seen all three.
    
5. We need to control efforts to game the system. Once again, we can turn to research in many fields, not just education. Pressure to meet performance targets often creates gaming — efforts to look good on the particular measures used. This is so common that it is called “Campbell’s Law” in social science and economics. The solution is not to avoid holding people accountable; it’s to watch for the gaming and take steps to keep it in check. For example, if teachers feel under pressure to raise test scores, someone in the system should be monitoring how they are trying to raise them.
    
6. Finally, we need to evaluate new interventions, monitor their effects once put into place, and change course if necessary. In many areas of public life, we expect this as a matter of course. We expect that we won’t be given medications until they are shown to be safe and effective. We expect that auto manufacturers will have to submit their cars for crash testing, and we expect that if dangerous flaws become apparent later, those cars will be recalled. None of this has been true in education during the period of test-based accountability. Time and time again, policymakers have made up programs they think will work and have then imposed them wholesale on schools without first trying them out. In addition, it has been hard to monitor their effects after the fact, in part because states and districts are not required to open their data to scrutiny. Perhaps even more troubling, when signs of problems have become apparent, they have often been ignored.

If you are a parent who is concerned or wanting to advocate for more sensible accountability measures, are there steps you can take?

I’d work my way up from the ground level, because an individual parent is likely to have more impact within their child’s school than outside of it. Teachers and principals face constraints imposed by the law and state and district policy, but you can encourage them to make use of the wiggle room they do have. It’s not always comfortable to do so, but if you see something that troubles you — say, lots of time spent on what seems to be boring test prep — talk with the teacher about it. By the same token, let the teacher know when you see something you want to encourage — say, an innovation in class that excites your child.

When you talk with the principal, come armed with examples of what you find troubling, what you find praiseworthy, and what you would like to see implemented. For example, if there is too much test prep, ask that the principal monitor it and take steps to reduce it. Ask how the principal (and the district) measures other things that matter, like the quality of a teacher’s instruction and the climate she creates in the classroom. I discuss other steps you might take in the final chapters of The Testing Charade.

After that, I would work my way up. Contact the relevant district administrators and members of your school board, and again, be prepared to be specific about what you would like tamped down and what you want the district to encourage. Doing this as a member of a group of like-minded parents might help. The bigger the district, of course, the lower the probability that this will have an impact, but if you don’t ask, you don’t get.

Illustration: Wilhelmina Peragine