In their recent book, Methods Matter: Improving Causal Inference in Educational and Social-Science Research, Professors Richard Murnane and John Willett offer guidance for those who evaluate educational policies. They cover basic principles of causal inference and introduce complex concepts previously inaccessible to nonspecialists: randomization by group, natural experiments, instrumental variables, regression discontinuity, and propensity scores. Here, they discuss what inspired the book, their collaboration, and lessons they took away from the experience.
Can you describe your collaborative experience? Murnane: John and I have collaborated for nearly a quarter of a century, meeting almost every Tuesday for a few hours during the school year in Gutman Library. Throughout the summers, we harass each other mercilessly by telephone and email, as we live at opposite ends of Massachusetts. Sometimes we meet alone; sometimes we invite others to join us. Sometimes we agree, mostly we don’t. Sometimes one of us teaches the other, sometimes the reverse. Our arguments do not always conclude in shared understanding, but they usually do. Some topics have taken us many years to resolve, but we’ve really enjoyed taking this journey of discovery together and learning each step of the way.
What are some of the lessons you’ve learned? Murnane: I’d say there are two great lessons that have come from our collaboration. First, if you challenge yourself constantly with difficult problems that fall outside your comfort zone, you’ll continue to learn something new. Second, there’s nothing better in one’s professional life than sharing such experiences with close friends and colleagues.
Willett: A successful scholarly collaboration does not come from mutual agreement. In fact, we often find that our disagreements are what spur the most productive and insightful advances of our collaboration. The critical thing, we think, is to not let scholarly and professional disagreements drive you apart personally. There has to be a deep and abiding friendship at the core of any collaboration that feels worth preserving, whatever the nature of the current disagreement over form or function or method.
What inspired this book? Willett: Over the last 15 years, it became clear to us that innovative research designs and analytic practices were being developed constantly, and applied in the social sciences and statistics. We thought that these new methods of causal inference had enormous potential for resolving critical problems that plagued education research. Yet, when we examined the scholarly literature that was supposed to inform educational policymaking, we found that most of the quantitative research could not even support credible statements of cause and effect. Consequently, it seemed sensible to facilitate the implementation of the new methods of causal inference in the fields of educational and social science research. We wanted to persuade scholars, policymakers, and practitioners that there were substantial and powerful methods that could improve causal research in education and the social sciences.
The book is related to a class you teach here at the Ed School? Murnane: Yes. Over the last decade and a half, as our own ideas about causal inference began to crystallize, we tried to draw an adept group of up-and-coming young scholars at our school into an advanced doctoral seminar on the topic, to worry about the issues with us. This book actually grew out of that seminar. In both, our pedagogic approach has been to embed the learning of innovative methods for causal inference in substantive contexts, using empirical research papers from a variety of fields to introduce, explain, and illustrate the application of these methods.
What sets Methods Matter apart from previous literature in the areas of educational research and policymaking? Willett: Rather than intensive mathematical and statistical methods, we’ve opted to use conceptual, graphical, and data-based explanations throughout the book. We discuss in depth the controversies surrounding numerous studies in the U.S. and abroad, providing accessible explanations and offering strategies to overcome similar challenges in future policy decisionmaking. Our objective is to widen the reach and appeal of the methods to scholars who do not possess the same deep technical backgrounds as the developers and early implementers of the methods. In every chapter we try to present new methods for causal inference in a way that’s sensitive to the practical realities of the educational and social context. We hope not only to make our readers receptive to incorporating these methods in their own research, but also to see the value of the guidelines provided in the book for judging the quality of the research studies they read.