Heather C. Hill

The Mathematical Knowledge for Teaching Measures: Refreshing the Item Pool (2016-2019)
National Science Foundation

This study describes an assessment strand late-stage design project that focuses on improving existing measures of teachers’ mathematical knowledge for teaching. Original measure development, which occurred at the University of Michigan during the period 2002-2010, had several goals: to identify the knowledge useful to teachers’ work with students and to explore the possibility that this knowledge is unique to teaching; to provide a set of measurement instruments that could be used in research on teachers’ knowledge; and to provide evaluators with an easy-to-use online administration and scoring system. These efforts resulted in widely disseminated instruments, numerous academic papers, and theoretical progress regarding the mathematical knowledge teachers use in their work. We now seek to update these measures. One reason is their wide use: anecdotal evidence suggests that up to 25% of our target teacher population may have taken a version of these instruments in pre-service training or in-service professional development. To respond to this issue, we will create over 300 items and 10 new sets of parallel forms in the most frequently tested grades and topics. Another reason to update the measures relates to new mathematics content and practice standards; a review of existing forms suggests we could better align our item pools to this key instructional guidance. We also seek to respond to a variety of user requests, and to also make our online delivery system, the Teacher Knowledge Assessment System (TKAS), more flexible in both the item formats it can accommodate and in responding to additional form updates.

Strengthening the Research Base that Informs STEM Workforce and Curriculum Improvement Efforts (2014-2017)
National Science Foundation

This study will produce a comprehensive synthesis of the research base that informs science, technology, engineering, and mathematics (STEM) teacher workforce and curriculum improvement efforts. The project team will utilize meta-analytic techniques to estimate the impact of STEM teacher professional development and novel curriculum materials on student outcomes, and analyze the relationships between program effectiveness and key moderators identified in the literature, such as duration, intensity, format, grade and disciplinary topic, and alignment with NCTM/NSTA standards. The completed synthesis will provide researchers, practitioners, and funders with evidence to inform future STEM workforce and curricular investments. Identifying the characteristics of effective teacher professional development and curricular improvement efforts has been identified as a ‘grand challenge’ facing contemporary U.S. STEM education (e.g. Wilson, 2013). A core intellectual contribution of the current study will be to produce the most comprehensive and rigorous synthesis to date of the causal research evidence on this topic. Prior reviews in this domain have suffered from a paucity of studies, which in some cases necessitated the inclusion of studies whose weak designs failed to rule out threats to validity. This project will improve upon prior reviews by synthesizing the substantial newly-available evidence from rigorous impact evaluations funded by the Institute for Education Sciences (IES) and the National Science Foundation (NSF), combined with additional causal studies identified via a comprehensive literature search. Researchers will utilize meta-analytic models and conduct theory-driven moderator analyses to identify characteristics that are associated with interventions’ effectiveness. This project will advance the goal of improving K-12 teacher professional development and curricular interventions for STEM disciplines in the United States. The comprehensive research synthesis report will provide information to the National Science Foundation on characteristics of effective and ineffective professional development and curricular investments, which could inform funding priorities. In addition, the project team will make results available to researchers and practitioners. Researchers will be able to utilize the study’s results to inform both theory-building and future study designs in these domains. To reach practitioners, the study team will produce short briefs and disseminate them via research-practice networks as well as professional organizations. Ideally, practitioners will utilize the study’s results to guide their own school- and district-level resource allocation decisions regarding professional development and curriculum investments. By strengthening the evidence base that informs investments in STEM workforce development and curriculum improvement efforts, this research advances the ultimate goal of bolstering instructional effectiveness and US students’ achievement and outcomes in mathematics and science.

Survey of U.S. Middle School Mathematics Teachers and Teaching (2014-2018)
National Science Foundation

For the last 25 years, three major goals have animated the U.S. mathematics education community: more knowledgeable teachers, more challenging curricula for students, and more ambitious instruction in classrooms. Yet despite volumes of policy guidance, on-the-ground effort and research over the past decades, few comprehensive and representative portraits of teacher and teaching quality in U.S. mathematics classrooms exist. Instead, most research into these topics has been conducted with small samples, non-representative samples (e.g., Kane & Staiger, 2012) or by research projects that use diverse instruments, meaning that it is difficult to ascertain what, if any, progress has been made toward the three goals. To provide information on such progress, we will collect data on teacher content knowledge, curriculum use, and instruction from a nationally representative sample of U.S. middle school mathematics teachers. A written survey component of this study will build on a similar study conducted in 2005 – 06 (Hill, 2007), allowing for the comparison of teachers’ curriculum use and content knowledge – and more specifically, their mathematical knowledge for teaching (MKT) –across time periods. An observational component will record and score videotapes of instruction, allowing for a description of current instruction as well as a comparison of current instruction with that observed during the TIMSS video study (Heibert et al., 2005). The new video dataset will also serve as a baseline for future studies of instruction, for instance ones comparing current instruction to that in 2025, to assess whether Common Core State Standards have been met. This study is largely descriptive, as are many other studies of its kind (Hiebert et al., 2005). However, as in our past research (Hill, 2007; Hill et al., 2008), we can inquire into relationships between key variables (e.g., curriculum use, MKT, and instructional quality). Such inquiry can form the basis for future research. We can also describe instruction as it exists across a wide variety of U.S. classrooms, for example, asking whether – as is often assumed – instruction in urban districts is inferior to those in other areas and whether differences in instructional or teacher quality by academic track (honors, general, or remedial) exist. Finally, we believe that describing the mean and range of U.S. instruction can have a profound effect on the field, much as the TIMSS video studies (Stigler & Hiebert, 1999; Hiebert et al., 2005) did over a decade ago. We argue that timely information on these topics would be useful for three reasons. First, mathematics educators have both researched and attempted to improve teacher knowledge, curriculum, and instruction for over two decades; examining current levels of each would provide evidence about the extent to which these goals have been met. Second, the Common Core has set new goals for teachers and teaching; it is important to establish a baseline for future research by describing the status quo as new curriculum materials and assessments reach classrooms. Finally, we believe that the work described in this proposal will demonstrate that teacher and teaching quality in the U.S. can be effectively tracked, similar to other national indicators such as NAEP.

Center for the Study of Interactive Knowledge Utilization (2014-2017)
IES

CEPR will lead instrument development and survey administration work for the Knowledge Utilization Center. We will engage in an iterative cycle of instrument development, piloting and revision during year 1. This will include literature review and construct development, item development, cognitive interviews and analysis, and rostering 15-20 positions for a sample of 1000 districts. We will also plan a ‘deep dive’ survey of 30 large districts where all positions within our sampling categories (positions likely to involve instructional decisions at the k-8 level) will be targeted, and conduct pilot testing with 200 respondents. In year 2, we will sample and administer surveys to one respondent each in 1000 mid-sized and large districts, as well as all positions rostered in the 30 deep dive districts. The surveys will be administered via mail, including a personally signed pre-letter, survey package, incentives and several follow-up contacts. We will supplement standard survey techniques with email contacts and the option for individuals to complete the survey online. We will participate in analytic work and contribute to paper writing in year 3. We will also do a small survey of schools, targeting a few individuals in each of 120 schools, with the same effort to roster, mail and follow-up as noted above. We will contribute to technical reports and begin publishing measures on the project website. In years 4 and 5, a small team will work to add instruments and tools to the website, along with other dissemination activities.

Developing Common Core Classrooms Through Rubric-Based Coaching (2014-2017)
National Science Foundation

This project will develop and field-test a mathematics-specific coaching protocol centered on the Mathematical Quality of Instruction (MQI) instrument, a Common Core-aligned observational tool. The coaching model will consist of three elements: a summer training workshop, personalized web-based coaching, and access to a library of resources to support coaching and instructional change. During the summer workshop, teachers will be introduced to the instrument, score high- and low-exemplar videos in small groups, and reflect on their own practice. Each teacher will then participate in 20 video feedback cycles with a personal MQI-trained coach. Teachers will record and upload a lesson which coaches will review, providing written feedback and choosing three short clips for discussion. The coach and teacher will then review the clips together, discuss feedback, set targeted goals for the next coaching cycle, and plan further review of exemplar clips from our video library of Common Core-aligned practices. We will work with 3rd through 8th grade mathematics teachers in the Appleton Area School District to implement and evaluate the coaching model. Because we believe the online format is limited in its capacity to address teacher content knowledge, we will exclude teachers with very low mathematical knowledge for teaching, then randomly assign teachers to treatment and control conditions, stratifying on grade level. We will evaluate the effect of the coaching model in both the concurrent and post-intervention year, as past studies have found effects only after teachers have completed the full cycle of coaching (Allen et al. 2011; Campbell & Malkus, 2011). Our primary outcome measures will include teachers’ classroom performance as captured by the MQI, students’ perceptions of their teachers’ effectiveness, and student test scores.

Exploring Methods for Improving Teachers' Mathematical Quality of Instruction (2012-2015)
National Science Center

In this exploratory study, we will develop and test professional development aligned with the Mathematical Quality of Instruction observational instrument. In the program, teachers will learn to score videotapes of instruction using the MQI framework first by completing online training and then through participating in a series of groups meetings with others at their school. These meetings will combine aspects of video clubs (Borko, Jacobs, Eiteljorg & Pittman, 2008; van Es and Sherin, 2006) and lesson analysis (Santagata & Angelici, 2010), using the MQI framework to view and score portions of lessons. A comparison group will enable us to understand the effects of the MQI on several measures, including teachers’ talk, their analysis of videotapes, and reflections on their own lessons. Owing to cost constraints, we will leave the collection of data on instructional or student outcomes to a future study. However, we believe that the outcomes of interest noted above will provide adequate proxies to practice, in that they have been linked to practice or are precursors to the improvement of practice. In addition, we will systematically vary the conditions under which the MQI-based professional development is delivered in order to gain knowledge about the best design for a scaled-up implementation. To do so, we will expose fifteen groups of fourth and fifth grade teachers to MQI-based lesson analysis. These fifteen groups will be randomly assigned into five conditions (three groups per condition), with each group engaging in weekly or bi-weekly lesson analysis using the MQI. However, the amount of facilitator intervention (less vs. more), the origin of the videotape (teachers’ own video vs. video from a library), and the delivery method (face-to-face vs. internet) will vary by condition.

Impact Evaluation of Math Professional Development (2012-2016)
U.S. Department of Education

AIR, in partnership with Harvard University, Clowder Associates and Measured Decisions Inc., is proposing to develop a rigorous impact study of a specialized mathematics PD intervention for grade 4 teachers designed to strengthen content knowledge and encourage the enactment of content knowledge in classroom instruction. The 42-month study, which is being designed in response to a request from the National Center for Education Evaluation (NCEE) at the Institute of Education Sciences (IES), is designed to estimate unbiased impacts of the PD intervention on teacher knowledge, pedagogy, and student achievement. The PD intervention includes the Intel Math program (80 hours), augmented by the Mathematics Learning Community (MLC) program (30 hours), and a half-time facilitator-coach per district. Key outcomes measures include an adapted form of the Mathematical Quality of Instruction (MQI), developed by Harvard University as well as teacher knowledge assessment items from the Mathematics Knowledge for Teaching (MKT) instrument. The experimental design uses teacher-level random assignment and will involve approximately 200 teachers in 6 districts. Primary data collection activities will occur during the 2013-14 school year.

Investigating the Effects of Professional Development, Mathematical Knowledge for Teaching, and Instruction on Student Outcomes (2009-2014)
National Science Foundation

In this research, the research team sheds light on twin problems facing efforts to improve teacher quality in mathematics. The first problem is theoretical: What mathematical knowledge do teachers need to effectively instruct children? Does teacher basic, advanced, or profession-specific knowledge (e.g, mathematical knowledge for teaching, or MKT) matter most to student outcomes? The second problem is more practical: Can a particular professional development program, Math Solutions, improve teachers’ mathematical knowledge for teaching, their instruction, and student outcomes? To address these twin problems, the researchers conduct a cluster randomized trial to examine the efficacy of Math Solutions on elementary teachers’ MKT, their instruction, and their students’ learning. We chose to embed the study of teacher MKT in an evaluation of professional development because it is not possible to randomly assign teachers to different levels of MKT. Instead, the tearchers are randomly assigned to Math Solutions with the goal of examining the contribution of changes in teachers’ MKT, as it develops in and through the professional development, to instruction and student achievement. Data collection takes place in the Albuquerque Public Schools over three years and involves 80 fourth and fifth grade teachers in approximately 12 schools. Half the eligible teachers are randomly assigned to the mathematics professional development and half receive coaching in another subject. Longitudinal data is collected on teachers’ mathematical knowledge, instruction, and their student gains, allowing us to examine the effect of Math Solutions on these outcomes.

Hill, H. C., Beisiegel, M., & Jacob, R. (2013). Professional Development Research Consensus, Crossroads, and Challenges. Educational Researcher, 42(9), 476-487.,(2013)

Hill, H.C. & Grossman, P. (2013). Learning from Teacher Evaluations: Challenges & Opportunities. Harvard Education Review 83, 371-384.,(2013)

Hill, H.C. (in press). The nature and effects of middle school mathematics teacher learning experiences. Teachers’ College Record.,(2012)

Hill, H.C., Umland, K. L. & Kapitula, L.R. (2011). A validity argument approach to evaluating value-added scores. American Educational Research Journal.,(2011)

Hill. H.C. (2010). The nature and predictors of elementary teachers’ Mathematical Knowledge for Teaching. Journal for Research in Mathematics Education, 41 (5), 513-545.,(2010)

Hill, H.C. & Shih, J. (2009) Research commentary: Examining the quality of statistical mathematics education research. Journal for Research in Mathematics Education.,(2009)

Hill, H.C. & Ball, D.L. (2009) The curious — and crucial — case of Mathematical Knowledge for Teaching. Phi Delta Kappan , 91, 68-71.,(2009)

Hill, H.C. (2009). Evaluating value-added models: A measurement perspective. Journal of Policy and Management, 28, 702-209.,(2009)

Delaney. S. F., Ball, D. L., Hill, H. C., Schilling, S.G., & Zopf, D. A. (2008). Adapting U.S. measures of “Mathematical Knowledge for Teaching” for use in Ireland. Under review at Journal of Mathematics Teacher Education,11, 171-197..,(2008)

Hill, H.C., Blunk, M. Charalambous, C., Lewis, J., Phelps, G. C. Sleep, L. & Ball, D.L. (2008). Mathematical Knowledge for Teaching and the Mathematical Quality of Instruction: An Exploratory Study. Cognition and Instruction, 26, 430-511.,(2008)

Hill, H.C., Ball, D.L. & Schilling, S.G. (2008) Unpacking “Pedagogical Content Knowledge”: Journal for Research in Mathematics Education,39, 372-400.,(2008)

Schilling, S.G. & Hill, H.C. (2007). Assessing Measures of Mathematical Knowledge for Teaching: A Validity Argument Approach. Measurement: Interdisciplinary Research and Perspectives (5), 2-3, 70-80.,(2007)

Hill, H.C., Ball, D.L., Blunk, M. Goffney, I.M. & Rowan, B. (2007). Validating the ecological assumption: The relationship of measure scores to classroom teaching and student learning. Measurement: Interdisciplinary Research and Perspectives (5), 2-3, 107-117.,(2007)

Hill, H.C., Dean, C. & Goffney, I.M. (2007). Assessing Elemental and Structural Validity: Data from Teachers, Non-teachers, and Mathematicians. Measurement: Interdisciplinary Research and Perspectives (5), 2-3, 81-92.,(2007)

Schilling, S.G., Blunk, M. & Hill, H.C. (2007). Test Validation and the MKT Measures: Generalizations and Conclusions. Measurement: Interdisciplinary Research and Perspectives (5), 2-3, 118-127.,(2007)

Hill, H.C., Ball, D.L., Sleep, L. & Lewis, J.M. (2007) Assessing Teachers’ Mathematical Knowledge: What Knowledge Matters and What Evidence Counts? In F. Lester (Ed.), Handbook for Research on Mathematics Education (2nd ed), p. 111-155. Charlotte, NC: Information Age Publishing.,(2007)

Hill, H.C. (2007). Mathematical knowledge of middle school teachers: Implications for the No Child Left Behind Policy initiative. Educational Evaluation and Policy Analysis (29), 95-114.,(2007)

Hill, H.C. (2007). Teachers’ Ongoing Learning: Evidence from Research and Practice. The Future of Children, 17, 111-128.,(2007)

Hill, H.C. & Lubienski, S.T. (2007) Teachers’ mathematics knowledge for teaching and school context: A study of California teachers. Educational Policy 21(5), 747-768.,(2007)

Learning Mathematics for Teaching. (2006). A Coding rubric for measuring the Quality of Mathematics in Instruction. Ann Arbor, MI: Authors.,(2006)

Hill, H.C. (2006) Language matters: How characteristics of languagecomplicate policy implementation. In M.I. Honig (Ed.), New directions in education policy implementation: Confronting complexity. Albany, NY: SUNY Press.,(2006)

Hill, H.C. (2005). Content across communities: Validating measures of elementary mathematics instruction. Educational Policy 19, 447-475.,(2005)

Hill, H.C., Rowan, B., & Ball, D.L. (2005) Effects of teachers' mathematical knowledge for teaching on student achievement. American Educational Research Journal 42, 371-406.,(2005)

Ball, D.L., Hill, H.C. & Bass, H. (2005) Knowing mathematics for teaching: Who knows mathematics well enough to teach third grade, and how can we decide? American Educator, Fall 2005, 14-22.,(2005)

Hill, H.C., Schilling, S.G., & Ball, D.L. (2004) Developing measures of teachers’ mathematics knowledge for teaching. Elementary School Journal 105, 11-30.,(2004)

Hill, H. C. (2004) Professional development standards and practices in elementary school mathematics. Elementary School Journal 104, 215-31.,(2004)

Hill, H. C. & Ball, D. L. (2004) Learning mathematics for teaching: Results from California’s Mathematics Professional Development Institutes. Journal of Research in Mathematics Education 35, 330-351.,(2004)

Hill, H.C. (2003) Understanding implementation: Street-level bureaucrats’ resources for reform. Journal of Public Administration Research and Theory 13, 265-282.,(2003)

Hill, H .C. (2001) Policy is not enough: Language and the interpretation of state standards. American Educational Research Journal 38, 289-320.,(2001)

Cohen, D.K. & Hill, H.C. (2001) Learning policy: When state education reform works. New Haven, CT: Yale University Press.,(2001)

Cohen D. K. & Hill, H.C. (2000) Instructional policy and classroom performance: The mathematics reform in California. Teachers College Record 102, 296-345.,(2000)