Calculating the Return on Investment in Summer Learning

National Summer Learning Association Interview with Linda Goetze, Senior Researcher, Center for Education Policy Research University of New Mexico

For over 10 years, New Mexico’s K-3 Plus program has been extending the school year for at-risk early elementary school children, with growth of the program fueled by positive findings from pilot studies. Minimum program funding is defined by law, but cost per student varies widely across jurisdictions. The National Summer Learning Association (NSLA) spoke with researcher Linda Goetze at the University of New Mexico to understand the intersection of policy, funding, and return on summer reading investments.

What prompted you to study this issue?

I have always wanted to work to eliminate poverty and I liked economics as a way to understand tradeoffs between equity and efficiency. I’m interested in how evidence fuels the evolution of policy. Preschool has that evidence base—studies show that it returns between $7 and $16 for every dollar invested. This has transformed the way that policymakers think about age at which public education should begin—it used to be first grade, then kinder half day, then full day and now preschool at age 4 or even 3. Similarly, the tradition has been a 180-day school year with summers off to work on the farm, do recreational activities, or take vacation. Today that is changing as we learn more about the gap that occurs in key math and reading skills during the summer. Evaluating the cost, effects and benefits of summer programs is a bit different than doing the same analysis during the school year.

How did New Mexico determine the funding formula for its summer program, and how has the funding evolved over the life of the program?

For the first four years of the K-3 Plus pilot, programs were funded based on the number of expected students, rather than actual attendees, giving districts vital time and budget information for planning, but creating inefficiencies because of lower attendance in the voluntary program. As a response, in 2010 and 2011, funding was reduced to zero for students that attended fewer than 18 days of the summer. At the same time that total K-3 Plus funding dropped, student enrollment significantly increased. This left districts to cover the shortfall by borrowing from school-year operational funds. Several school districts that had been long-term K-3 Plus participants withdrew from the program in these years. In 2011, the findings that K-3 Plus associated with positive student outcomes in literacy and numeracy from the New Mexico Standards Based Assessment data for third grade, resulted in a new legislated funding formula that assigns a percent of the school year unit or “per pupil” value to support K-3 Plus.

What challenges remain in getting the funding formula for summer learning right?

Current law specifies that funding be at “no less than 30 percent of the unit value per student…” Schools and school districts are now participating in the program at record rates but there continue to be barriers to participation, especially in rural schools. Statewide only 25 to 35 percent of eligible students participate, and the cost per student increases significantly if some classrooms are under-enrolled. Recruitment and teaching strategies are more complex when there are English-language learning, dual language and monolingual classrooms — a frequent occurrence in New Mexico schools. While the current formula is an improvement over previous versions, it may not sufficiently adjust for rural and at-risk variations that exist in K-3 Plus participating schools and districts.

The real challenge comes from not knowing how much funding will be available before the summer starts. Legislation requires that applications for funding be submitted by March 15th and that funding be awarded by April 15th. However, that isn’t much time to recruit teachers, students and families and order supplies since some programs start K-3 Plus in June. New funds are not available until the new fiscal year starts on July 1st, so carryover funds have become a critical source of continuity to the program.

What are the typical cost centers for the K-3 summer program?

The most common cost centers for operating the summer program are teacher and other staff salaries, curriculum and supplies, technology for instruction and assessment, and transportation. Marginal costs of facilities and year-round administrators are trickier to factor in. There are real costs associated with operating buildings that would otherwise be closed for the summer.

How has the investment in summer learning improved student achievement?

From 2010 to 2012, researchers at Utah State University received an Investing in Innovation (i3) validation grant from the U.S. Department of Education to rigorously evaluate the K-3 Plus program. Preliminary outcome data suggest that students in K-3 Plus enter kindergarten with higher scores in reading, writing and numeracy than the students that were randomly assigned to the control group. Comprehensive cost data are being collected as part of the i3 grant to inform future funding of the program including replication and scale-up and to do a cost-effectiveness analysis by bringing together the cost and outcome data.

How do you relate cost and achievement to each other to arrive at cost-effectiveness?

The average cost per child of the K-3 Plus programs studied, accounting for all direct and indirect costs, was $1,213. For kindergarten students, this investment results in effect sizes of .352 in reading improvement, .365 in writing, and .219 in math. Compare these to a study that found effect sizes of .18 and .10 for school-day interventions (Lipsey et al, 2012). The summer program had a larger “per day” effect, and thus a greater return on the investment.

What should policymakers take away from this research?

Sustainable funding formulas that can adapt to changing conditions are important, since legislative appropriations do not always keep pace with demand. Over the first two years of the K-3 Plus program the appropriation stayed constant, but enrollment nearly tripled, leaving districts to cover the shortfall from their operating budgets. The policy issue that develops is whether a minimum class size must be met to operate a program or will the state funding formula support, at a much higher per student funding rate, very small classrooms during the summer.

Summer learning is proven to be a cost-effective strategy for boosting achievement of young students. We need to build a menu of summer learning that allows policymakers to select summer programs that are within their budget and that achieve targeted outcomes for specific students that they serve. This includes a menu for high risk and low risk, English-language learners, special education, and rural and urban students across various ages and grades. The menu should be specific and show the cost and outcomes or benefits achieved for specific students.

What are the challenges in determining funding levels for a statewide program?

Districts allocate costs differently in their annual budget, making cross-district comparison tricky. Our research had to account for this with estimations. Furthermore, there are very diverse rural and urban school districts throughout the state of New Mexico and transportation costs vary widely. Expenditure analysis suggest the program can be very costly in smaller classrooms and schools and yet students in rural areas may be the students that benefit most from having a stimulating summer educational program available to them.

What are the next steps in this research? 

There is stronger evidence than ever that students need to be actively engaged in learning during the summer, whether it is checking out books at the local library, going to summer camps or in a full-day, school-based program. Without these stimulating summer learning activities, students regress from where they were in the spring, and teachers have to waste time re-teaching material in the fall. Disparities in these activities contribute significantly to the achievement gap. There is still much we need to learn about what works best for which students at what age to achieve which outcomes and for what cost.

To continue to build the evidence base about what works for which students, requires methodologically rigorous studies that deliver different types of services to students and their families. A critical element to cost effectiveness is designing programs to increase participation — creating greater economies of scale — and making programs desirable for families of very different cultures and circumstances. Families may have different goals for their children, and students have different needs to be met during the summer. We need to understand these differences so that we design programs that families will choose for their children and that students will want for themselves.

Finally summer learning programs need the longitudinal cost-benefit data that is so crucial to national scale-up efforts. This research starts to show the long-term savings to states, participants, and taxpayers from investments in education programs that are based on what works for students.

References: Goetze, L. D., Cann, D., Tran, Hanh, T., and Vargas, Yamil, StartSmart K-3 Plus Making Summer Count by Increasing Kindergarten Readiness. Paper presented to the American Educational Research Association, 2013. Goetze, L.D., & Hughes, S. (2013, March). Innovations in Education: Financing a Voluntary Extended School Year Program. Paper presented at the American Educational Finance Association Meeting, New Orleans, LA