Our Research Questions
This evaluation tests the Charter Restart Model’s success in two domains. First, did the CRM adhere to its Theory of Action? That is to say, did the CRM succeed in its installation and operation? Second, did the CRM succeed in achieving its aims? That is, did the CRM result in improved learning outcomes for students?
Did the CRM succeed in its installation and operation?
In New Orleans, the legislative and regulatory prerequisites for the installation of the CRM were either in place at the start of the study period or built during the course of the evaluation. Independent of the CRM, RSD officials hold the option of taking over failing schools. In addition, procedures for school turnaround pre-dated the CRM, consisting of awarding schools to willing operators. These foundations are essential for the CRM to function but not sufficient conditions for the success of the CRM. Comparison with Tennessee, where ASD never fully assumed these functions, reinforces that these prerequisites are necessary for a CRM to operate.
The original proposal to the US Department of Education’s Investing in Innovation grant program was organized around three goals. These provide a useful template for assessing the degree to which the CRM Theory of Action was adopted and operated.
Goal #1: Turn around 27 schools by recruiting high performing CMO operators and transferring the schools to them to operate.
The Theory of Action taps high-performing CMOs to turn around schools that were taken into the RSD. Recruitment of successful CMO operators required a selection process and a willing supply of turnaround operators. As discussed in Q1, a selection process was designed, including screening criteria for aspirants, an application and interview protocols, and a scoring rubric to ensure fair assessment of candidate operators. The design of this process could be considered a success that aligned with the Theory of Action.
The implementation of the selection process experienced significant detours which compromised the Theory of Action. These deviations are responsible, in part, for the differences in organizational functioning discerned by the Performance Management Organization rubric discussed in Q6 and consequently the number of schools with poor impacts on student academic progress, discussed in Q8. In addition, in New Orleans, it was difficult to find the intended number of CMOs willing to take on school turnarounds, so Goal #1 was only partially realized.
Goal #2: Develop a permanent infrastructure to continuously improve the quality of schools in New Orleans.
To support the Theory of Action, NSNO and RSD were expected to expand the capacity of the CMO community to undertake school turnarounds. For reasons further delineated in the Organizational Capacity Study Final Report (click here), NSNO’s efforts to incubate three new CMOs in New Orleans fell short of the intended mark. In addition, fewer local CMOs stepped up. So the ready pipeline called for in the Theory of Action did not materialize.
In addition, the CRM requires permanent access to timely and accurate information on all schools in the community. For the duration of the grant, CREDO agreed to complete an annual analysis of school performance for all New Orleans public schools. (It is telling that the RSD had difficulty getting the necessary information on its own.) The delivery of the annual school performance report was hindered at several times by changes in law at the Louisiana state legislature that diminished the timeliness and availability of necessary data to researchers, especially those not located in the state. Neither NSNO nor RSD could prevail to change the situation. Legislative change is expected to rectify the problem in 2018, but the “permanent” part of routine information has not been developed.
Third, the Theory of Action sought to cultivate in the general public and among key local officials a sustained commitment to the lengthy process of improving schools for New Orleans’ students. Significant efforts were undertaken by RSD and NSNO, and improved results were seen at the end of the grant period. Public opinion of the importance of good schools and support for charter schools in general showed important gains. These accomplishments were overshadowed by the end of the grant period by a growing public sensibility that RSD should return any school performing above the level of failure to Orleans Parish School Board. Concerns about local control and elected governance eclipsed the longer arc envisioned in Goal 2.
The final area called for by the Theory of Action related to Goal #2 was systems-level proactivity to ensure that the CMOs and their schools presented as few barriers as possible. Here, NSNO and RSD proved nimble and adaptive as they responded to problems encountered by the CMOs. A unified enrollment system, and uniform policy for handling expulsions and a comprehensive community-wide approach to better serving students with special needs all occurred during the grant period. NSNO and RSD delivered solutions that permanently improved access and equity for all students in New Orleans.
Goal #3: Scale the CRM to prove its greater viability.
Goal #3 is not directly reflected in the Theory of Action; rather, it presumes that the Theory of Action is transferrable to other communities. Transfer of the CRM to Tennessee was intended as the primary test of this scalability. Other less comprehensive activities included the provision of technical assistance to other interested communities about the CRM and the production of documentation about the CRM for national dissemination.
In the early years of the CRM, a regional think tank produced New Orleans-Style Education Reform (link), describing the regulatory and legislative groundwork needed to consider launching a CRM in other parts of the country, the imperative of a strong human capital pipeline, and a charter school incubation strategy. None of these are specific strategies within the CRM, but the publication served as a precis for many reform activists.
Consultations about the CRM and related efforts were extensive in the second and third year of the grant period. And these efforts contributed to the formation of “recovery districts” in other states.
But the most crucial test of the CRM’s scalability was the adoption of the CRM by the Achievement School District in Tennessee. Since policy and educational context will always shape the way a program takes shape in a particular place, an exact replica of the CRM would not be expected. However, the basic features of the Theory of Action are needed to declare the scaling effort a success.
Full analysis of the scaling of the CRM can be found in the Organizational Capacity Study Final Report (link to download). In brief, the ASD adopted the turnaround operator selection process and oversaw the turnaround of eight schools. They successfully recruited national CMOs to come to Tennessee by tapping the ASD’s other authority: they were empowered as a charter school authorizer to award charters for multiple schools, conditioned on performance. In so doing, they tried to overcome the financial uncertainly for national operators of entering a new region. Beyond these Goal#1 activities, the ASD pursued very little in the area of Goal #2, and certainly did not try to address systems-level issues that slowed the success of the turnaround schools. Their role as authorizer compromised their ability to work as full partners with the schools, who feared that transparency would negatively affect their future. The lack of a second partner also sacrificed the expansion of bandwidth, talent, thought partnership and advocacy. (Two local reform advocacy organizations began after the grant concluded.) Finally, no Tennessee CRM schools achieved performance targets, and only two created any positive impact for student learning. As such, the scaling of the CRM in Tennessee can be only partially credited.
However, the failure to develop the core components of a CRM in Tennessee is partially counterbalanced by the development of NSNO-like harbormasters in other cities around the United States. With support from Education Cities, NSNO has become a leading example of these new organizations. Through the CRM, NSNO successfully illuminated the need for a harbormaster-type entity, the roles and responsibilities held by such entities, and the ways in which such entities can evolve to fill the gaps and support systems-level functions in choice-friendly districts. This represents an unqualified success of Goal 3, as well as a partial success of Goal 2 as NSNO’s internal capacity-building represents strengthened infrastructure to support the CRM or other school improvement efforts going forward.
If this evaluation identifies a crucial fatal flaw in the implementation and operation of the CRM it is this: the CRM cannot succeed if it is operates turnarounds as a single-period solution. To be clear, the CRM had a longer time horizon in mind by virtue of its formulation of Goal 2 expecting permanent systems-level capacity to support the practice of school turnaround in perpetuity. These included promoting a ready supply of CMOs willing to undertake school turnarounds, permanent access to timely and accurate performance data on schools and an activated community to demand and support the drive to better quality. The thinking was that continuous attention to the “bottom 5 percent” would lead to a virtuous upward cycle of continuous improvement.
But the CRM Theory of Action does tacitly support a single period of intervention: identify a strong operator, assign them a closing school, and turnaround follows. The CRM Theory of Action suggests that the ecosystem stabilizes as strong CMOs replace weak schools, but this presumes CMOs operate in a steady state – they come in strong, they stay strong, they absorb shocks (both endogenous, like leadership turnover; and exogenous, like pressure to accelerate phase-in or demographic shifts impacting enrollment) with no negative impact on their functioning or, importantly, their schools’ functioning.
By presuming this steady state, the CRM creates inherent vulnerability to exogenous and endogenous shocks. That is to say: the environment will never not be dynamic, organizations (CMOs and schools) will never not be dynamic, and as such schools will never be fully inoculated against shocks. This is not to argue that the CRM cannot presume stability, but stability does not preclude dynamism. Rather, stability implies an equilibrium between dynamism and capacity to respond effectively to dynamism.
An ideal CRM does not treat the experience of turning around a school a one-period solution, but an n-period solution. The CRM as designed entails a single inflection point from closing to high performing school. But in order to scale, the CRM must enable an n-period approach in which selection is but the first time point. This evaluation indicates that some schools will not succeed after a single intervention and instead require multiple turnarounds in order to achieve positive learning outcomes for students. As such, the mechanisms for intervention – takeover authority, systems-level supports to CMOs and schools, multi-level accountability levers – must be viable and active within the CRM in perpetuity.
Did the CRM result in improved learning outcomes for students?
Ultimately, the CRM succeeds or fails on its ability to drive positive learning outcomes for students. The CRM systems-level partners promised to turn around 27 schools, serving 15,281 students, from lowest five percent to the top 25% in Tennessee, top third in New Orleans.
By this standard, the CRM only partially achieves its goals. Only 25 schools (21 included in the evaluation) received i3 funds for CRM turnaround. By 2015-16, these 25 schools were serving 9184 students. Total capacity might have been higher by the end of the study period, but the decision to allow fresh start schools participate in the CRM means that not all CRM schools achieved full capacity in the course of the study period.
In New Orleans, 5917 students were being served by the end of the study period in all CRM schools. But, only seven of the 13 New Orleans CRM schools demonstrate any positive student impact, and only five CRM schools achieved top-third status on either reading (Einstein, KIPP Believe Primary) or math (Carver Collegiate, Cohen, Tubman) within the New Orleans landscape. No New Orleans CRM school achieved top-third status in both academic domains. In Tennessee, 3267 students attended CRM schools by the end of the study period. Only two of those schools demonstrate any positive student impact and none achieved top 25%.
While no schools achieved top 25/33 percent status statewide, New Orleans saw the aforementioned five schools achieve top 33% status locally in either reading or math. Additionally, the evaluation finds that nine schools across the two states achieved positive learning gains for students to at least some degree. By this more generous standard of accounting, the CRM still does not meet its target of 15,281 students served, but we can see that 5069 students attended schools with positive learning outcomes.
|Total Students Served||Students served in schools achieving top 33/25% locally||Students serve in schools achieving top 33/25% statewide|
|Fresh Start Schools||4104||1301||0|
|Full School Turnarounds||5080||2139||0|
Given that the CRM did not fully achieve its original goals, the evaluation then considers whether CRM schools achieved positive student impact relative to the schools they replaced. Here, we have data for 19 of 21 schools (two schools lacked tested grades). Of these 19, we find that 13 outperformed the Closing schools they replaced in at least one subject. Ten of these 13 schools were in New Orleans; three were in Tennessee. This is perhaps a pyrrhic victory: NSNO and partners set out to avoid precisely the trap of revising expectations downward from “high quality schools” in the absolute to a relative standard of “marginally better than the closing schools.” By allowing this slippage in the standard to which schools are held – from selection onward – we see exactly the results one might expect in terms of inconsistent positive effects for students.
Ultimately, then, this evaluation finds that from the original investment of federal and philanthropic funds, the strongest return is that of NSNO’s development as a harbormaster. While the CRM was neither implemented with fidelity to its Theory of Action nor fully successful in achieving its student impact goals, it has resulted in the establishment of permanent infrastructure that is positioned to support a CRM or otherwise similar all-charter district moving forward.
Note, though, that any CRM is vulnerable to a dynamic policy environment if it does not have enough runway to achieve long term impact. In the case of this evaluation, we extended the study period from five to seven years in order to capture positive growth that accumulates in schools in their third and fourth years of operation. Note also that the CRM in New Orleans and Tennessee did not adhere perfectly to the Theory of Action as designed. Some shifts in implementation represented non-strategic reactions to exogenous shocks, while other changes represented intentional and strategic course corrections (indeed, the Implementation study writ large examines this very issue). But, regardless of the reasons behind these changes, this evaluation tests what actually happened in New Orleans and Tennessee. It cannot speak to the ways in which results might have differed in a more fidelitous implementation.
As we look to the future, NSNO and the New Orleans CMOs are preparing for the return to a unified system under the authorization of OPSB. The impending return of schools to OPSB raises questions about the sustainability of the very mechanisms that did show promise or success in the CRM, which in turn may threaten systemwide gains observed thus far. In Tennessee, ASD has seen its authority eroded from within the Tennessee Department of Education and by the state legislature. In both cases, new existential threats to the CRM are beginning to arise. While CMOs in New Orleans express guarded optimism regarding the return to local control, NSNO and RSD arrive at the termination of the i3 evaluation period preparing to re-envision yet again the pathway to creating 50K high quality seats.