Our Research Questions
The CRM is fundamentally a large scale change management system, operating a multiple levels: system-wide, CMO networks, individual schools. This evaluation set out to test not only the extent to which individual schools implemented their CMOs’ promising models, but also the extent to which the CRM overall was implemented with fidelity to its Theory of Change. This Research Question examines the fidelity of implementation of the CRM overall; Research Question 4 examines fidelity of implementation at the school level.
In its original formulation, the CRM laid out three overarching goals. Each of these goals detailed a set of articulated activities along with designated lead actors to hold those responsibilities. Each goal also has a predefined outcome. On no single goal did NSNO and RSD unilaterally succeed, although we see that partial success was achieved. The table below lays out the goals, the targets and the reality:
|Goal 1: Build the capacity to incubate and expand charter restart operators||Proposed Outcome 1:||Actual Outcome 1:|
|27 schools turned around (19 New Orleans, 8 Tennessee)||25 schools received funds, 21 of which were included in this study. Of those 21, ten schools produced average learning gains for students, five schools did not produce learning gains for students but continue to operate, and six schools closed and/or turned around again.|
|15,281 students served||In these 21 schools, 9127 students were served, but only around 6000 students were attended the successful schools.|
|Permanent capacity including 3 new CMOs established to continue executing turnarounds beyond the life of the i3 grant||Four (rather than 3) new CMOs were incubated.|
|Goal 2: Provide infrastructure to sustain charter restart schools||Proposed Outcome 2:||Actual Outcome 2:|
|RSD and ASD have full-time dedicated personnel and systems in place to monitor charter restart performance, identify schools for turnaround, and build stakeholder engagement||RSD has stabilized its staffing and internal processes, although the return of schools to OPSB oversight raises new questions about systems for oversight and engagement.|
|NSNO oversaw development of annual analysis of school performance.|
|RSD and NSNO addressed significant barriers by implementing unified enrollment system, create community plan for serving SPED students, addressed teacher pipeline shortages, and adopted a common expulsion policy.|
|ASD facilitated service agreements with Shelby County Schools to deliver key operational services to ASD schools.|
|ASD built a school performance framework to set performance expectations, but this framework was rejected by most schools as fatally flawed in its design and too compliance oriented.|
|ASD did not build community-wide support for turnaround efforts, and in fact opposition to ASD mobilized efforts (such as Shelby County’s i-Zone initiative) which undermined ASD’s ability to take over schools for turnaround.|
|Goal 3: Scale the CRM strategy by codifying and replicating the model||Proposed Outcome 3:||Actual Outcome 3:|
|ASD is successfully launched in TN with RSD assistance||ASD faced funding and policy challenges from within TDE and from outside funders.|
|NSNO assists with the launch of a comparable nonprofit entity in TN||No local partner emerged during the study period to authoritatively fill the harbormaster role.|
|Blueprint for the New Orleans CRM is disseminated broadly||ASD has failed to sustain as a robust entity, suffering leadership instability, severe staffing and budget cuts at the end of the study period.|
|The blueprint for the CRM was disseminated widely supported by NSNO’s direct advocacy as well as Education Cities’s support in seeding harbormaster organizations in cities nationwide.|
Overall, the bare minimum requirements for the CRM to function did obtain: after all, CRM schools were indeed selected, funded, and operated. However, creating sufficient conditions for the CRM to operate does not create the necessary conditions for the CRM to succeed. Merely setting up the mechanics of implementation is clearly inadequate, given the lack of positive student impact at half the CRM schools.
A discussion of each CRM goal follows.
Twenty-five schools serving 9184 students received i3 support, falling short of Goal 1 expectations. Further, NSNO and RSD expanded on their original commitment to full school turnaround and allowed fresh start charters to receive i3 funds from the very first selection round. While the administration of the federal SIG grant program provided a regulatory basis to legitimize fresh starts as a school improvement strategy, this decision violated NSNO’s own eligibility criterion that grantees serve all students enrolled in the Closing school being turned around. Further, not every turnaround was a success: by the end of the grant, only half had posted positive results and six were closed or reconstituted due to troubled management and/or poor results.
NSNO, RSD, and ASD were neophytes in large scale change management. None of the three systems-level partners built the processes or infrastructure necessary to fully activate the CRM before funding schools. RSD and NSNO in New Orleans relied heavily on existing professional networks and relationships. While they made efforts to build capacity through Communities of Practice, targeted trainings, and support services offered to the CMOs in the CRM, these took a narrow focus and were time-limited. NSNO did look to build data/research capacity inside its organization by hiring a quantitative analyst in the third year of the CRM evaluation. However, he left in the final year of the study and was not replaced. Larger efforts to address community-wide barriers such as building a stronger teacher corps and constructing a teacher pipeline for NOLA charters were not exclusive to the CRM; their benefits were not universally distributed across the CRM schools.
In Tennessee, the leaders of the ASD were themselves new to the landscape and did not invest in deep learning about the local history and context before taking action. In addition, the Tennessee CRM relied entirely on a single laser-focused leader (Christopher Barbic) who strong-armed the implementation of reforms in alignment with his own vision rather than that of the CRM per se.
In both locations, personal relationships drove organizational practice and informed organizational cultures, retarding the creation of sustainable infrastructure that might outlast the tenure of any particular individual. In New Orleans, we observe an eventual recognition of these infrastructure needs, often in response to external pressures from other mission driven partners (a lawsuit brought by the Southern Poverty Law Center on behalf of underserved SPED students, advocacy from the youth development and juvenile justice communities). In Tennessee, where ASD had no partner organizations to work with in the first place, we see little evidence of a focus on building long term, sustainable infrastructure. ASD created community councils to inform selection processes, but these councils were widely viewed with skepticism by school and community actors. ASD , which accounts in part for the uncertainty of ASD’s sustainability as of this writing.
RSD and ASD experienced significant evolution in personnel and systems, as their focus on operating schools directly in addition to the turnaround work ebbed. In addition, it took both entities time to settle into their irreducible functions of oversight and compliance assurance. Both RSD and ASD shrunk their footprints, and ASD in particular has moved away from maintaining the organizational structure or capacity necessary to support charter restart as a mechanism for school improvement. While this is in part attributable to staffing and budget cuts through the years of the CRM evaluation, we also observed intentional shrinkage as organizational missions shifted away from the CRM in its original formulation.
Perhaps the most publically visible shortcoming within Goal 2 was that of community engagement. Neither RSD nor ASD ever successfully managed stakeholder engagement as a core commitment. In New Orleans, RSD and NSNO both refused to engage communities in authentic decision-making processes, framing “engagement” as community acceptance of decisions that had already been made. Both agencies struggled to retain staff dedicated to interface with parents and community groups, and took superficial approaches to the racial dynamics of school takeover. Tennessee observed these challenges and attempted to offset controversy by seating Community Councils. However, those Councils were viewed by many as highly contrived and convened largely to rubber-stamp ASD’s preconceived course of action. In both Louisiana and Tennessee, we see the responsibility for community engagement shift over the course of the evaluation from a systems-level function to a school-level function, as schools build relationships with their geographic neighbors, families of students, and local neighborhood partners (e.g. businesses and churches). In the final year of the evaluation, we see a single successful community engagement process in New Orleans: the turnaround of Wilson. Managed by RSD’s recently hired Deputy Superintendent of External Affairs, Wilson’s turnaround engaged parents of the closing school, the closing school’s board of directors, and multiple charter operators willing to work with an array of stakeholders in determining which CMO would ultimately turn around Wilson.
Much like Goal 2, we observe mixed results for Goal 3: no comparable nonprofit entity emerged in Tennessee, although the Memphis Educational Fund was founded in 2014 with a mission of playing a harbormaster role for that city (not for all ASD schools statewide). ASD has suffered setbacks within TDE and the state legislature each year as new TDE leadership disfavored school closures and turnarounds as an improvement strategy. Lack of leadership continuity at ASD (due to the extended medical leave of the ASD Superintendent) further diluted ASD’s presence at key policy tables. The full effect has been one of waning support for the ASD. As of this writing, ASD had undergone a recent restructuring (including layoffs). The dissemination of the CRM began early in the i3 initiative – well ahead of confirmation that the model actually works – and has tapered off in the final years of the initiative. For a full exploration of these findings, see the Organizational Capacity report (link). However, the work of Education Cities and the installation of harbormaster-type organizations across the U.S. indicates that at least one core function of the CRM has successfully scaled.
Systems Learning Regarding the Three Goals
The removal of district control from the school system did not obliterate the notion of a functional system per se. Rather, the decentralization of New Orleans schools replaced a single point source management system (i.e. a district) with a multi-point source management structure; and created a set of levers (citywide school choice, CMO- and school-level autonomy, decentralized student supports) to drive school quality in the absence of a single oversight authority. Those levers, coupled with the expectation that the CRM would require a single event of school turnaround rather than an on-going series of decisions and gateways, created externalities which the CRM’s system-level partners either did not anticipate, or were not equipped to resolve.
The CRM Theory of Action presumes decentralization, but we see little evidence in the design or early years of implementation of an understanding by system-level actors of how to calibrate decentralization with necessary oversight or with intra-system knowledge-sharing. Indeed, in a decentralized array of CMOs who compete against each other for students and grants, it is surprising the degree to which the CMO leaders were willing to collaborate, although the threat of stronger sanctions may have played a role. Rather, the externalities inherent in an all charter district were either unanticipated or underestimated as existential threats to the CRM. Systems-level partners missed key opportunities to intervene at the systems level to protect CMOs and schools from exogenous shocks (e.g. facilities moves, dearth of social service supports for students) and to prevent common endogenous shocks (e.g. teacher development, leadership stability).
However, we did observe important instances of learning among system-level actors regarding implementation with fidelity to the CRM Theory of Action. The aforementioned disincentive to enroll highest needs students in New Orleans resulted in the installation of One App in 2012 and later the SPED Initiative and the Differentiated Funding Formula, which equalized the cost burdens across schools and alleviated the disincentive. The dearth of human capital – a likely phenomenon in any urban district – was met with a dedicated effort by NSNO to develop local talent by establishing Relay GSE in New Orleans, supporting the Achievement First fellowship, and procuring/administering TIF and Arnold Foundation funds specific to the development of human capital pipelines and teacher quality improvement. The administrative, financial, and accountability functions necessary to manage large scale funds and system-wide change were nearly all in place by Y4 of the CRM evaluation. In Tennessee, however, little evidence of such learning emerges. In fact, ASD doubles down on a compliance mindset, choosing to focus more on their direct run schools than their CRM charters until the close of the evaluation (at which point ASD barely functioned as a state agency).