Our Research Questions

The CRM is fundamentally a large scale change management system, operating a multiple levels: system-wide, CMO networks, individual schools. This evaluation set out to test not only the extent to which individual schools implemented their CMOs’ promising models, but also the extent to which the CRM overall was implemented with fidelity to its Theory of Change. This Research Question examines the fidelity of implementation of the CRM overall; Research Question 4 examines fidelity of implementation at the school level.

In its original formulation, the CRM laid out three overarching goals. Each of these goals detailed a set of articulated activities along with designated lead actors to hold those responsibilities. Each goal also has a predefined outcome. On no single goal did NSNO and RSD unilaterally succeed, although we see that partial success was achieved. The table below lays out the goals, the targets and the reality:

Overall, the bare minimum requirements for the CRM to function did obtain: after all, CRM schools were indeed selected, funded, and operated. However, creating sufficient conditions for the CRM to operate does not create the necessary conditions for the CRM to succeed. Merely setting up the mechanics of implementation is clearly inadequate, given the lack of positive student impact at half the CRM schools.

A discussion of each CRM goal follows.

NSNO, RSD, and ASD were neophytes in large scale change management. None of the three systems-level partners built the processes or infrastructure necessary to fully activate the CRM before funding schools. RSD and NSNO in New Orleans relied heavily on existing professional networks and relationships. While they made efforts to build capacity through Communities of Practice, targeted trainings, and support services offered to the CMOs in the CRM, these took a narrow focus and were time-limited. NSNO did look to build data/research capacity inside its organization by hiring a quantitative analyst in the third year of the CRM evaluation. However, he left in the final year of the study and was not replaced. Larger efforts to address community-wide barriers such as building a stronger teacher corps and constructing a teacher pipeline for NOLA charters were not exclusive to the CRM; their benefits were not universally distributed across the CRM schools.

In Tennessee, the leaders of the ASD were themselves new to the landscape and did not invest in deep learning about the local history and context before taking action. In addition, the Tennessee CRM relied entirely on a single laser-focused leader (Christopher Barbic) who strong-armed the implementation of reforms in alignment with his own vision rather than that of the CRM per se.

In both locations, personal relationships drove organizational practice and informed organizational cultures, retarding the creation of sustainable infrastructure that might outlast the tenure of any particular individual. In New Orleans, we observe an eventual recognition of these infrastructure needs, often in response to external pressures from other mission driven partners (a lawsuit brought by the Southern Poverty Law Center on behalf of underserved SPED students, advocacy from the youth development and juvenile justice communities). In Tennessee, where ASD had no partner organizations to work with in the first place, we see little evidence of a focus on building long term, sustainable infrastructure. ASD created community councils to inform selection processes, but these councils were widely viewed with skepticism by school and community actors. ASD , which accounts in part for the uncertainty of ASD’s sustainability as of this writing.

RSD and ASD experienced significant evolution in personnel and systems, as their focus on operating schools directly in addition to the turnaround work ebbed. In addition, it took both entities time to settle into their irreducible functions of oversight and compliance assurance. Both RSD and ASD shrunk their footprints, and ASD in particular has moved away from maintaining the organizational structure or capacity necessary to support charter restart as a mechanism for school improvement. While this is in part attributable to staffing and budget cuts through the years of the CRM evaluation, we also observed intentional shrinkage as organizational missions shifted away from the CRM in its original formulation.

Perhaps the most publically visible shortcoming within Goal 2 was that of community engagement. Neither RSD nor ASD ever successfully managed stakeholder engagement as a core commitment. In New Orleans, RSD and NSNO both refused to engage communities in authentic decision-making processes, framing “engagement” as community acceptance of decisions that had already been made. Both agencies struggled to retain staff dedicated to interface with parents and community groups, and took superficial approaches to the racial dynamics of school takeover. Tennessee observed these challenges and attempted to offset controversy by seating Community Councils. However, those Councils were viewed by many as highly contrived and convened largely to rubber-stamp ASD’s preconceived course of action. In both Louisiana and Tennessee, we see the responsibility for community engagement shift over the course of the evaluation from a systems-level function to a school-level function, as schools build relationships with their geographic neighbors, families of students, and local neighborhood partners (e.g. businesses and churches). In the final year of the evaluation, we see a single successful community engagement process in New Orleans: the turnaround of Wilson. Managed by RSD’s recently hired Deputy Superintendent of External Affairs, Wilson’s turnaround engaged parents of the closing school, the closing school’s board of directors, and multiple charter operators willing to work with an array of stakeholders in determining which CMO would ultimately turn around Wilson.

The removal of district control from the school system did not obliterate the notion of a functional system per se. Rather, the decentralization of New Orleans schools replaced a single point source management system (i.e. a district) with a multi-point source management structure; and created a set of levers (citywide school choice, CMO- and school-level autonomy, decentralized student supports) to drive school quality in the absence of a single oversight authority. Those levers, coupled with the expectation that the CRM would require a single event of school turnaround rather than an on-going series of decisions and gateways, created externalities which the CRM’s system-level partners either did not anticipate, or were not equipped to resolve.

The CRM Theory of Action presumes decentralization, but we see little evidence in the design or early years of implementation of an understanding by system-level actors of how to calibrate decentralization with necessary oversight or with intra-system knowledge-sharing. Indeed, in a decentralized array of CMOs who compete against each other for students and grants, it is surprising the degree to which the CMO leaders were willing to collaborate, although the threat of stronger sanctions may have played a role. Rather, the externalities inherent in an all charter district were either unanticipated or underestimated as existential threats to the CRM. Systems-level partners missed key opportunities to intervene at the systems level to protect CMOs and schools from exogenous shocks (e.g. facilities moves, dearth of social service supports for students) and to prevent common endogenous shocks (e.g. teacher development, leadership stability).

However, we did observe important instances of learning among system-level actors regarding implementation with fidelity to the CRM Theory of Action. The aforementioned disincentive to enroll highest needs students in New Orleans resulted in the installation of One App in 2012 and later the SPED Initiative and the Differentiated Funding Formula, which equalized the cost burdens across schools and alleviated the disincentive. The dearth of human capital – a likely phenomenon in any urban district – was met with a dedicated effort by NSNO to develop local talent by establishing Relay GSE in New Orleans, supporting the Achievement First fellowship, and procuring/administering TIF and Arnold Foundation funds specific to the development of human capital pipelines and teacher quality improvement. The administrative, financial, and accountability functions necessary to manage large scale funds and system-wide change were nearly all in place by Y4 of the CRM evaluation. In Tennessee, however, little evidence of such learning emerges. In fact, ASD doubles down on a compliance mindset, choosing to focus more on their direct run schools than their CRM charters until the close of the evaluation (at which point ASD barely functioned as a state agency).