Evaluation of Scaling the New Orleans Charter Restart Model:

Our Research Questions

9. How did the CRM give rise to systems-level coordination functions in order to support 50,000 high quality seats?
The CRM was designed to be a lever for universal high quality in all public schools. Did NSNO’s CRM investments result in universal high quality for every public school student in New Orleans?

What were the necessary systems-level functions required to support and sustain the CRM?

The CRM, as formulated for the purposes of the i3 grant, set out to transform 27 schools serving 15,281 students. But Goal 2 of the CRM – the establishment of permanent infrastructure to turn around the lowest performing 5% of schools in perpetuity – speaks to a larger ambition beyond the i3 grant project. The installation of the CRM is not an end in itself but rather a mechanism to create a New Orleans system with universal high quality. The CRM was designed as a mechanism to create 50,000 high quality seats, one for every student in New Orleans based on citywide enrollment projections for 2018.

To achieve such system-wide goals, the CRM (indeed, any large scale change management initiative) necessitates a set of system-wide coordinating functions. While CMOs are positioned in the CRM as the (decentralized) levers for high-quality educational service delivery, there still exists a need for (centralized) strategic, operational, and accountability functions that run above the CMOs. These functions include strategy development for the entire system (including the anticipation of and response to externalities); CMO support; stakeholder engagement; and continuous improvement in pursuit of shared goals across all actors within the CRM. Additionally, the CRM must also provide safeguards against the tendency to reinstitute bureaucracy and over-regulate. This was particularly relevant in Tennessee during the course of our evaluation and is becoming increasingly salient for New Orleans as schools transition from RSD to OPSB oversight in the coming year.

How were necessary systems-level functions managed?

At the outset of the CRM initiative in 2010, systems-level strategic, operational, and accountability functions were not consistently owned by any single entity. Role confusion between NSNO and RSD in the early years of the CRM – particularly in the domains of stakeholder engagement, CMO support, and strategy development – resulted in uneven implementation of some functions, inadequate implementation of others. The expectation that CMOs would “become the system” overestimated capacity at the CMO level and underestimated the degree to which some functions require centralization to ensure system-wide equity (enrollment/expulsion, SPED) or require coordinated solutions to system-wide challenges that impact all actors within the CRM (teacher pipelines).

We observe numerous instances in which CRM strategy was developed opportunistically rather than proactively in the first years of the evaluation. For example, RSD and NSNO consistently missed opportunities to build broad support for the CRM as a cohesive system designed to serve all students. In the early stages of the CRM, NSNO hosted periodic “step-backs” to review the implementation of the CRM and invited RSD to participate; engagement was limited and the practice quickly wore out. Instead of sustaining these strategic step-backs, we observe that NSNO and RSD mobilized narrow coalitions to solve discrete problems such as designing professional development for math teachers or offering CMO leadership training for only the first turnaround leaders. Further, responses to externalities, engagement of stakeholders, and system-level learning often arose from a reactive stance by NSNO and RSD rather than internally generated strategic mobilization. External pressure to make the all-charter system (and, by extension the CRM) more student-centered and community-friendly came from parents and community, from student activists, from local and national media, and in the form of the SPLC SPED lawsuit. These exigencies forced NSNO and RSD to rethink certain parameters and functions of the CRM.

Other functions struggled to find a home at either RSD or NSNO due to limitations in capacity, expertise, or willingness to grapple with difficult issues. For example, NSNO aimed in the early years after Katrina to build systemwide supports by developing a Board Bank. But NSNO could not sustain these functions very long: by the start of the study period, NSNO divested from those activities internally. NSNO developed early in-house professional development offerings, but never had the capacity to meet the degree of need citywide; their offerings always required supplementation from external providers (e.g. national consultants to manage leadership trainings) to begin to address systemwide need. Community engagement at the systems level went under-supported for years, with CMOs and individual schools managing as best they could in their own communities, but with no system-wide strategy to communicate the Theory of Action, emerging benefits, or proposed solutions to challenges faced by families within the CRM. Only after 2012, when Patrick Dobard assumed the superintendency of RSD and Dana Peterson filled the role of Deputy Superintendent for External Affairs did family/community engagement function at the systems level.

Despite these early challenges in coordination of system-level functions, we do observe systems-level learning and stabilization in later years. NSNO and RSD – even when not motivated by internal strategy or explicit commitment to continuous improvement – did rethink and reorganize the management of strategic, operational, and accountability functions for their organizations more generally beginning in Year 3 of the evaluation. NSNO concentrated grant development towards improving teacher quality, winning both federal and philanthropic support for the effort. We see increased professionalization within NSNO and RSD, as well as a greater willingness to support efforts of other organizations in order to accumulate greater systems-level capacity without having to build additional capacities within RSD or NSNO per se. For example, NSNO divested from their governance support work (including the aforementioned Board Bank), but that work eventually found a home in LAPCS, where it currently sustains. Similarly, NSNO scaled back on developing in-house school support offerings, shifting instead to more strategic approach of identifying and recruiting expert national providers (Relay GSE, Achievement First, New York City Charter School Center) for the majority of systemwide technical assistance provision. (Note that this shift coincided with a leadership transition at NSNO, in which NSNO shifted from “calling the shots” to a servant leader stance.)

By Year 4 of the study, we see increased clarity in NSNO’s and RSD’s role definition; more intentional convening and strategic planning around the goal of 50K high quality seats; as well as a commitment to identifying systemwide interventions to remediate proactively the threats posed by externalities. This resulted in a more cohesive CRM ecosystem and an equilibrium between autonomy and centralization. As NSNO and RSD grew into their respective roles, and as various organizations assumed responsibility for core systems-level functions, NSNO also garnered resources, leveraged convening power, and advocated to RSD and other stakeholders to incubate local interventions (SPED, therapeutic center). To date, NSNO’s most visible, highly appreciated, and longstanding direct support provision takes the form of their school reviews; but their role by the end of the study period is best understood as service procurer rather than service provider.

In this way, NSNO became a national leader in the development of the harbormaster organization. The CRM as originally designed distributes systems-level functions – technical assistance, community engagement, intrasystem learning, quality management supports – across NSNO and RSD in New Orleans. But, ultimately, the coordination across these functions, and the monitoring of organizations and efforts to sustain these functions, has fallen to NSNO as a harbormaster: the single, central actor responsible for maintaining the tide that lifts all CRM boats.

Replication of this harbormaster function in Tennessee never fully materialized. In the design stage, the Tennessee Charter School Center was positioned to play the role of harbormaster, with dedicated technical assistance from NSNO. However, TCSC’s mission and strategy shifted away from school/CMO support before ASD had received full legislative authorization, and NSNO’s technical assistance was not realized, leaving ASD without a viable nonprofit partner. As such, ASD was from its inception left with two sets of responsibilities to manage: that of harbormaster and that of authorizer.

This mixed role never cohered: ASD struggled to make strategic decisions about how to prioritize and activate the sometimes conflicting responsibilities of support provider and oversight agency. Schools expressed consistent confusion and frustration regarding ASD’s shifting expectations, regulations, and unkept promises of support. ASD eventually abandoned technical assistance functions and reverted to a compliance management stance, leaving harbormaster functions wholly unsupported in Tennessee.

How did the development of a harbormaster function impact the larger CRM goal of 50K High Quality Seats?

As noted above, the CRM originally proposed the turnaround of 27 schools in the eventual service of 15,281 students, but Goal 2 of the CRM indicates a much larger commitment: the turnaround of the lowest performing 5% of schools in perpetuity toward the end of high quality public education for every student in New Orleans. The CRM represents a multiplier mechanism whereby NSNO achieves its mission: to leverage the turnaround of 19 schools (as proposed; 13 schools were implemented) in order to deliver on the promise of excellent public schools for every child in New Orleans or, put another way, to create and sustain 50,000 high quality public school seats throughout the New Orleans system of schools.

This evaluation provides a point-in-time test of whether NSNO’s investments are likely to result in 50K high quality seats. To be clear: the CRM as originally formulated within the aegis of the i3 initiative did not set out to create 50K high quality seats per se; indeed, the CRM’s target of 27 schools would not have yielded 50K seats even if all 27 had been in New Orleans and had achieved positive student impact. Further, NSNO’s goal of 50K high quality seats is orthogonal to their work in Tennessee, which aimed to test scalability of the CRM but was not intended to count toward the 50K seats in New Orleans. ASD’s own goal was to move the performance of their CRM investments – along with the schools they ran directly – into the top 25 percent of the state. As such, we test the claim of 50K high quality seats in order to contextualize the i3 grant within the larger set of change management initiatives in which NSNO engaged during the study period.

In terms of student impact, NSNO’s investments did not result in the desired increase in high quality public school seats. NSNO did not support turnaround in the full complement of 27 schools promised in the original grant proposal; in fact, NSNO could not identify enough fully capable CMOs with capacity to effectively turn around all 27 schools as promised. Further, of the 21 schools included in this evaluation, six not only fall short of student learning goals but ultimately exit the field (click here to learn more about these six schools). Only seven of 13 New Orleans schools and two of eight Tennessee schools show positive significant learning gains for students. Most importantly, only two New Orleans schools achieve the CRM target of top 33% in reading and only three New Orleans schools achieved this target in math. In Tennessee, no schools – even those posting some positive impact for students – reached the target of top 25%.

While NSNO and RSD have not created 50K high quality seats, we do see evidence of system-level learning regarding levers for systemwide quality improvement. NSNO hoped to encourage full school turnaround, but found that full school turnaround presents additional complexities and challenges to student learning as compared to a fresh start school improvement strategy (click here). Externalities that NSNO, RSD, and ASD failed to anticipate created both threats and opportunities for their organizations and for the CRM overall. These externalities, particularly the disincentive for individual charter schools to educate highest needs students (click here), would represent a challenge in any portfolio or all-charter system, but were intensified by a restart ecosystem in which the incentive to marginalize hardest-to-educate students is exacerbated by the unmatched depth of difficulty of achieving student success in a high stakes turnaround environment. In the face of these externalities, NSNO was slow to adopt a strategic, continuous improvement stance. But NSNO did come to recognize the impacts of those externalities. Further, NSNO evolved in its understanding of how best to respond: by building internal capacity, cajoling partners, identifying outside experts, holding others accountable. In this way, NSNO assumed a cohesive set of harbormaster functions, which in turn allowed for NSNO to manage continuous improvement across systems actors without having to provide singlehandedly a never-ending array of individual supports.

The “innovation” of investing in CMOs and systems-level actors did not achieve the grant target of 15,281 students served in high quality schools, let alone create 50K high quality seats. Yet, NSNO successfully leveraged the i3 award to garner additional attention and support (both dollars and expertise). That leverage in turn provided the time and resources required for NSNO to establish not only the CRM ecosystem, but to establish its own place and role within that ecosystem.

How can the development of the harbormaster function in the CRM inform other districts and cities?

In the early years of the CRM, NSNO was more reactive than proactive and more responsive than strategic in the face of challenges to the CRM. Yet over the years of this evaluation, NSNO developed an increasingly sophisticated understanding of systems-level needs, effective ways to meet those needs, and the capacities NSNO can hold versus those better delegated in order to create an ecosystem that supports continuous improvement for all actors within the CRM. Stability in NSNO leadership had a strong contribution to make in this evolution; a shared CEO arrangement allowed for greater expertise and experience in the role and successful collaboration between the two CEOs deepened the focus, motivation and effectiveness of their efforts.

As this understanding developed, NSNO worked to communicate its learning in alignment with the CRM’s Goal 3. NSNO continues to play a strong leadership role in the national landscape of neophyte harbormasters which in turn reinforces its capacities to serve the New Orleans community. In this way, similar organizations around the country have benefited from the knowledge that systems-level strategic, operational, and accountability activities require a designated home and a dedicated actor who can maintain focus on the functioning of the system, rather than solving the problems of every organization within the system. We recognize that NSNO came into the harbormaster role through an exceptional set of circumstance in New Orleans. But the findings of this evaluation suggest that the harbormaster role is necessary to ensuring that the CRM – or local replications thereof – can take hold.

This is not to argue that the role of harbormaster is sufficient to guarantee success. Student impact findings for Louisiana indicate that this is sadly not the case. But our observations of NSNO’s burgeoning organizational development in New Orleans, coupled with the devolution of ASD’s organizational effectiveness in Tennessee, indicate that harbormaster functions are crucial for any instantiation of the CRM to obtain.