Evaluation of Scaling the New Orleans Charter Restart Model:

Our Research Questions

11. What are the implications of Scaling the New Orleans Charter Restart Model going forward?
What implications for policy and practice arise from the findings regarding the CRM?

At the conclusion of this evaluation of Scaling the New Orleans Charter Restart Model, we find NSNO occupying an authoritative role as harbormaster, while RSD shrinks its New Orleans footprint in anticipation of schools currently under its aegis returning to the Orleans Parish School Board (OPSB), the local education agency in New Orleans. We find a vastly reduced ASD, which is focused on its direct run schools but no longer showing appetite for charter turnaround. NSNO and RSD invested in thirteen New Orleans schools and twelve Tennessee schools (eight of which are included in this study). This falls short on both number of schools and number of seats the CRM had proposed, although the CRM did incubate four new CMOs. Of these, six had such poor early experience that they were closed, reconstituted, or dropped from the CRM program. Overall, the CRM schools have improved relative to their Closing schools, but very few New Orleans schools (and no Tennessee schools) achieved the benchmark for success envisioned by the CRM: top 33% of performers in New Orleans, top 25% of performers in Memphis/Nashville.

Ultimately the CRM suffered for discrete failures within its design as well as numerous failures of implementation. What does this evaluation tell us about how to move forward in light of the CRM experience? Where does the CRM go from here? How promising is it for other parts of the country?

Selection

The CRM found itself at a disadvantage from its earliest days due to weaknesses in the selection process. The CRM as designed requires a deep bench of potential operators who not only meet the quality thresholds for CRM consideration but who also commit specifically to turnaround work. Even high performing CMOs may need to be shepherded into a CRM if they have little experience with or desire for the extremely difficult work of turnaround. In both New Orleans and Tennessee, we see challenges of recruiting a critical mass of high quality operators to turn around as many chronically failing schools as existed in those locales. This indicates that districts or regions considering adoption of the CRM must first test their landscape of charter operators: are there enough high performing operators to do turnaround, and do it well, to suggest that the CRM can succeed?

The CRM also requires a set of objective, measurable selection criteria. These criteria must include a consideration of leadership at both the CMO and school levels: leadership in a CRM ecosystem is multi-tiered, and strong leadership at one level cannot compensate for weak leadership at another. Additionally, operators need deep knowledge of the grades they will serve: CMOs pursuing a K12 pipeline by expanding up the gradespan from elementary or down the gradespan from high school did not, in this evaluation, produce much in the way of successful results.

Pre-existing or robust CMO infrastructure was less important to CRM school success than a team of implementers fully prepared to tackle the challenges of turnaround. Local CMOs were on average better equipped than non-local CMOs, but the presence of robust CMO infrastructure did not guarantee success, nor did its absence presage failure. In fact, the two full school turnarounds that posted positive impact for students – Tubman and Einstein – lacked CMO infrastructure prior to joining the CRM. They did, however, have a full commitment to the work of turnaround, and to doing that work specifically in their school’s neighborhoods and communities.

The CRM selection criteria specified that Type 2, 3, or 4 operators (those that were already running at least one school) must demonstrate a .1 effect size in student growth for consideration. But little distinction was made during the selection process between networks with a .1 effect size across multiple schools versus networks with a single school that hit the .1 threshold. In reality, this created a more fluid requirement than originally intended, since CMOs could (and did) receive i3 funds based on a single school within their network which met the effect size criterion even if their other schools did not.

Further, even those that met the effect size threshold found themselves sorely taxed by the work of turning around a chronically failing school. For Type 2 applicants, building CMO infrastructure while simultaneously managing a CRM turnaround school and an existing school undergoing a leadership transition (as the Flagship principal transitioned to the CEO position) too often proved detrimental to both the CRM school and its network. For Type 3 and Type 4 applicants, the introduction of a chronically failing turnaround school stretched their networks, not necessarily fatally, but always enough to impact both the resource-intensive CRM school as well as other schools reliant on network wide resources disproportionately mobilized to stabilize the CRM school.

Downgrading or diluting the existing selection criteria represents a failure of implementation of the CRM. We also find a failure of design in the selection criteria: too little attention was paid to a proven commitment to continuous improvement. The CRM requires that all actors (system-level partners, CMOs, schools) approach the work as an n-period solution with the understanding that some schools may require more than one turnaround intervention to achieve success. The selection process does not anticipate this per se.

Additionally, continuous improvement within schools also received too little attention in the evaluation of applicants. For turnaround to succeed, operators need a continuous improvement orientation baked into their DNA. CMOs must not flag in their commitment to critical reflection and strategic action in order to enact successive approximations of highest quality implementation. No operator can turn around a chronically failing school by expecting to transport unchanged a set of practices that worked elsewhere. Our findings indicate that even CMOs with high performing Flagship models needed to articulate and activate an unwavering commitment to continuous improvement throughout the years of the evaluation to have any hope of success. This commitment to continuous improvement, and an enumeration of the processes whereby that commitment would roll out in schools, was never tested by the selection process.

In addition to lacking selection criteria such as continuous improvement orientation, we also observed a willingness by RSD and NSNO to abandon existing criteria when faced with a choice between operators who cannot meet selection criteria and no operator at all. The CRM partners, especially in the early years of implementation, lacked the political will and/or the political cover to reject weak applicants outright. RSD and NSNO were loath to let selection cycles pass in which no operators were selected. The pressure to make grants in every round, even when presented with no viable operators, allowed for a systemwide downgrading of the CRM’s primary goal: to create an ecosystem of high performing schools, not merely an ecosystem of schools incrementally better than the ones they replaced. This more than anything represents an existential threat to the CRM in New Orleans or anywhere else: the CRM requires absolutely that partners maintain a long term focus on the task at hand, to transform chronically failing schools into high performers.

The challenges of selection – both in design and implementation – raise critical questions for districts considering the installation of a CRM. Failures at selection impact both fresh start and full school turnarounds equally. Strong starting endowments of the CRM at every level – from the universe of available operators to the quality of leadership to the ethos and commitment of all partners to continuously strive for the BHAG of top performing schools – are necessary prerequisites for a CRM to succeed.

The CRM Theory of Action: CMOs as the locus of intervention in turnaround

At its core, the CRM posits that CMOs are the primary lever for educational quality to improve. This evaluation indicates that placing CMOs as the locus of intervention has both benefits and detriments.

CMOs are closer to the classroom than a district office would be. CMOs are endowed with flexibility and autonomy to act in the interest of students first. CMOs are also endowed with resources that districts may lack from private philanthropic funding, human talent, and/or public funds specific to charter startup and operation.

CMOs also possess the potential to hold crucial knowledge about local contexts. Local CMOs, or nonlocal CMOs with a longstanding commitment to particular locales, hold both formal and informal knowledge about the regulatory, geographic, and cultural contexts within which they operate. Their leaders cultivate networks of individuals and organizational partners whose talent and expertise can be brought to bear. Further, their knowledge and networks render CMOs the best situated actors in a turnaround environment to build authentic support across stakeholders for school and student success.

But we see real risk to positioning CMOs as the primary locus of intervention. In this evaluation, we observe wide variation in board oversight and governance of CMOs, creating a potentially fatal weakness for any individual organization and, by extension, for the credibility of the entire system. Additionally, the work of simultaneously growing a CMO and turning around a school risks diluting the success of both efforts. We also observed role confusion among many CRM schools and their CMOs regarding core operating functions and responsibilities. Some of this is attributable to utterly expected organizational growing pains (especially for Type 1 and Type 2 schools). But, there are aspects of turnaround – particularly full school turnaround – that require very different approaches to leadership, culture, and instruction than assuming operation of schools that are not chronic failures. For CMOs to support the turnaround of chronically failing schools, CMOs must be built intentionally to understand and manage turnaround work.

In addition to these challenges at the level of CMO operation, we also identify systems-level detriments to placing CMOs as the sole levers for change. CMOs are decentralized actors. This is, of course, by design. But this also creates a set of externalities which threaten the sustainability of a CRM. Most fundamentally, no individual CMO faces any incentive to consider the entire system within which they operate. A fully decentralized system incentivizes CMOs to pursue their own goals. This lack of affinity – which we heard consistently articulated in the course of this evaluation – can and did encourage CMOs to maximize benefit to their own organizations at the expense of the collective good.

Without external intervention or oversight, a fully decentralized system will not prevent predatory behavior. Equity must be imposed – it is not baked into the CRM design, despite the commitment to educational equity that a school improvement initiative might imply. To the credit of the system-level partners, this realization eventually led to the installation of One App; centralized expulsion policies; and systemwide professional development, SPED and mental health supports. But even with these leveling functions in place, we see little in the way of consequences for bad actors. We believe that front-end equity functions must exist above the CMOs, but that back-end remediation must also exist for a CRM to sustain. CMOs may be the most effective lever for change, but the CRM includes no inherent controls to prevent that change from coming at the expense of other organizations.

More broadly, CMOs will not “become the system” without thoughtful, carefully calibrated structural incentives to do so. In New Orleans, which will soon have a 100% charter district, this point is especially salient. During the early years of this evaluation, the system-level partners behaved as if leaving the CMOs maximally unfettered would result in the emergence of an equitable, functional, sustainable system. That was not – and will not – be the case. CMOs may be the locus of school improvement, but they are not the locus of systems improvement.

The CRM Theory of Action: Systems in support of CMOs

If the CMOs are not the locus of systems improvement, from where does systems development emerge? As mentioned above, at the outset we see an overreliance on CMO autonomy to execute multiple aspects of the Theory of Action. CMOs were expected to hold responsibility for not just school turnaround in a single building but also self-governance, resource generation and procurement, and the mobilization of community support to activate and perpetuate the CRM.

Early on, NSNO and RSD suffer for the lack of clear distinctions in roles and responsibilities. They initially failed to intervene proactively on issues that are far larger than the auspices of any single CMO or school, such as facilities limitations and the woefully inadequate human capital pipeline. Moreover, the early interventions of the systems-level partners were comprised of investments in short term capacity building efforts, such as Executive Development training for the first and second cohorts of turnaround leaders, that left no lasting effect beyond the individuals who directly participated. These initial capacity building efforts had no multiplier effect, leaving NSNO and RSD confronting the same challenges across a growing universe of CRM schools year over year.

Eventually, though, the system-level actors – those entities who oversee, coordinate across, and support the CMOs – came to recognize themselves as exactly that: keepers of the system. The systems-level actors evolved from an initial expectation that CMOs will become the system to a much more strategic orientation of [systems surrounding CMOs surrounding schools]. This meant looking strategically across CMOs to identify universal need and high leverage solutions, rather than responding to each individual request with a scramble of activity to identify a one-off intervention. This meant anticipating change, both internally motivated or exogenously imposed. And this resulted in smoother operations at all levels, in addition to growing community support and demand for high quality schools.

Fidelity to Flagship

For evaluation purposes, CREDO brought more emphasis to the notion of Flagship fidelity than actors in the CRM did in their treatment of schools and CMOs. The focus on Flagship schools provided a preliminary picture of what a CMO held a successful school to be, and the variation across the Flagships that we studied bore out the wisdom of pursuing that viewpoint. Nevertheless, we have found in the evaluation that Flagship status does not serve as effectively as an anchor for baseline comparison or a predictor of CRM school success than we originally hypothesized. We have, however, come to understand the factors that are responsible for the differences, and so the comparisons were useful where they were possible.

At the point of selection, Flagship schools were considered demonstrations of proven success for their CMOs. While this may have conceptually informed selection considerations, CRM schools rarely considered themselves pure replications of their Flagships’ models. Further, we find little relationship between a CRM school’s fidelity to its Flagship and the CRM school’s success. As such, we conclude that the presence of a Flagship when selecting CRM operators or as a point of reference for CRM schools is neither a necessary nor a sufficient condition for success.

This finding is unsurprising. Most Flagship schools were greenfield schools – the first of their CMOs’ schools, and as such barely past the point of beta testing their models themselves. There was no expectation that CRM schools improve upon their Flagship models, even in those cases where Flagships were weaker than selection criteria recommended.

CMOs also reported fairly consistently that microcontexts – especially geographic neighborhoods, school catchment areas, and/or a school’s enrolled student body – were extremely important to schools. Schools in both states entered the turnaround process with long histories, often embodied by active alumni associations or neighborhood-school affinity groups who felt a sense of ownership for the Closing schools and had very particular expectations for the CRM schools. Additionally, schools in different neighborhoods served student who may have appeared similar on paper, but who may have had very different experiences. Students’ level of trauma – already high across the board in New Orleans and Memphis/Nashville – could differ widely from one neighborhood to the next depending on levels of community and domestic violence, variations in quality of housing stock, and, in New Orleans, extent of hurricane destruction.

As such, transporting a Flagship model to a CRM school – even if that school sat just a few blocks away – was framed as CMOs as a risky proposition. In New Orleans, some CRM schools looked to recruit from the neighborhoods in which they would be permanently located in order to minimize disruptions to their student bodies and maximize their access to students who fit the profile they ultimately hoped to educate. New Orleans schools that focused less on their permanent neighborhood suffered loss of enrollment when schools moved from temporary to permanent locations.

Similarly, schools serving neighborhoods that (as described by the CMOs themselves) had not had a high performing school in living memory faced the critical task of not only installing a functional school, but also educating neighbors and parents about what to expect from such a school. Tension between community members and alumni who mourned the loss of school bands and football teams – longstanding points of pride in places where academic success had seemed inconceivable for decades – and CRM operators looking to extend instructional time often resulted in compromise solutions that allowed schools to protect beloved cultural institutions, but in doing so forced a diversion from their most obvious path to increasing academic rigor. Losing instructional time to band practice does not necessarily require a step back from academic rigor – some CRM schools found creative solutions to meet both needs. But in some cases, we find CRM schools are unable to achieve the balance of academics and enrichment observed in their Flagships due in part to how their unique microcontexts behaved.

Finally, the Flagship schools were for the most part mature organizations by the time the CRM initiative began. This creates an intertemporal problem in positioning Flagships as a basis for comparison. Flagships, as fully enrolled and operational entities, may better serve as aspirational targets for CRM schools than as a fair test of how a school should behave during the startup years of turnaround.

Schools as Performance Management Organizations

Our ratings of schools on the Performance Management Organization (PMO) rubric reinforce the notion that schools are incredibly complex organizations. As such, schools require unwavering attention to a number of different functions simultaneously in order to be effective. Further, these functions are not solely related to the daily business of classroom instruction, but rather implicate a more holistic set of responsibilities that undergird the cultivation of purpose, drive, and joy for students and adults alike.

Our findings from the PMO analyses suggest that successful schools must put students at the center of every decision they make and every activity in which they engage. In order to do this well, schools must be thoughtful in what they do and in what they choose not to do: student learning must be front and center, ancillary activity can be outsourced. Schools must connect their attention to student results, constantly driving toward student learning.

Securing buy-in from every adult in the building also supports school success: CRM schools report orienting even their janitors and cafeteria staff to school values and practices. Teacher commitment also matters. While we see high turnover across the years of the evaluation, school leaders report a growing realization over time of the need to hire teachers who commit fully to their profession, to the development of their craft, and to their geography.

Most fundamentally, successful schools need to infuse continuous improvement into their very ethos. Learning and adapting infuse all levels of successful schools. Adults and students alike must have unending opportunity to engage with robust feedback. Principals, teachers, staff, and boards must all stay laser focused on students and steeped in a school’s sense of purpose, drive, and joy. It is through a commitment to upholding these principles that we see increasing resilience among schools when they encounter inevitable shocks.

The Maturation of CRM Schools

The resilience of schools is a core consideration for the CRM because shocks – exogenous ones such as budget cuts or endogenous ones such as staff turnover – are inevitable. This evaluation finds that schools’ starting endowments matter greatly in setting a school up to weather both expected and unexpected difficulties. We find great variation in the strength of school leaders, the availability of supports to CMOs and to schools, the quality of human capital, and the initial installation of school culture. We also see vast differences in the commitment to and mechanisms of continuous improvement. Some CRM schools struggle for failing to commit to the difficult work of critical reflection and action planning. Others struggle for misdiagnosing problems when they arise, or for generating ineffective solutions.

We observe CRM schools struggling to stabilize these types of operational and instructional functions well into their third, fourth, and fifth years. The runway for turnaround as originally conceived by the CRM may well be too short. But a protracted runway is not the solution either. Budgetary limitations would prevent incubation periods of any longer than a year, and slower turnaround trajectories would leave too many students in under-performing schools for too long.

This suggests a need in any future instantiations of the CRM to leverage as much learning as possible prior to opening. Further, this learning must be turnaround specific. Visiting only high performing schools put CRM leaders at a disadvantage in understanding how to overcome the challenges of startup. CRM principals had excellent exposure to what they wanted their schools to become, but not necessarily to the ways in which they could get there.

The communities of practice which NSNO attempted to establish in the middle years of the evaluation period represent a potential avenue for leveraged learning. While the communities of practice as implemented were both short-term in duration and shallow in their levels of engagement, they could have had greater effectiveness if they had been better conceived and resourced. Indeed, CMOs reported that they would have derived more value from the communities of practice if the meetings had gone deeper into topics of interest and been better structured to maintain CEO attendance (rather than allowing CEOs to attend once and then delegate future meetings to staff).

Another tactic we propose to support leveraged learning implicates Goal 3 of the original CRM. While NSNO successfully communicated the CRM model to national audiences, they did so from the perspective of the harbormaster, i.e. their own perspective. However, we see little communication of how the CRM lives and breathes at the school level. An elegant invention here would be the creation of a problem inventory: a simple census of the challenges encountered by CRM schools, coupled with information about how the schools met those challenges, and the results of their efforts. Over successive iterations, such an inventory of problems and solutions would evolve into an operating manual of sorts for CRM schools.

An asset inventory would provide additional value to the CRM schools and CMOs. Such tools have a precedent in New Orleans:, the work of entities such as the YouthShift intermediary and The Data Center serve to identify and clearinghouse community and regional assets for youth-serving organizations. An inventory of assets generated by CRM schools enumerating how and when they were able to rely on specific assets to address specific needs would complement a problem inventory in providing guidance to future CRM operators. Further, engagement with entities already involved in asset mapping would push CRM schools to rethink community engagement. This evaluation finds that community and family engagement provided the greatest boost to those schools that approached engagement as an opportunity to build an ecosystem of support around students, rather than as an instrumental one-way relationship to bring resources from communities into schools. Casting community organizations and members, as well as families, as assets to be perpetually cultivated rather than as resources to be raided would produce mutual benefit to communities, families, and schools.

Finally, NSNO might leverage their school reviews to build out SWAT teams equipped to intervene in the areas of concern which the school reviews identify. This evaluation finds that the high performing CRM schools made yearly use of NSNO’s school reviews, and that reviews of high performing schools increased in sophistication (and hence usefulness) over time. If NSNO were to build capacity – internally or elsewhere in the system – to provide short-term intervention as well as review, schools could receive support earlier and more effectively in areas that are best addressed before reaching a crisis point.

It is important to note here that as the CRM schools stabilized, matured, and built resilience, the system was simultaneously maturing around them. In the CRM’s first two years, NSNO and RSD stood back while CMOs were expected to solve, ignore, or supersede systems-level barriers that they were, in fact, ill equipped to impact. But as the system evolved, NSNO and RSD recalibrated their activity to build the connective tissue – the system – that operates above and between CMOs and schools. Community support for the system also evolved in this time, as evidenced by the yearly Cowen Institute public opinion polls, the reduction in community protests, and the nascent engagement of families with their children’s schools and CMOs. As the system evolved, we see thinking shift about what responsibilities CMOs can and cannot be reasonably expected to hold as levers for change systemwide. This evolution continues, and districts considering the viability of the CRM would do well to continuously examine which functions are best held by which actors, and in which instances are interventions or activities more effective when they are not mediated by CMOs.

Where does the CRM go from here?

In final consideration, the CRM fell far short of its target for 15,281 seats in top performing schools. Only half the CRM schools posted positive impact for students, but most CRM schools perform better than the schools they replaced. Given that the CRM instituted some level of improvement, what is the prescription for the CRM to extract additional improvements?

We know from research literature, and we find in this evaluation, that the culture and function of an organization crystallizes early, and that an organizational ethos is incredibly difficult to change once established. We know that chronically failing schools can carry the baggage of bad management, dysfunctional culture, and low performance forward through a turnaround. We know that chronically failing schools require tremendous effort to stabilize and raise, more so than closing a school outright and starting fresh. But, we also know that closing a school outright places a heavy burden on students forced out of that closing school. Unless students find seats in higher performing schools, they will experience a net negative impact on their learning, even if their new school is no worse than the one they left.

Full school turnaround is resource intensive (perhaps at times prohibitively so), and fresh start schools carry great risk to closing school students. Where does that leave those who remain committed to creating high quality schools for all students? This evaluation demonstrates the very real pitfalls of both full school turnaround and fresh start interventions. We also find that, despite deep investigation into drivers of success (or failure), we still have incomplete information about precisely which features, endowments, or behaviors a school needs at the outset to guarantee positive results. Some points, however, emerge with clarity.

First, policy actors need to have a more diversified toolkit to intervene. The full-versus-fresh turnaround debate is a false dichotomy. As such, we can conceive of full school turnaround and fresh start schools as two of a larger set of school reconstitution strategies that state agencies (RSD, ASD), districts (OPSB, Shelby County), and harbormasters (NSNO) may want to build into their arsenals. Other strategies might include systems-level intervention into the barriers to school success. We see the beginnings of this in, for example, NSNO’s engagement of Relay Graduate School of Education. CRM systems may require the flexibility to seed and reward second, third, and fourth attempts to improve schools: the CRM at the school level is not a one-period solution, but rather an n-period solution, and systems-level partners should engineer for that. This does not suggest that schools be allowed to flounder through multiple failed iterations. Rather, in addition to more clearly defined pathways through successive approximations of success, CRM systems also require well-understood limits beyond which schools have no option but to close. In a truly student-centered system, the distinction between schools on a protracted runway to success and schools who will never achieve altitude becomes clear. As such, we recommend an absolute commitment to putting students first in all considerations and decision-making, no matter what the disruption to adults.

Finally, we see an opportunity to consider learning models that either directly address or delicately bypass systems-level barriers. High quality, thoughtfully applied instructional strategies such as targeted mentorship, place-based learning, blended learning and others can provide better instructional options for students without necessarily implicating the typical burdens of human capital constraints, facilities challenges, integrated holistic services, et cetera. In this way, we suggest that future instantiations of the CRM consider the value not just of reforming schools, but of reforming learning itself.