Evaluation of Scaling the New Orleans Charter Restart Model:

Our Research Questions

5. As the CRM matured over time, what lessons were learned regarding school management, network management, and systems management?
Successful turnaround requires optimal functioning in multiple domains. How did the CRM schools evolve systems, practices, and resources to successfully turn around low performing schools?

The CRM Theory of Action suggests that investment in strong CMOs will result in high performing schools, but does not specify a timeframe for the achievement of high performance beyond a school’s state mandated 3rd year review. To test the viability of this assumption, we documented schools’ trajectories of maturation across multiple operational domains. While we do observe change over time for some schools in some domains, our overarching observation here is that schools’ starting endowments matter greatly. Schools rarely managed to overcome slow or dysfunctional start-up periods. Without a strong foundation in place at opening, schools did not develop the agility to respond to exogenous shocks or to adequately serve highest needs students.

We examined CRM schools’ experiences as related to the origin of their CMO (local or non-local). We observed schools’ experiences and responses within the domains of: facilities and location moves, family and community engagement, school culture and whole-child supports, and management of operations and student performance. We also examined human capital issues: principal experience, performance management, and turnover; as well as teacher recruitment, retention, and development. We then tested a subset of these domains for associations with student impact. Not all domains are amenable to quantitative treatment, but we examined relationships to student impact for the following areas of inquiry: community engagement, facilities, local/non-local CMO, and principal performance management. Across these domains, we observed wide variability in maturation trajectories. Similarly, we find variation in which of these factors are ultimately associated with impacts on student learning.

Local versus non-local CMO

We examined CMOs by type in Question 1 but found no statistically significant associations between a CMO’s type and its CRM school’s performance. However, we do find an association between CMOs’ locale of origin and their CRM school’s performance. Namely, CRM schools which are operated by homegrown CMOs – those CMOs that develop in the same place that their schools are located – show significantly higher student growth in both math and reading. We do observe exceptions to this rule: the homegrown Gestalt CMO in Memphis closed both its CRM schools by the end of the study period, while the KIPP schools in both New Orleans and Memphis showed some positive learning gains for students. But on average, CMOs that developed as entities and then operated schools in their home region showed better results than CMOs that entered a locale in order to transport a model developed elsewhere.

While we cannot attribute causality to this relationship between local CMOs and stronger student growth, we can identify qualitative findings that speak to potential drivers of this association. Non-local CMOs contended with two simultaneous operational challenges when entering a new locale. First, they had to build local capacity to manage local operations that are most efficiently handled in situ rather than in a remote central office. For example, Aspire entered Memphis with an intention to create local administrative capacity, but planned to limit local decision-making power – all major decisions would run through the California headquarters. This set-up proved untenable: some decisions needed to be made more quickly, or with deeper local knowledge, than a remote headquarters allowed.

This also raises the second issue faced by non-local CMOs. Regulations, cultures, and communities differ across locations, and non-local operators needed a deep understanding of local contexts in order to build trust in their communities and to navigate local educational, accountability, and political systems. This may help account for KIPP’s success in both Memphis and New Orleans. KIPP, a national CMO, had maintained a presence in both locales for a number of years by the start of this evaluation. KIPP had fully operational and quasi-independent regional infrastructure that allowed for agility and cultural competence when faced with local problems. Their presence had been normalized in the community to an extent that Aspire, Future is Now, or Rites of Passage had not.

That the local nature of a CMO’s central office matters for student performance raises critical questions regarding the scalability of the CRM. Our findings suggest that markets with a critical mass of local, or long-standing national, CMOs in place may have better success with a CRM installation than markets that need to import operators to meet student need.

Facilities

Many of the CRM schools experienced extensive facilities challenges. Schools in both states reported that such facilities challenges undermined their ability to serve students. In New Orleans, these responses are unsurprising, given the extent of destruction to school facilities caused by Hurricane Katrina. Delays to the implementation of the New Orleans Master Facilities Plan created additional complications: schools spent years in temporary buildings never intended for multi-year occupation, and/or had to move multiple times to accommodate construction schedules. Schools reported that facilities challenges impacted instruction and daily operations, and that degraded facilities required more management attention and financial resource than schools anticipated. Moving from one site to another disrupted enrollment and community engagement. We recognize that the facilities challenges faced by schools in post-Katrina New Orleans were extreme, but we observe echoes of those challenges in Tennessee as well. Facilities requiring upkeep created a drain on administrative and financial bandwidth. Tennessee respondents also noted that school buildings did not accommodate innovative pedagogical approaches, citing specifically the size or arrangement of classrooms, the lack of structures that might accommodate maker spaces, and the need for architectural divisions between upper and lower grades.

Despite these substantive concerns regarding quality of facilities and changes of site location, we do not find statistical associations between facilities issues and student performance. Changing locations – moving from one school site to another – has no relationship to student outcomes. We see individual instances in which schools show positive impact despite their facilities limitations. We interpret this to mean that strong schools can overcome weak facilities, but this in no way suggests that facilities should be ignored when considering turnaround strategies.

Family and Community Engagement

Family and community engagement are baked into the CRM Theory of Action, and were originally formulated as systems-level functions held by RSD/ASD. However, as events played out, engagement came to be understood as a multi-faceted enterprise that operated on school, CMO and community levels. In the early years of this evaluation, family and community engagement became highly contentious as communities pushed back against the state takeover of schools and families expressed frustration with the opacity of the CRM’s operation in both Tennessee and New Orleans. In New Orleans, these challenges resulted in a shift of responsibility from systems-level partners (RSD, NSNO) to CMOs and finally to school-level actors. This shift decentralized engagement but also re-defined engagement more closely around each school’s community of students and neighbors. Only in the final year of the study do we observe an RSD-managed community engagement process, at Wilson, that resulted in satisfaction among all stakeholders. In Tennessee, ASD nominally upheld its responsibility to manage engagement, but seated community councils with individuals who had already expressed agreement with ASD’s predetermined actions.

In both New Orleans and Tennessee, all schools assumed responsibility for engagement after selection, defining “community” variously as the community of individuals served by the school (i.e. students and their families), geographic neighbors, local resource providers, and/or larger citywide entities such as businesses. In all instances, schools framed engagement of community-based entities – be they geographic neighbors, local businesses, nearby churches, or nonprofits – as a way to bring additional resources into schools. Engagement was tactical and defined primarily around targeted school-level needs.

Schools in both states consistently nominated the lack of strong family engagement as a hindrance to school success. We observe persistent recognition of the need for family support in schools, yet schools reported making little progress on identifying real solutions to this challenge year over year, even despite year-over-year increases in support for the CRM in the broader community according to yearly Cowen Institute polling. Nearly all schools created a family coordinator position, and teachers worked to build rapport with parents by sharing positive aspects of students’ experiences rather than calling home only when students exhibit problem behaviors. Schools instituted regular positive parent contacts (“Muffins for Moms” and “Donuts with Dads”) and in a few sites created parent resource rooms in school buildings. But these efforts never achieved the degree of family-school relationship building schools express they desire.

Family and community engagement challenges are not significantly associated with student impact – nearly all schools regardless of where they fall on the performance spectrum consider engagement a major challenge. Nor does fresh start status confer observed advantages in managing engagement. But we do observe some instances in which family and community engagement created real disruption to schools’ operations (even if these disruptions are not statistically associated with eventual student outcomes).

During an early selection process at Craig, the community mobilized to reject RSD’s proposed operator. That operator, FirstLine, expressed a commitment to working with communities and bowed out of selection rather than find themselves forced upon a community that did not trust them. Friends of King were later identified to operate Craig, while Firstline re-entered the CRM selection process and ultimately landed at Clark. Firstline confronted contentious family and community engagement at Clark as well: student protests supported by students’ families and other community members led to the removal of a principal in the school’s third year of operation.

Community protests at Carver Collegiate caused the school to cancel classes for a day in its second year of operation. Carver Collegiate ultimately used these protests as a launchpad for dialogue with the community: in the years following the protests, Carver Collegiate changed some of its most controversial policies regarding student behavior, allowed the Carver alumni association to have a presence in the school, and engaged with community activists to create a school culture that was both sensitive to community concerns and predicated on academic rigor. In Carver Collegiate, we see our clearest example of a maturation trajectory regarding engagement as the school shifted from no engagement to antagonistic engagement to authentic engagement of stakeholders all focused on creating a nurturing and challenging student experience.

Where engagement rested on an assumption that families engage merely to support the work of schools, or that communities engage merely to fill resource gaps in schools’ offerings, we observe little maturation across schools’ years of operation. But in both states, we observe that family and community engagement provided the greatest benefit to schools when framed as a relationship of the sort Carver Collegiate achieved in its later years. The aforementioned selection process of Inspire NOLA to run Wilson typifies this observation. Wilson had operated as a stand-alone charter school with an extremely active board. When RSD targeted Wilson for closure in 2014, the outgoing board as well as a parents group demanded a voice in the takeover process. RSD – learning from earlier mistakes – implemented a good faith negotiation among parents, school and state officials, and operators interested in taking over Wilson. This resulted in the ultimate selection of Inspire NOLA as the operator most aligned with parents’ values, RSD’s academic standards, and the CRM selection process.

School Culture and Whole-Child Supports

The notion of “school culture” had no consistent definition across CRM schools, although it manifested most frequently in terms of maintaining an orderly environment, installing appropriate behavioral management systems, and setting high expectations for academic performance and college access. Throughout the evaluation, we observe a strong sense of commitment to developing a cohesive, rigorous, and nurturing school culture among schools. CRM schools consistently listed school culture as one of their top strengths, and expressed a sense of pride in a positive school atmosphere, year-over-year improvement in student buy-in, and incremental improvement to systems (behavior policies, lunchroom rules, transition processes) that resulted in more orderly environments. For nearly all CRM schools, pride in school culture exceeded academic accomplishments.

Yet we also observe that culture – particularly as defined by student behavior and quality of the learning environment – had not stabilized for most schools by the end of the evaluation. Teacher and leader turnover disrupted schools’ culture consistently throughout the years of the study. Fresh start schools reported some advantage here, as did elementary schools: culture is more easily established with fewer students and adults to onboard, and more easily established with students who have fewer years of experience in low performing schools. But fresh start status was not a guarantee of positive culture, especially for high schools. Brick Church (a middle school), Carver Collegiate, Clark, and Cohen (high schools) all reported early struggles to bring both students and adults into alignment on their schools’ missions and cultural expectations. Co-location with a closing school also complicated culture. Brick Church in particular reported that co-location with their Closing school undermined their ability to communicate and embody the culture of high expectations they had hoped to instill.

Stable school culture may be best thought of as a necessary precondition for learning, but not a sufficient condition to guarantee school success. Schools with low teacher expectations and chaotic environments (Clark, Cornerstone, Craig, FIN, Humes) struggled to meet their minimum student growth targets. But schools with warm environments and clearly communicated expectations for learning also failed to create positive student impact (McDonogh 42).

Teachers and principals most frequently attributed the difficulty in establishing positive school culture to students’ behavior. In most CRM schools, this was framed as a depth of student need – often explicitly connected to trauma – that schools lacked the resources and expertise to meet. Many school leaders reported a growing realization over time of the intensity of need that their student populations possess. Schools cited large percentages of students with high needs (including many who are not formally classified and therefore ineligible for state-funded intervention services). Schools also noted limited staff and financial resources as well as a lack of community resources as major barriers to providing the full support that students need.

The CRM schools showed maturation across the study period in regard to their ability to provide or procure holistic supports for students. Over time, the CRM schools refined systems for intervention, identified external partners to provide mental health and social work support, and took increased advantage of city and nonprofit services that came online throughout the course of the study. But even in their third and fourth years of operation, we still find that schools expressed a profound inability to serve all students well.

Management of School Performance and Operations

The trendline of schools’ overall management of performance and operations is one of tremendous fluctuation. We observe pervasive evidence of schools' unpreparedness for the difficulty of managing complex systems in intricate environments. In schools’ initial years of operation, we see persistent struggles to achieve stability in many areas of school operations including class scheduling, facilities use, day-to-day operations and human capital management. System–wide challenges such as high needs student populations, lack of human capital, and enrollment issues played an influential role. Leadership turnover (discussed below) exacerbated these struggles, as school-level systems were disrupted every time a new principal arrives.

Schools and their CMOs demonstrate highly dynamic relationships through the study period. As CMOs expand their central office capacities, and as the number of schools in their networks grow, the CRM schools report some role confusion or changing dynamics in how they interact with their CMOs. Regardless of these (often anticipated) growing pains as both CMOs and their CRM schools matured, the CRM schools report ongoing alignment and benefit to network membership. CRM schools report that their CMOs provide them with numerous supports including financial management, back office support, human resource management, advertising, and curriculum support. Many schools report that their CMOs will secure any resource the school can justify asking for, although others note that their CMOs can provide everything the schools ask simply because the schools do not ask for things they know the CMO cannot provide.

Fresh start schools have an advantage in managing performance and operations: by building operational and accountability systems when student enrollment is small, fresh start schools were able to test the viability of their systems and to bring a higher ratio of adults to bear on startup functions. But again, this raises critical questions for the CRM: if CMOs are positioned in the Theory of Action as the lever for change, and CMOs ostensibly provide schools with the supports schools need to focus purely on student development, why did the CRM schools in this evaluation struggle to install and maintain operational systems? Schools in the CRM faced not only the challenge of implementing building level systems, but also the challenge of knowing which functions belonged to building level versus CMO level systems. We observe inconsistencies within schools across time regarding the management of financial accounting and budgeting, procurement, curriculum, and professional development; at times schools managed these functions in-house, at other times CMOs hold some or all of these responsibilities.

School staff did show a growing realization over time of just how challenging school turnaround becomes when systems do not take hold early. CRM schools demonstrated maturation of student supports and data use in Years 4 and 5 of the study, as well as a stronger focus on implementing school systems with fidelity and consistency. Overall, however, our findings here align with earlier work in other charter schools nationally, which suggests schools must establish functional systems immediately upon opening. Schools cannot recover the ground they lose when those systems are less than optimal early on.

Performance Management of Principals

The most consistent threat we see to CRM schools’ stability across schools over time is that of principal turnover. Twelve of the CRM schools had at least one school leader turnover during the study period, and one school had as many as five leadership transitions. We observe that frequent leadership turnover created inevitable disruptions to the continued maturation of school operations, as new leaders learned systems, rebuilt relationships with staff, and often introduced operational and/or pedagogical approaches that differed from their predecessor. Further, leadership turnover significantly negatively impacted student growth in math (although not in reading).

CRM principals came to their work with varying levels of previous leadership experience. Only thirteen of thirty-four CRM school leaders had previously served as a principal, although all reported some form of leadership training immediately prior to assuming school leadership. Principal satisfaction with opportunities for their own professional development varied, but many principals reported struggling to access the supports they required to develop as leaders. Principals reported barriers including time constraints, limited availability of trainings/mentorship, or an expectation within their CMOs that principals would seek out their own development opportunities rather than proactively investing in their development as leaders.

Despite limited professional development opportunities, most CRM principals received feedback on their schools from NSNO or ASD in their early years of operation. Most schools reported little in the way of activity or impact from NSNO or ASD reviews after the third year of operation. However, we do observe that in New Orleans, higher performing schools continued to take advantage of NSNO’s school reviews. Further, those schools report that NSNO’s reviews became more nuanced and sophisticated as schools matured. While we cannot make causal conclusions, these observations suggest that student success relies in part upon a continuous improvement orientation in which principals can receive and act upon feedback about their schools and themselves.

Recruitment, Retention, and Performance Management of Teachers

We observe consistent efforts – but little unqualified success – from CRM schools and program partners to establish stable, high quality teacher corps. Systems-level partners recognized early in the implementation of the CRM that the supply of qualified teachers could not meet the need in either New Orleans or Memphis (we do not observe teacher recruitment challenges in Nashville, though this is likely an artifact of having only a single CRM school there). The extremely limited supply of high quality teachers emerged immediately in this evaluation as a core threat to the viability of the CRM (as it would threaten any district attempting large scale quality improvement). This threat showed little evidence of alleviating throughout the course of the evaluation.

We do, however, observe attempts to mitigate the dearth of teachers. These efforts to bolster the limited teacher pipeline were both slow to emerge and imperfect, but signal necessary (if not sufficient) recognition by systems-level partners of the need to establish permanent infrastructure regarding this particular issue (click here). In Memphis, schools benefited early on from the presence of a local nonprofit, the Memphis Teacher Residency, which worked with local teaching colleges to recruit, train, and retain classroom teachers. Beginning in 2013, ASD helped develop the “Teacher Town” initiative, which convened local funders, school and district officials, charter operators, and other local stakeholders toward the end of building a more robust teacher pipeline. In New Orleans, we do not observe systems-level partners developing their own capacity for human capital management or professional development, but NSNO did take a strategic and targeted approach to the procurement and disbursement of TIF and JLAF funds, and eventually recruited Relay Graduate School of Education to New Orleans, toward the same end.

Despite these systemwide interventions, school leaders noted both the limited number of qualified applicants and the increasing competition to hire strong teachers as more schools open and more grades are added in every year of the evaluation. We did not observe any reported advantage to Fresh Start schools here after the first year (when they hired only for a single grade). Fresh starts and full school turnarounds both reported struggling every year to find the teachers they needed. Schools reported that they prioritized candidates’ mission alignment and culture fit more than experience or demonstrated effectiveness in early years. However, as understanding of student need and the challenges of teacher development matured, leaders refined recruitment strategies to better target desirable candidates. Schools gave greater priority to prior experience. They also came to value teacher commitment, both to the profession and to geographic region.

The majority of teachers in CRM schools were happy with their jobs, but also described deep concerns about the sustainability of their work. They report that their hours worked and work-life balance mutually exclude other life goals such as having a family or pursuing graduate studies, which in turn encouraged turnover as these life goals shift in priority relative to teaching in turnaround schools. Principals cited staff turnover as a disruptor to the development of a strong teaching corps.

Principals appreciated the recruiting and hiring assistance that their CMOs provided, though as networks expanded, CMO management of recruitment sometimes restricted principal autonomy and created competition for teachers among schools within a single CMO. We do not observe innovation among CMOs around recruitment strategies: they relied on teacher fairs, online postings, and (later in the study period) the pipeline-building efforts of their systems-level partners.

Schools (and to a lesser extent CMOs) also dedicated effort to developing the teachers they had. Teachers and school leaders shared fairly cohesive understandings of student learning standards across time. Most teachers felt supported by their principals and prepared to work with their student populations. Despite this, high teacher turnover remained a challenge to schools throughout the study, which in turn disrupted attempts to develop those teachers. We observe consistently across schools/across time that teacher training too often prioritized the basics of classroom management to support the steady influx of new teachers, leaving little bandwidth for schools to provide differentiated training to more seasoned staff.

Unsurprisingly, teacher quality was principals’ most often-cited hindrance to student growth. And, indeed, although teacher turnover was never a preferred strategy for performance management among CRM schools, principals reported that they came to rely on turnover of low performing staff as a way to improve overall teacher quality in the later years of the study.

Maturation, Summarized

Overall, the maturation curve of CRM schools indicates tremendous hurdles in their first years of operation, and continued fluctuation between stabilization and disruption in their following years. The Theory of Action suggests that strong CMOs will operate high performing schools, but does not specify a timeframe for the achievement of high performance. Given our observations that schools struggled with at least some core functions through their third, fourth, and fifth years, we conclude that the required runway for school turnaround is years longer than anticipated, especially for full school turnarounds. We see some advantage to the fresh start model here: operations, culture, and to a lesser extent teacher recruitment benefit from having fewer students whose needs must be met. But for the majority of domains we consider, maturation of schools’ functioning appears to matter less than beginning operations with a strong starting endowment. Given research evidence that the stability of a school in early years strongly indicates the school’s ultimate success (click here), the absence of consistent and efficient maturation processes in CRM schools presents cause for concern when considering scalability.