Reading about Robert Bobb’s plans to turn the worst-performing DPS schools into charters, I’ve increasingly realized how limited is my knowledge about K-12 education in general, and charter schools in particular. In an attempt to remedy this deficit, I’ve spent some time combing through recent literature on charter schools, standardized testing, and evaluation of teacher performance. In this post, I’d like to share some of the more interesting findings I came across in these areas of study.

An issue brief from the Economic Policy Institute exhaustively makes the case for why use value-added modeling of test scores to evaluate teachers must be incorporated only as one of many criteria, and they provide a detailed alternative for appraisal, the use of “systematic observation protocols with well- developed, research-based criteria to examine teaching.” Basically, reliance on test scores seems like the easy, cheap way out, but they are flawed measures of teacher effectiveness. There is no substitute for administrators taking the time to systematically observe and evaluate teachers.

Other findings of note from the EPI brief:

The tests themselves are very limited too, mostly multiple choice on math and reading, not reflecting the plethora of things students are supposed to be acquiring in the classroom that are not so easily measured by this particular format (like, can you actually communicate effectively?).

Teachers have a strong bias for working in stable districts with stable funding and kids from stable backgrounds who have adequate resources at home to come to school ready to learn. I know, what are they thinking? The most experienced teachers go to those districts, and so the poorest districts with the neediest kids are destined to end up with less experienced teachers. You would have to pay them a lot more to compensate the experienced teachers for working in such disadvantaged environments. Needless to say, that’s not what generally happens.

My personal take on the subject: Even good teachers can only do so much. Call it the tyranny of low expectations, but absent increased allocation of resources, how can kids in high-poverty school systems help but perform worse? State and federal legislators don’t seem to have any interest in increasing those resources, only in using the low success rates as an excuse to pull funding. It’s exacerbated by the atrocious corruption of local administrators like those at DPS, who systematically ran the schools into the ground and squandered hundreds of millions of dollars in the process. No one should blame the teachers for the sins of the school board and high-level administration officials.

(That said, they’re going to take the hit anyway, because the money is gone. As any former employee at Enron or Bear Stearns can attest, when your employer is going belly-up, it doesn’t matter how hard you worked or that it isn’t your fault. DPS teachers can picket and complain at meetings all they want, but the plain fact is that they will be lucky to keep their jobs, and in the rosiest scenario they should certainly plan on major cuts to their total compensation over the next few years. If I worked for DPS, I’d be running for the exits right now and applying for every job outside the district I qualified for. Like I previously wrote about gas prices, who really didn’t see this coming?)

I also found a number of interesting papers on charter schools, much of them not especially favorable to charters.

For example, evidence from North Carolina indicated charters are recruiting a lower-quality pool of teachers compared to publics:

I … found that charter schools were drawing below average teachers from school faculties… (T)he quality distribution for future charter teachers was significantly left of the quality distribution for teachers who never taught in a charter school…charter schools were not drawing unambiguously superior teachers from the mainstream sector as a whole, nor were they skimming above-average teachers from mainstream faculties… North Carolina’s end-of-grade exams reflect a set of appropriate cognitive standards… Regarding this dimension of teacher quality, North Carolina charter teachers compared unfavorably to other teachers while they were teaching in mainstream schools…

Closer to home, Michigan charters historically have failed to make our state’s public schools more competitive. A study examining “the competitive effects of charter schools on the efficiency of traditional public schools,” and utilizing “a statewide school-level longitudinal dataset of Michigan schools from 1994 to 2004… (found) that charter competition had a negative impact on student achievement and school efficiency in Michigan’s traditional public schools.”

The external validity of these two studies is unclear, since they are after all confined to single states; however, according to RAND Corp., “after several studies, there is still little evidence that the presence of charter schools affects the achievement scores of students in nearby traditional public schools either positively or negatively.”

While charters overall perform worse than public schools, the evidence suggests that the comparative performance depends on a host of factors:

Context and location: Brookings* cites the five randomised trials comparing charters and public schools have consistently suggested that students from poor urban schools do better in charters, whereas students from middle-class and suburban backgrounds do worse.

Race: Raymond et al**, however, found a counterintuitive effect when it came to race: black students perform significantly worse in charters than in public schools (and this effect outweighs the benefit of charters to poor students). Given these contradictory findings, it is very ambiguous that charters would improve performance in the lowest-performing schools at DPS.

Age level. Raymond’s findings suggest charters are especially beneficial for middle school students, slightly less for elementary. High school students tend to do much worse in charters, and multi-level schools worst of all. DPS needs to take extra care about who they put in charge of high schools.

Length of time a student has attended the charter. Raymond et al found students in their first year in a charter do much, much worse than their peers in public schools. But second year students seem to do about the same as public school peers, and third year students do better.

State policies. States with caps on the number of schools see decrease in achievement by charter students compared to public school peers. States with multiple authorizers see a huge decrease in charter achievement. States where families can appeal charters’ admission decisions see a slight boost in achievement compared to publics.

A caveat from Brookings about the randomised trials it cited:

(These) results also do not necessarily generalize to students whose families have not tried to gain admission for their children into a charter school. To the extent that the success of some charter schools depends on motivated parents who buy into the school’s approach and make the extra effort that may be required to get their child to measure up to the school’s demands, such schools might not succeed with students whose parents do not have that motivation or are unwilling to make that commitment…

Since there is a significant subset of charters that do poorly, Raymond et al argue we must have policies in place to close the bad schools. Yet, they lament, authorizers seem unwilling to close them in practice. There is always some argument mustered by the school’s stakeholders for why that particular school shouldn’t be closed. Raymond et al conclude:

(W)e know now that first year charter students suffer a sharp decline in academic growth. Equipped with this knowledge, charter school operators can perhaps take appropriate steps to mitigate or reverse this “first year effect.”… (A)uthorizers must be willing and able to fulfill their end of the original charter school bargain: accountability in exchange for flexibility. When schools consistently fail, they should be closed.

Overall attainment at the DPS schools slated for conversion is so poor that it would be easy to assume they couldn’t do any worse under charter management; but as some of these authors have pointed out, a lot of the factors behind that poor attainment may be outside of educators’ control, and charters are no panacea.

I am very new to the education literature, so I appreciate any other references or resources you could share. I’d especially appreciate high-level reviews on these topics where they are available, and/or high-quality quantitative studies where they are not. I also welcome your corrections if I’ve made any errors in interpreting these papers.

* A December 2010 report from a panel of education experts at Brookings Institution, which extensively reviews the literature on charters.

** I came upon the Margaret Raymond study via Diane Ravitch, who called it “the only major national evaluation of charter schools.” It is a national pooled analysis of charters’ effects on student achievement, as measured by standardized math and reading tests.