Wednesday, September 23, 2009

A Few Thoughts on the NYC Charter Schools Evaluation Project

The NY Times profiled a report on charter schools that was released today.  The group conducting the investigation was led by Caroline Hoxby, an economist based at Stanford's Hoover Institute who has published a number of articles in prestigious journals that find positive returns to school choice.  The report is written in plain English and is very easily readable even for those who have little background knowledge or statistical training (which was quite a surprise to me given the highly technical nature of most of her work).

The bottom line of the report is that students who enroll in charter schools do much better than those who enter lotteries to enroll in charter schools, but don't have their number picked out of the hat.  The methodology of comparing "lottery winners" to "lottery losers" is probably the best way to go when evaluating charter schools because you should end up with comparison and treatment groups that are demographically similar.  Though, of course, that doesn't mean it's perfect.  A few thoughts:

-It's unclear whether students who are lottery winners apply to more charter schools than those who are lottery losers (one could imagine that a more motivated parent would apply to 10 charters and be more likely to end up with their child in one of them than the parent who applies to 1).

-It's also  somewhat unclear how they treat a student who leaves or enrolls in a charter school in a subsequent year.  They mention that about 14% of students leave charter schools (p. 71), but I didn't notice a part where they addressed the latter.  (correction: the 14% figure is the percentage of treatment group students that have transferred to traditional public schools over the life of the study, at least 17% have transferred to other schools and 24% have dropped out of the study for various reasons (p. 72))

-The study points out that lottery losers are 32% more likely to transfer to a private school or a school outside of NYC than are lottery winners (p. 72), but I'm not sure they fully investigate the effects of this.  It sounds as though lottery-losing leavers have test scores that are no different than lottery-winning leavers, but this isn't really the full story.  Current test score is only one predictor of future test score, and since they're measuring growth in test scores, I'd have to imagine that motivation is a better predictor of growth in test scores.  And as far as I can tell, it's entirely feasible that more motivated students are more likely to leave the system if they're not enrolled in a charter school.

-They estimate the average growth in test scores for charter school students to be .12 (math) and .09 (reading) standard deviations above and beyond those of lottery losers each year (p. 42).  For one year, that's not particularly impressive, but they calculate that these effects add up over time to create quite a powerful effect.

-Perhaps more interesting are the effects they calculate for individual schools.  Charter schools are supposed to vary widely, so it shouldn't really be possible to generalize to all charter schools.  Here are how the effects of charter schools broke down in each subject (p. 57-59)

0 - .1 SD
.1 - .2 SD
> .2 SD

Assuming that these measurements are accurate and that higher test scores reflect more meaningful growth as well (neither of which should be taken for granted), I'd argue that schools over .2 might be worth getting excited about and schools between .1 - .2 are probably doing something right.  Making those two dangerous assumptions, we might say that a majority of schools seem to be doing something right.  It also becomes evident why, as I've argued in the past, charter school advocates should advocate closing more charters.  Think how much higher the estimates would be if the bottom 20% or so that are getting returns that are negative or no different from zero were closed.

-Why are charter schools doing better (same two assumptions apply)?  The authors measure the effects of a wide variety of variables on page 64.  The most notable number is that adding 10 days to the school year boosts achievement .15 SD.  This is the only number that is both meaningful to me and statistically significant at traditional levels.  The average charter school has about 10 more days of school than the average traditional public school.  Meanwhile, the average charter school advantage is about .10-.11 SD.  Which raises the possibility that charters are doing better mostly because they have longer school years -- a reform which could very easily be replicated in all schools.

-You should ignore all the calculations regarding the benefits of attending a charter high school.  In the last year of data, there are four charter high schools in NYC.

I'll e-mail the team about the first two bullet points, hopefully I'll hear back -- I'll post the answers if and when I do.


Rachel said...

There are few things that leave me more cynical about Ed Reform (with a capital E and R) than the way "charter vs. regular" has become the lens to compare schools.

Is it the governance model that produces the differences that Hoxley sees, or the length of the school year? Or some other factor such as class size or teacher pay?

It's possible that how decision are made affects what kind of decisions are made. But it's the "what" that really affects students.

And though calling for the abolition of traditional school districts appears to be the new new thing in Ed Reform, I think it would be a lot easier to add 10 days to the school year than to re-make the governance structure of public education.

Dora Taylor said...

On our website, you will see our thoughts regarding charter schools.

Claus von Zastrow said...

Corey, I can't find your reference you cite to students leaving charters. (You pegged the number at 14% and said it appears on p. 71). Was that an annual attrition rate (which seems to high) or the average attrition rate over 8 years (which seems to low)?

A sound study, but I'd appreciate your thoughts on student attrition.

Corey Bunje Bower said...

Claus: It's pg. 71 in my Adobe Acrobat, but it's labeled page VI-I in the report. It reads "So far, about 14 percent of charter school students analyzed in this report have returned to the traditional public schools." It's the percentage of students who have left over the course of the study -- I'm not sure how long the average student in the study has been attending a charter school. Another good question for me to ask the report writers (or to read more closely to try and find).

Though, actually, now that I think about, the number that leave charter schools is at least 17% (look at the table on the next page). I'll correct the original post.

Claus von Zastrow said...

Thanks, Corey--

Hoxby seems to address the attrition question on the following page. Though I'm not entirely convinced by her analysis, she does not simply ignore the attrition problem, as claimed elsewhere.