Monday, September 28, 2009

Sunday Commentary: Do NYC Charter Schools "Cream?"

Probably.  At least that seems to be the most sensible interpretation of the recently released report on NYC charter schools.  And it's important to note that the Wall St. Journal gets this dead wrong in their editorial on the report.  The main conclusion of the report was that students who enrolled in NYC charter schools performed much better than did students who tried, but failed, to enroll in these charter schools.  So I found it more than a bit odd that the Wall St. Journal's headline trumpeted something else.  Why lie when there's plenty of truth to extol?

Nowhere in the report is the claim made that charter schools do not "cream" (select only the top students, like one would cream the fat off of milk, for those unfamiliar with the lingo).  The claim that charter schools don't cream seems to be based on the fact that charter applicants are more likely to be black or poor than the average NYC student.  This is an important, and entirely unsurprising, piece of information.

But here's what they missed: the authors go out of their way to stress that they cannot accurately compare the prior test scores of charter school applicants to non-applicants, writing that "it is not possible to draw conclusions about how charter school applicants' achievement compares to that of students in New York City's traditional public schools" (p. 28).

Nonetheless, they display a chart on the same page with calculations that the average test score of charter school applicants is no different than the average score of non-applicants.  This despite the fact that applicants were much more likely to be poor or Black and much less likely to be White or Asian.  And any cursory glance at test scores in NYC will tell you that the average poor Black student scores far below the average non-poor White or Asian kid.  Which means that we should see average scores considerably below the citywide average from charter school applicants.  The fact that there is no difference would indicate, to me at least, that charter school applicants are out-performing other demographically similar students.

Similarly, the authors write on page 71 that students who are accepted into, but do not enroll in, a charter school have prior test scores no different from those who do enroll.  But, at the same time, admitted students who choose not to enroll are much more likely to be White or Asian.  Which raises two questions: 1.) If the authors had previously argued that there wasn't test data on enough students to make any claims about test scores, why are they now making claims about test scores? and 2.) If the non-enrolling admitted students are more likely to be White or Asian, would we not expect them to have higher previous test scores than do enrolling admitted students?

Besides, comparing test scores is, to some extent, beside the point.  It might very well matter more how motivated a student and their family is, what kind of support they are receiving at home, etc.  If you were running a school and your goal was for students to make as much progress as possible, which would be more important qualification for students: previous success on tests or the willingness to work hard?

In the end, we need to see both disaggregated data (e.g. comparing Black enrollees to Black non-enrollees) and comparisons of test scores and other variables controlling for various demographic variables before we can make any firm claim about NYC charter schools creaming students.  But the authors' position is firm: they can't tell.  And if you ask me, all signs indicate that they likely do.  Either way, the Wall St. Journal still needs run a correction.

Last, but certainly not least, should we care if charter schools cream?  If charter schools enroll more motivated or more capable students, might that not be a good thing (especially since they're clearly attracting students from the most at-risk demographics)?  The answer, of course, is that it depends.  It's hard to imagine that enrolling more motivated and higher achieving kids in a school wouldn't be a good thing for those who attend that school.  At the same time, there's a possibility that it might negatively influence kids in the surrounding schools that are losing the cream of the crop.  But it's hard to say to what extent either of these scenarios are playing out.  At this point, whether or not creaming makes charter schools bad is largely in the eyes of the beholder.

But, more importantly, it affects the way that charter schools should be analyzed -- because, ultimately, charter schools are supposed to serve as incubators of ideas that spread to other schools.  Which means that we should make an effort to figure out which of these ideas are worth replicating.  And if charter schools cream a little or a lot and we don't take this into account, it skews both the analysis of their performance and the analysis of which ideas should be replicated.  Which has the possibility of harming the students we ultimately want to help.

5 comments:

Roger Sweeny said...

Last, but certainly not least, should we care if charter schools cream? ...[Yes,]it affects the way that charter schools should be analyzed -- because, ultimately, charter schools are supposed to serve as incubators of ideas that spread to other schools.

Perhaps one of the ideas we should take from charter schools is that every high school should cream. By the end of eighth grade, all children should be able to read, write, and do basic math but we shouldn't require them to stay any longer in a special building full of desk and textbooks and classes of 20-30 people all their own age.

Corey Bunje Bower said...

And then the children who aren't allowed in the school do what, exactly?

Roger Sweeny said...

Right now, employers look at a high school diploma and they can't be sure the applicant is functionally literate or numerate. But they know that he or she at least has the motivation and self-control to make it through 12 or 13 years of schooling. The value of the degree has little to do with the academic subjects the student has taken, most of the content of which he or she has forgotten.

It is certainly true, as the marketing for schools tells us, that good paying jobs like auto technician require reading and math skills. But these skills are actually not terribly advanced. I would require them for 8th grade graduation.

After that, it seems wrong to make kids who don't have much of a desire for academics stay in an academic institution. And, of course, in a place like New York City, many don't. All the best people consider them to be failures, and they often consider themselves to be failures.

There are lots of productive, decent paying jobs that don't require the knowledge gathered in high school and college courses. But they do require basic literacy and numeracy--and the ability to show up on time and do what has to be done. Let's concentrate on that.

Anonymous said...

You miss the argument entirely. Charter schools educational gains over traditional public schools, according to this report, cannot be based on creaming because the admissions process is random lottery and the lotteries are oversubscribed. By comparing students that are admitted to those that are not admitted, you control for the motivational factor of applying to a charter school.

Corey Bunje Bower said...

It's odd that every comment criticizing my writing on charter schools seems to come from an anonymous poster.

Anonymous: you miss my point. I never said that NYC charters weren't out-achieving traditional public schools in NYC. In fact, in other posts I've said it's more likely than not that they are. My point was that it's factually incorrect to say that this study proves charters don't cream. Charters can both cream and perform admirably with similarly motivated kids -- in fact, it would be a pretty logical explanation for the findings.