Schools Rallying Communities

I have strongly mixed feelings about charter schools, but my biggest concern is one I almost never see mentioned by charter proponents, detractors, or neutral observers.  We hear a lot about how communities affect schools, but almost nothing about the reverse.

I grew up in a suburban district where people routinely headed to the local high school for football games, basketball games, school plays, and scads of other events.  And, to a lesser extent, the elementary and junior high schools brought in community members for fairs, concerts, etc.  All in all, the schools brought the community together quite often for various reasons.  And that's not uncommon.  Or at least, historically, it hasn't been uncommon.

But that might be changing.  If we imagine a world where schools and neighborhoods are completely decoupled and people from one town go to scads of different schools all over the place, that relationship almost ceases to exist.  We won't read stories like this piece in the NY Times about a small-town HS football team that's rallying the community.

Granted, it might be worth the trade-off if the new non-neighborhood schools dramatically outperformed our traditional school system, but it's important to recognize that there is a trade-off involved here.  And that schools have larger ripple effects on society beyond the academic performance of their current students.

Asking the Right Question About Charter School Skimming

This NYT article seems incomplete, but I like it for one simple reason: people talk all the time about charters cherry picking or cream skimming kids, but never seem to ask the right question . . . this article does. There's a ton of evidence that most charters do not take the highest scoring students (see, for example, this chapter from this new book) and those data are used as evidence that charters don't skim.

Case closed, right?  Do charters skim?  No, they don't.

But that's the wrong question. The issue shouldn't be whether charters take the highest scoring students, it should be whether they enroll the best-behaved and/or most motivated students (and then nudge out those who are unruly and/or unmotivated).

In other words, we should be asking if charters enroll kids who are better students instead of asking if they're enrolling students who previously earned higher scores.  Why?  Once you get a class or school full of motivated, attentive, and polite students it's a heck of a lot easier to teach them.  And a heck of a lot easier to see large gains in test scores.

I have yet to see any rigorous analysis of the extent to which charters do, in fact, enroll or retain better students.  Instead, I read a lot of anecdotes like the one from the NYT article I linked to above.  Were I to hazard a guess, it would be that there's at least one charter out there that enrolls/retains substantially better students than the surrounding schools.

Even if I'm right, whether or not that's a good thing or a bad thing is a whole separate discussion.  But let's start that discussion by asking the right questions.

"Selection Effects" and Charter Schools: A Clarification

A number of skeptics argue that one reason for some charters' success is that they skim some of the best students from traditional public schools.  Matthew Yglesias fundamentally misunderstands this argument (as, I suspect, do many others) in this post on recent research on KIPP when he writes that the authors are "able to look in a rigorous way at whether the high performance of KIPP students relative to demographically similar non-KIPP students is merely the result of some kind of selection effect".

While it's true that some charter schools do attract students who score higher, on average, than their peers, no serious education wonk is arguing that this alone is why the KIPPs of the world have higher test scores (which is not to say that nobody is making this argument).  Indeed, if we look at research on these high-flying charters -- note, I said research and not the popular press -- the statistics cited aren't usually snapshots of how many kids passed a certain test but, rather, longitudinal examinations of the growth of kids' test scores over time.  In this sense, simply having higher achieving kids from the start wouldn't help much -- and could conceivably hurt a school.

So what is meant by "selection effects" then?  Well, when skeptics argue that charters often skim off the best students, they mean best students in a more holistic sense.  If you ask a teacher to identify their best students, they wouldn't just point you to the kids with the highest test scores -- they'd point you to the kids who worked hard, cooperated, asked questions, turned in assignments on time, showed up every day, and generally did what was asked of them.  And having a school full of students in this mold would make teaching easier, hallways quieter, and a school's climate more positive -- all of which would aid student growth.

As far as I know, there hasn't been much research on whether charters do, in fact, recruit and retain kids who are "better students" in this sense (please note that I'm not saying there hasn't been any, only that I'm unaware of it -- and, actually, if you know of some I'd appreciate it if you sent it my way).  But there's plenty of reason to suspect that at least some charters' student bodies might skew in this direction.  Probably the most cited reason is that it takes extra effort for a parent to enroll their kid in a charter school -- making it quite logical to assume that more motivated parents are more likely to fill out the application (of course, maybe the parents' motivation is driven by hatred or their current school or something rather than desire for their kid to excel).  Secondly, there are various indicators that some charters are more likely to give kids the boot, or at least threaten to do so, than are traditional public schools.  For example, I watched one video in which a KIPP principal walks in the first day of school and tells a kid who's not cooperating that if this school isn't for him that he can leave -- that's not something that traditional public schools can really do.

Anyway, the point is this: when people talk about charters benefiting from "selection effects" they're talking about schools enrolling "better students" in the sense that they're more motivated and more cooperative, not that they simply enroll higher-scoring students.  I don't know whether or not charters actually have better students, but it's easy to imagine that a more enthusiastic, better behaved student body would make a school far more productive.

Today's Random Thoughts

-Hope everyone is having a great holiday season.  It always seems weird to me that people put their lights and decorations up weeks or months before, and then by the time we get to the 12th day of Christmas they're all down.

-Aaron Schutz has an interesting theory regarding the TFA civic engagement study -- if a number of people who go through TFA feel disempowered by their experience, they may be less likely to feel like they can make a difference in the future.  Although I'd think that those who had the worst experiences would be more likely to drop out . . .  Meanwhile, Andrew Rotherham and Rob Reich mostly agree with my feeling that TFA admitees are an elite group and that small differences within this group aren't Earth-shattering.

-Diane Ravitch says that some charter school operators are taking advantage of the set-up to pull in huge salaries.  She says she's heard of principals earning $400,000 or $500,000 and of one who made millions selling school supplies to the charter school he operates.  I'd have to think that this problem isn't very widespread, but I've been wrong before and likely will be again.  I guess it's like I've argued before: if you're going to argue that schools should be run like businesses . . . be careful what you wish for.

-I don't understand Jay Mathews' argument that scripted curricula could transform schools but that nobody will give it a shot.  If I've noticed one classroom trend since I started teaching, it's been the rapid spread of scripted and semi-scripted curricula, particularly in high-poverty, urban schools.

-Chad Alderman remarks on the fact that students from the poorest 40% of households made up 11% of the student body at a group of elite, private colleges in 2008-9 -- up from 10% in 2001-02.  Apparently the surge in financial aid during that time period didn't attract a ton more students from the poorest families.

-Robert Pondiscio reports that the Broader, Bolder Coalition is on the lookout for low-performing schools that artificially boost test scores through excessive test prep, curriculum narrowing, and other educationally dubious tactics.

-CREDO today released a report finding gains of .06 standard deviations in reading and .12 standard deviations in math for students in NYC charter schools compared to similar students from the same geographic locations.  For those of you who aren't statisticians, those gains are pretty small.  That students in charter schools would perform a little better than similar students in traditional public schools in NYC seems plausible, if not likely, to me.  Similar to the recent Hoxby student, there were some schools that did worse, some that did better, and many that were no different.  To me, the most important question is why some of the schools did better and whether that can be replicated and scaled up.

Today's Random Thoughts

-If nothing else, I like the title of this LA Times story: "Controlling a classroom isn't as easy as ABC" (hat tip: Flypaper).  Speaking from my experiences both teaching and working with and talking to other beginning teachers, there are an awful lot of little things about classroom management that seem obvious to veteran teachers and aren't to newbies.

-Jay Mathews says that the achievement gap is "useless as a measure of school improvement".  I sort of see his point that a school (or school system) could get worse while, at the same time, the achievement gap shrinks.  But that's only a reason to not make the achievement gap the only measure we examine.

-A new study finds that teachers in charter schools are 76% more likely two switch schools or leave the profession at the end of any given year, and that most of the attrition and turnover is due to dissatisfaction with their school.  My initial reaction is that this is probably largely b/c charter schools tend to hire different teachers -- namely younger, more idealistic teachers who may be less intent on making a career of teaching and at the same time are more willing to explore different schools and careers instead of committing to one.

-Diane Ravitch compares the support of charter schools by social elites to "origins of free schooling in certain northeastern cities in the early 19th Century, when wealthy men decided that it was their civic duty to help civilize the children of the poor".  I don't know if the motives of many donors are quite so paternalistic, but I do share her concern that the reliance of charter schools on this funding means that "our society will increasingly rely on the good will of wealthy patrons to educate children of color".

Closing Charters: Help Many, Hurt Some?

I think I mostly agree with Jay Mathews' latest post on closing low-performing charter schools.  But I can't tell because he uses flimsy evidence and makes at least one wild assumption.  The post centers around a charter school he says is performing poorly and why, nevertheless, the school isn't slated for closure.  I've asked before whether closing a charter school was actually easier than closing a traditional public school -- an assumption on which the market theory backing charter schools relies -- and I think it's an important question.

But I have no idea how bad the school in question is.  He cites a few test score statistics and then compares them to what I assume are the two highest-performing charter high schools in DC (one of which, SEED, is actually a boarding school and receives a plethora of outside dollars).  He leaves me with no clear idea how this school compares to the average school in DC, nor does he provide any qualitative evidence that the school climate is poor or students unmotivated.  Part of me is willing to take his word for it, but the evidence is certainly lacking.  Nonetheless, even if he picked the wrong school his point still stands -- there are some poor charter schools out there, and in order for charters to work as intended they need to be closed fairly swiftly.

But my biggest problem is with the last sentence in the post, in which he writes about a parent who thinks her kid is doing well at the school he fingers, saying that she needs to be convinced "that temporary disruption in her child's life will give him a better future".  What?

First of all, there's plenty of evidence that moving negatively impacts a child's performance in school the following year.  Given that the child in question is already in high school, he may not have a year to spend adjusting to a move.

Secondly, it may very well be the case that the child actually is excelling at his new school.  It's rather arrogant to assume that the parent is clueless in this particular case.

Lastly, the odds may not be particularly high that the student ends up in a better situation if he switches schools.  I'm unsure how many better schools are readily available to him, but it's probably not too many.  And if a school is "better" overall, that doesn't mean the student will be a good fit there.

Mathews' assumption is a good example of a problem that a lot of policy people run into when telling people what to do.  A policy can be both better for most people in the long-run and worse for certain individuals in the short-run.  Sure, the country is probably better off if low-performing charters are closed swiftly.  But the individual student mentioned -- or, for that matter, many other kids in that particular school -- may be better served if they remain where they are.

Today's Random Thoughts

-Did you know that half of pro football players and 21% of pro basketball players have college degrees?  I assume that baseball and hockey are lower due to the high number of people drafted out of high school in the former and the high number of people playing in junior leagues in the latter.  Perhaps more striking is that 60% of pro basketball players are broke within 5 years of retiring -- and the number might be even higher for pro football players.  So if you have any kids in your class who refuse to try in school b/c they're going to be a rich and famous pro athlete, you might want to let them know that that's even less likely than they might think.

-Some completely unsurprising (but still important) statistics regarding education and crime.  16 to 24 year-olds that drop out of high school are 47 times more likely to be incarcerated at any given moment than are those who have completed a 4 year college degree.  Of course, even the most enthusiastic education advocate will understand that this is at least as much due to who drops out and who completes college as it is to how much more education helps.

-I really wish that charter school advocates would stop claiming that the recent report proves beyond a shadow of a doubt that charter schools are the best invention since sliced bread.  The report provides important evidence, but it still leaves a number of unanswered questions (not to mention that it only covers one city).

-Keep your eyes open for a special piece on Sunday that answers the question "should we educate poor kids differently" from another angle.

Aha! I Figured Out the Solution!

As anybody who reads about education policy these days knows, only two things matter: charter schools and teacher quality.

For example:
Q: How do we close the achievement gap?
A: Create more charter schools and recruit better teachers

The only problem is that many of the most successful charter schools rely almost exclusively on a certain kind of teacher.  And given that teacher turnover is higher in charter schools than others, we need to find even more of them than we might otherwise -- especially if we want to create more successful charter schools.  But where will we find talented people to sacrifice their time and earning potential to work 80+ hour weeks helping low-income children at a charter school?

Do not fear: Corey is here.  And I've figured out the solution.  We need to train our kids to become charter school teachers.  As charter schools expand, the students of today will start entering the workforce ready to take their place on the cutting edge of education.  How will we do this?  Simple: use what we know from research about what makes a good teacher and what drives people to teach in a charter school.

What we know: Recent research on how teachers select which school they teach at finds that teachers tend to select schools similar to those where they went to school -- both demographically and geographically.  In other words, White, upper middle-class people (who make up the bulk of the teaching force), usually aim to teach in mostly white, upper middle-class schools.  Who were the exceptions to this rule?  Mostly people who wanted to teach for a few years rather than as a lifetime -- people who viewed teaching as a form of community service rather than as a career.

What we can do: Impart to our kids the importance of community service and helping others.  If more kids grow up wanting to help others, we'll have more people entering TFA after college and postponing their plans to start their own hedge fund or law firm.

What we know: The search to finding a formula to predict who will be a good teacher has not borne much fruit.  But we do know that test scores of potential teachers seem to matter only a little or not at all (depending on which study you read).  On the other hand, an understanding of why kids get certain problems wrong has proven to be an effective indicator of teaching ability.  We also know that people retain more of what they teach than what they hear.


What we can do: Instead of test-prep, we can have students spend time teaching other students.  Whether it's tutoring kids from their own class, teaching lessons to younger kids, or simply presenting what they've learned as part of "jigsaw" lesson, kids will end up knowing more about how students learn.  They'll enter adulthood with slightly lower test scores, but a better understanding of pedagogy.  And probably more knowledge.

So, really, the solution is simple: train more kids to be charter school teachers when they grow up.  We'll solve the labor shortage problem and close the achievement gap.  And, heck, maybe there will even be some positive externalities.

Arne Duncan on the Colbert Report

Three Observations:

1.) I've never seen such softball questions from Colbert.  Alexander Russo reported today that the staff didn't know what to ask him, and it showed.

2.) Duncan said that he wanted schools to serve as "community centers" and that they're often "the centers of a neighborhood and the heart of the community."  I'm not sure how this is compatible with his desire to create more charter schools, most of which are not true neighborhood schools.  Does he want traditional public schools to serve as community centers only for those students not enrolled in charter schools?  Or perhaps he wants charter students to attend school elsewhere but attend events and participate in after-school activities at the nearby traditional public school?

3.) Duncan was also factually incorrect when he said that our school calendar is "based on the agrarian economy".  That's a common myth, but it's wrong -- summer break and agriculture don't go very well together.

Updated: I've added links to Arne Duncan's interview as well as "The W0rd" segment with fifth grader Andy Gelman on whether or not summer break should be shortened (couldn't get the clips to embed for some reason).  I also changed the wording above to reflect direct quotes instead of paraphrasing.

Arne Duncan Interview


The W0rd

Charter Schools: Is More Always Better?

In the wake of the Hoxby et al report on the effectiveness of NYC charter schools, quite a few people seem to be jumping on the charter expansion bandwagon.  Andy Smarick, calls Bloomberg's plan to expand the number of charter schools "fantastic."  The NY Daily News prods the state Education Commissioner to create more of these "roaringly successful" schools.  And the NY Post, in their article praising the report, asks "only one question left: Will Albany let more of these better schools open?"

But I can't help but wonder if this is the appropriate response.  In the words of Lee Corso, I'd like to say "not so fast, my friends."

When you were a kid, did you ever wish you could eat pizza (or ice cream, or whatever your favorite food was?) as every meal?  You really, really loved it when you had pizza instead of broccoli or whatever boring food you had with most meals.  And you were absolutely convinced that the world would be a better place if only you could have pizza for breakfast, lunch, and dinner every day.

But now that you're grown up, you know that that's silly.  For a while, you'd be living the high life.  Maybe you'd love every piece of pizza you had for a week or two, but eventually, you'd get sick of it (and probably literally get sick too).

And I wonder if a similar issue doesn't exist with charter schools.  For a second, let's assume that the report is right: that NYC charter schools are outperforming traditional public schools in NYC.  The appropriate reaction to that is not "quick, build more charter schools" it is "I wonder if that will still be true if we build more charter schools."  Maybe it's my research training speaking, but I fail to understand those who cheer on policies before they're proven to work.  If we truly want what's best for our children and our society, we should be agnostic about which policy is best at the beginning of an experiment.

Unproductive cheerleading aside, I really do wonder what the optimum number of charter schools is.  Right now, I think there are around 80 charter schools and 1,000 traditional public schools in NYC.  There's some evidence that the former are doing better than the latter.  But will that be true if we have 400 charter schools and 800 others?  or 2,000 charter schools and 0 others (charter schools tend to be smaller, which is why the total number of schools keeps climbing in these hypotheticals)?  We can make some guesses, but nobody actually knows.  I can be convinced that it's worth exploring, but I cannot be convinced that doing as such will automatically yield spectacular results.  Here's why:

Scaling up is tricky.  Just like it's easier to cook a good meal for two than it is for 200, I wonder whether it's easier to create good charter schools for 5% of the population than it is for 50%.  Since charter schools are, for the most part anyway, highly decentralized this might be less of an issue -- if each one is self-contained, there aren't as many scale-up issues.  But, as I've argued before, one crucial aspect of the charter school model is that low-performing schools are quickly closed in order to make way for better ones.  And I wonder if this will be harder to monitor with 800 charters than 80.


Talent and funds are not unlimited.  Many of the most successful charter schools have received an enormous amount of philanthropical funding.  This may or may not be instrumental to their success, but I think they'd all agree that the talent they're able to attract is.  Charitable donations may or may not expand at the same rate that charter schools do, but I'd think it's extremely unlikely that the number of talented, hard-working people willing to dedicate themselves to working long hours for middling pay at a charter school likely is.  KIPP and other schools run nationwide searchers and rely heavily on Teach for America alums.  Well, TFA only accepts a few thousand people each year and only some fraction of those people will be willing to spend years three through ?? at a charter school.  How many more talented, dedicated, sacrificing people are out there?  I really have no idea.  There certainly aren't enough to staff the million plus schools in the country if we want the same level of talent/qualifications (and have the same level of turnover) that we do at the most famous charter chains, but I suppose it's possible that there are enough to staff more charters in NYC.  How many more?  I guess we'll have to wait and see.

Not all charters are created equal.  The charter report draws a distribution of the effects on test scores of all the charter schools in NYC.  Some do worse than traditional public schools, some do the same, and some do better.  Which type will we create if we expand the number of charter schools?  If funds and talent run out, two things could happen: 1.) we could create middling, or even poor, schools; and/or 2.) we could dilute talent and funds and currently successful schools -- ultimately hurting the children of NYC.  It's certainly possible that distribution of charter success will be the same if we double the number of charter schools -- in other words, the new schools we create might be the same as the ones we have currently -- but I see no reason to be sure of this.  It seems equally plausible that the new schools we create will fail to live up to the hope we have for them.

It seems almost certain that the number of charters in NYC will continue to grow.  And I, for one, am not going to pretend to know how that will pan out.  But I eagerly await the results.

Apples to Apples? Not Necessarily

Mike Petrilli has a good post over at Flypaper on the Hoxby et. al. charter school study that was recently released.  It's good to see I'm not the only one pointing out that the Wall St. Journal and others shouldn't be writing NYC charters don't cream -- because they probably do.

But I have to take issue with one thing that Petrilli writes about the study: "because it’s a gold-standard random-assignment study, we can be sure that it’s an apples to apples comparison".  Not so fast Mike . . . the random assignment part might well be true, but there are plenty of reasons to think that this isn't quite an apples-to-apples comparison.

First, in regards to the random assignment* designation: it may not actually be random assignment.  I have yet to hear back regarding the number of charter schools each kid applied to, but it's certainly possible that those who won a charter school lottery applied to more charters, on average, than those who lost a charter school lottery.  In other words, it's possible that the winners were more motivated than the losers to begin with.  On the other hand, it's also possible that losers were more likely to apply to charters with long waiting lists -- and it might be true that those with longer waiting lists are superior schools that attract superior students.

In addition to these possibilities, I can think of at least one other way that the study design could result in a comparison that is not quite apples-to-apples (note that the authors did address some of these in the paper):

Attrition may not be the same for charter enrollees as it is for the others.  22% of the lottery-losing students transferred to a school outside of NYC or a private school, while 14% of the charter students transferred to a traditional NYC public school.  While it's likely that both groups leave their current school due to some level of dissatisfaction, it may be the case that those who transfer from charter schools to another NYC school do so because they're struggling at their school and/or are encouraged to leave due to discipline or other problems.  Meanwhile, it may be the case that those who transfer from traditional public schools to non-NYC schools do so because they view their schools as failing them and think they can do better elsewhere.  In other words, it's plausible that the most motivated non-charter students transfer out while the least-motivated charter students transfer out.  The authors do take a look at this and conclude that there's no difference in the relationship between test scores and likelihood of leaving between the two groups, but there could be no difference in test scores but a large difference in motivation, attitude, family support, or a myriad of other factors that are more likely to lead to growth in test scores (which, remember, is the outcome variable in the study).


*For those of you without a research background, a random assignment study is just what it sounds like -- a study in which those being studied were randomly assigned to a control group or treatment group (for example, if we flipped a quarter for every kid who walked in the door and those who got tails were assigned to Mrs. Johnson's class and those who got heads were assigned to Mr. Smith's class).

(Why) Are NYC Charters Different?

Now that the Washington Post has also taken the findings of the latest charter school report as gospel, I think it's important that we examine some of the issues it raises more in-depth.

As Diane Ravitch points out, there a number of other studies finding blase or discouraging results for charters both nationally and in other cities.  So if we assume that the report's findings are correct (while I think the report skims across a lot of issues, it does seem fairly likely that charter schools are outperforming traditional public schools in NYC), the first question we should ask is why charters in NYC are different.  I can think of at least four reasons why NYC charters may be outperforming charters elsewhere when compared to other local schools:

1.) NYC schools are uniquely bad.  If the public schools in NYC are incredibly atrocious and dysfunctional, then it shouldn't be that hard to create a school that outperforms most local schools.  From experience, I can tell you that there are certainly a number of schools in NYC that are incredibly atrocious and poorly run.  While teaching there I regularly heard horror stories from other teachers (not to mention the things I experienced in my school) that should make anybody cringe.  I really can't say, though, that these horror stories make NYC schools any worse than those in Philly, Chicago, L.A., etc. -- I'm sure you can find more than any school's fair share of horror stories coming from other big cities as well.

2.) NYC charters more easily attract talent and funding.  NYC is, to many people, the most important city in the world.  There's certainly more talent and money floating around the city than there is in most other American cities.  I have no evidence either way on this, but it's certainly plausible that NYC charter schools both find it easier to attract both talent (both in terms of management and employees) and outside funding -- both when compared to charters in other cities and when compared to other NYC schools.  Especially since there are only around 80 charter schools in the city compared to around 1,000 or so traditional public schools.

3.) The NY Charter Review Board outperforms others.  It's certainly possible that the board that approves the creation of charter schools in NYC does a better job of screening and monitoring than do other boards.  One of our teachers during our pre-teaching summer training session was in the process of creating a charter school in Harlem.  From what he said, it sounded like an awfully involved process involving a heck of a lot of thought and work.  And maybe this has resulted in better planned and more successful charter schools.

4.) The other studies are all flawed.  In other words, maybe Hoxby et. al. are right and everybody else is wrong.  I'm sure all the other studies are flawed, but so is this one -- so I don't really find this idea convincing.  It's not out of the realm of possibility that charters are generally outperforming other schools and we just haven't proved it yet, but I find it more plausible that charters are performing differently in different places.

Sunday Commentary: Do NYC Charter Schools "Cream?"

Probably.  At least that seems to be the most sensible interpretation of the recently released report on NYC charter schools.  And it's important to note that the Wall St. Journal gets this dead wrong in their editorial on the report.  The main conclusion of the report was that students who enrolled in NYC charter schools performed much better than did students who tried, but failed, to enroll in these charter schools.  So I found it more than a bit odd that the Wall St. Journal's headline trumpeted something else.  Why lie when there's plenty of truth to extol?

Nowhere in the report is the claim made that charter schools do not "cream" (select only the top students, like one would cream the fat off of milk, for those unfamiliar with the lingo).  The claim that charter schools don't cream seems to be based on the fact that charter applicants are more likely to be black or poor than the average NYC student.  This is an important, and entirely unsurprising, piece of information.

But here's what they missed: the authors go out of their way to stress that they cannot accurately compare the prior test scores of charter school applicants to non-applicants, writing that "it is not possible to draw conclusions about how charter school applicants' achievement compares to that of students in New York City's traditional public schools" (p. 28).

Nonetheless, they display a chart on the same page with calculations that the average test score of charter school applicants is no different than the average score of non-applicants.  This despite the fact that applicants were much more likely to be poor or Black and much less likely to be White or Asian.  And any cursory glance at test scores in NYC will tell you that the average poor Black student scores far below the average non-poor White or Asian kid.  Which means that we should see average scores considerably below the citywide average from charter school applicants.  The fact that there is no difference would indicate, to me at least, that charter school applicants are out-performing other demographically similar students.

Similarly, the authors write on page 71 that students who are accepted into, but do not enroll in, a charter school have prior test scores no different from those who do enroll.  But, at the same time, admitted students who choose not to enroll are much more likely to be White or Asian.  Which raises two questions: 1.) If the authors had previously argued that there wasn't test data on enough students to make any claims about test scores, why are they now making claims about test scores? and 2.) If the non-enrolling admitted students are more likely to be White or Asian, would we not expect them to have higher previous test scores than do enrolling admitted students?

Besides, comparing test scores is, to some extent, beside the point.  It might very well matter more how motivated a student and their family is, what kind of support they are receiving at home, etc.  If you were running a school and your goal was for students to make as much progress as possible, which would be more important qualification for students: previous success on tests or the willingness to work hard?

In the end, we need to see both disaggregated data (e.g. comparing Black enrollees to Black non-enrollees) and comparisons of test scores and other variables controlling for various demographic variables before we can make any firm claim about NYC charter schools creaming students.  But the authors' position is firm: they can't tell.  And if you ask me, all signs indicate that they likely do.  Either way, the Wall St. Journal still needs run a correction.

Last, but certainly not least, should we care if charter schools cream?  If charter schools enroll more motivated or more capable students, might that not be a good thing (especially since they're clearly attracting students from the most at-risk demographics)?  The answer, of course, is that it depends.  It's hard to imagine that enrolling more motivated and higher achieving kids in a school wouldn't be a good thing for those who attend that school.  At the same time, there's a possibility that it might negatively influence kids in the surrounding schools that are losing the cream of the crop.  But it's hard to say to what extent either of these scenarios are playing out.  At this point, whether or not creaming makes charter schools bad is largely in the eyes of the beholder.

But, more importantly, it affects the way that charter schools should be analyzed -- because, ultimately, charter schools are supposed to serve as incubators of ideas that spread to other schools.  Which means that we should make an effort to figure out which of these ideas are worth replicating.  And if charter schools cream a little or a lot and we don't take this into account, it skews both the analysis of their performance and the analysis of which ideas should be replicated.  Which has the possibility of harming the students we ultimately want to help.

A Few Thoughts on the NYC Charter Schools Evaluation Project

The NY Times profiled a report on charter schools that was released today.  The group conducting the investigation was led by Caroline Hoxby, an economist based at Stanford's Hoover Institute who has published a number of articles in prestigious journals that find positive returns to school choice.  The report is written in plain English and is very easily readable even for those who have little background knowledge or statistical training (which was quite a surprise to me given the highly technical nature of most of her work).

The bottom line of the report is that students who enroll in charter schools do much better than those who enter lotteries to enroll in charter schools, but don't have their number picked out of the hat.  The methodology of comparing "lottery winners" to "lottery losers" is probably the best way to go when evaluating charter schools because you should end up with comparison and treatment groups that are demographically similar.  Though, of course, that doesn't mean it's perfect.  A few thoughts:

-It's unclear whether students who are lottery winners apply to more charter schools than those who are lottery losers (one could imagine that a more motivated parent would apply to 10 charters and be more likely to end up with their child in one of them than the parent who applies to 1).

-It's also  somewhat unclear how they treat a student who leaves or enrolls in a charter school in a subsequent year.  They mention that about 14% of students leave charter schools (p. 71), but I didn't notice a part where they addressed the latter.  (correction: the 14% figure is the percentage of treatment group students that have transferred to traditional public schools over the life of the study, at least 17% have transferred to other schools and 24% have dropped out of the study for various reasons (p. 72))

-The study points out that lottery losers are 32% more likely to transfer to a private school or a school outside of NYC than are lottery winners (p. 72), but I'm not sure they fully investigate the effects of this.  It sounds as though lottery-losing leavers have test scores that are no different than lottery-winning leavers, but this isn't really the full story.  Current test score is only one predictor of future test score, and since they're measuring growth in test scores, I'd have to imagine that motivation is a better predictor of growth in test scores.  And as far as I can tell, it's entirely feasible that more motivated students are more likely to leave the system if they're not enrolled in a charter school.

-They estimate the average growth in test scores for charter school students to be .12 (math) and .09 (reading) standard deviations above and beyond those of lottery losers each year (p. 42).  For one year, that's not particularly impressive, but they calculate that these effects add up over time to create quite a powerful effect.

-Perhaps more interesting are the effects they calculate for individual schools.  Charter schools are supposed to vary widely, so it shouldn't really be possible to generalize to all charter schools.  Here are how the effects of charter schools broke down in each subject (p. 57-59)

	reading	math
negative	8%	14%
0 - .1 SD	16%	17%
.1 - .2 SD	45%	59%
> .2 SD	31%	10%

Assuming that these measurements are accurate and that higher test scores reflect more meaningful growth as well (neither of which should be taken for granted), I'd argue that schools over .2 might be worth getting excited about and schools between .1 - .2 are probably doing something right.  Making those two dangerous assumptions, we might say that a majority of schools seem to be doing something right.  It also becomes evident why, as I've argued in the past, charter school advocates should advocate closing more charters.  Think how much higher the estimates would be if the bottom 20% or so that are getting returns that are negative or no different from zero were closed.

-Why are charter schools doing better (same two assumptions apply)?  The authors measure the effects of a wide variety of variables on page 64.  The most notable number is that adding 10 days to the school year boosts achievement .15 SD.  This is the only number that is both meaningful to me and statistically significant at traditional levels.  The average charter school has about 10 more days of school than the average traditional public school.  Meanwhile, the average charter school advantage is about .10-.11 SD.  Which raises the possibility that charters are doing better mostly because they have longer school years -- a reform which could very easily be replicated in all schools.

-You should ignore all the calculations regarding the benefits of attending a charter high school.  In the last year of data, there are four charter high schools in NYC.


I'll e-mail the team about the first two bullet points, hopefully I'll hear back -- I'll post the answers if and when I do.

Today's Random Thoughts

-When looking at the browser requirements for filling out the FAFSA, I noticed that anybody who regularly updates their computer is unable to fill one out online.  The FAFSA website can only handle up to Internet Explorer 7 and Firefox 2.0 -- they're up to versions 8 and 3.5 now, respectively.  This is something the Department of Education should've fixed months ago.  While I'm on the subject of the federal govt. updating their websites, why can't they combine the login systems for e-filing tax returns and applying for financial aid?  Why must one type in all their same tax info all over again?

-I continue to see little discussion of the fact that about 2/3 of teachers don't teach tested subjects when discussing merit pay.  Indeed, NYC recently passed out 12,000 teacher evaluations based on state test scores . . . there are 87,000 teachers in NYC.  Assuming these evaluations were accurate, how would we evaluate the other 75,000 teachers? 

-Speaking of merit pay and evaluating teachers using standardized test scores . . . many object to the latter on the grounds that we might fire a good teacher based on faulty measures.  What if, instead, we reward mediocre teachers for high performance based on test scores -- breaking the bank and not helping our schools at the same time?  And, it's clear that faulty measures are being used (check out the last graph in this piece if you believe at all that NYC's school grading system is statistically valid), leading to 97% of NYC schools receiving a grade of A or B.  New York's faulty tests also resulted in a doubling of the budget for teacher bonuses this year.  Merit pay is supposed to be a fiscally efficient reform, but it's not under these types of conditions.

-The AP has a story on private investment in charter schools.  I wonder how this will play out if charter schools continue to expand.  If private monies dry up, we might not hear much more about it.  But if private monies start going disproportionately to charter schools, it might be hard for public schools to compete.  On the other hand, it might make charter schools even better, which would probably be a good thing for those attending charters.

-I wrote before that we seem to be seeing a charterization of urban public districts.  Not only are charter schools spreading, but charter-like public schools (read: small, specialized schools often not for just one neighborhood) are as well.  The NY Post has a little blurb on the vast array of specialties that this year's crop of new schools offer.  If this continues I wonder what the ramifications of having community schools in the suburbs and beyond and specialty schools in the inner-city will be.

Today's Random Thoughts

-Flypaper has a piece on the startling drop in enrollment in traditional public schools in Washington D.C. (down 17% from last spring). It seems enrollment is at 37,000 right now compared to a little over 44K last year. Apparently everybody is leaving city schools for charter schools. Except for one little detail that the Washington Post reports: not everybody enrolls on the first day of school. In fact, last year at this time enrollment stood at 15,000. I'm not going to pretend to know whether or not 7,000 more students will enroll before the Oct. 1st deadline (deadline for funding purposes, that is), but I think I'd wait a couple weeks before panicking if I were running the D.C. schools.

Besides, the more interesting part of the piece (to me, anyway) is why enrollment has surged from 15K to 37K. Not only was the start of enrollment moved up a couple months, but principals have been hosting BBQs and recruiting parents at other community events. I've always wondered exactly how traditional public schools would respond to competition from charter schools. It seems that this might be one way D.C. has chosen to respond. Of course, for the good of society I think we'd hope they're response would to make their schools better. But maybe BBQs are a first step in that direction.

-Tennessee has had one of the more restrictive charter school laws in the country. It's my understanding that only students who fail state tests or attend schools deemed in need of improvement are eligible to enroll in charter schools. And we have relatively few of them here. But of those few, one has had trouble dropping kids off on time (9pm one night), and another is now on academic probation for low test scores (story here). I was told a while back that the KIPP here is one of the lowest performing in the country -- I have absolutely no evidence to present of this, it's simply what I heard from somebody who volunteered there. And I can't help but wonder if severely limiting who can enroll in charters in this manner makes it much more difficult for charters to succeed. Given that these are the kids that charter laws are supposed to help, that would be somewhat troubling.

-the SAT results are being parsed, and it doesn't look good for anybody who was hoping to see a shrinking achievement gap. Aaron Pallas points out that the average Asian student in NYC outscores the average Black student in NYC by 151 points in math and Whites outscore Hispanics by 108 points in writing. Elsewhere, Checker Finn argues that the lack of closure in the gap between students of different races and socioeconomic statuses means that reform hasn't yet penetrated high schools. I think there's more to it than that, but most of the recent reform has been focused on grades 3-8.

Sunday Commentary: Why Charter School Advocates Should Advocate Closing Charter Schools

The biggest difference between the American system of education now and back in the 20th century is probably the spread of charter schools. Ten years ago most had never heard of them, but now they seem to be the first two words out of every politician's (of both parties) lips when they discuss education reform.

$5 billion of stimulus money has been made contingent on a number of policies being in place, perhaps most notably liberal laws regarding the founding of charter schools -- which has resulted in at least 7 states recently deciding to allow more charters. I have no idea what we'll think of charter schools in fifty years, but they seem a lock to proliferate further for at least the next five or ten. It seems that everybody (including both Bush and Obama and both Spellings and Duncan) is in favor of closing down more "failing" schools and opening up more charters.

But traditional public schools shouldn't be the only ones that are closed when failing. Advocates of charter schools should be advocating that low-performing charter schools be promptly closed as well. Why? Two main reasons, one theoretical and one practical:

1.) The theory behind charter schools is based largely on free market principles. Which is part of the reason they're so attractive on paper. The theory is that schools pop up in areas that are underserved by good public schools, then compete and win students over because they offer a superior education. Then more charter schools start and offer more competition -- ensuring that only the strong survive. Those that can raise their game to a new level draw students, those that cannot wither and die. In this way, we're left with only the best schools -- all others simply cease to exist. Like I said, it's quite attractive on paper. But one of problems with the plan is that it relies on fairly frequent school closures to work. And school closures are not usually eagerly anticipated by those involved with a school. But if charter school advocates are truly interested in charter schools succeeding they'll advocate that all low-performing charters be closed sooner rather than later to make room for more successful charters.

2.) We frequently hear about charter schools that have done something miraculous or closed the achievement gap. As such, many tend to believe that charters get results that are far superior to traditional public schools. But there is zero empirical research that supports this on a nationwide level. Every study comparing charters to traditional public schools finds somewhere near (and sometimes less than) zero advantage to attending a charter school. Maybe this is cynical to suggest, but if charters are your pet project and you're intent on proving their superiority, there's a fairly easy way to do this -- work to close all those charter schools that are seeing below average gains in student test scores. If we were only left with the KIPPesque charter schools that can turn water into wine, think what the results of national studies would look like -- then what politician, or citizen for that matter, could dispute the superiority of charters?

Still not convinced that the best way to push the propagation of more charter schools is to advocate closing more charters? Imagine a world in which charters fail: charter schools draw students because of proximity and special programs rather than performance, see their young, enthusiastic teaching corps grow old and weary, and activist parents that choose charters resist any closure plan. None of these would happen if charters were closed with the ruthless efficiency the theory would seem to dictate.

Charter Schools Unionize . . . And then What?

The NY Times has a piece this morning about charter school unionization that's worth reading.

It starts thusly: "Dissatisfied with long hours, churning turnover and, in some cases, lower pay than instructors at other public schools, an increasing number of teachers at charter schools are unionizing." Randi Weingarten strikes a similar note, saying that the unionization of some big-name charter schools this year is a "precursor" of things to come.

Meanwhile, one charter school advocate says “They’ll have a success here and there but unionized charters will continue to be a small part of the movement.”

I'm not sure that last part is true. I don't know, maybe he's right -- maybe unions and charter schools will rarely interact -- but for a lot of these schools it makes quite a bit of sense that teachers would eventually want to unionize. A lot of the so-called "no excuses" schools rely on recruiting bright, young teachers (often with no families) who are willing to work crazy hours for a few years before moving on to something bigger and better -- rather like joining the peace corps, in other words.

But what happens when some of these teachers get a little older, get married, have kids, tire of working 80 hour weeks, but still believe strongly in the mission of their school? “I was frustrated with all the turnover among staff, with the lack of teacher input, with working longer and harder than teachers at other schools and earning less,” says one teacher. And I can see that turning into a recurring theme.

We can only have so many schools relying on overworked young idealists before that labor pool will start to dry up. And a school can operate with such a staff for so long before the staff starts to age. And both occurrences almost have to lead to more unionization.

Charter schools are so new that nobody knows what will happen with them over the next five years, yet alone the next fifty. I can't imagine unions playing that big of a role in the next five years, but I can in the next fifty. What happens if, fifty years from now, the vast majority of charter schools are unionized? Will it ruin the charter school movement? Save the charter school movement?

I think that largely depends on two things: how unions interact with the administrations of individual schools and how many charter schools are shut down when they stumble. While I'm largely skeptical of any claims that unions are the only reason anything is wrong with American schools, if enough schools unionize there are bound to be some where poisonous administration-staff relationships severely hurt the performance of the school. If this becomes the norm, then the charter school movement might be stopped in its tracks. But, if charters that struggle are quickly closed then all bets are off.

Personally, my guess is that unions won't strangle the charter school movement. I tend to believe that in most schools the lack or presence of a union has more subtle effects than many would believe. Particularly if individual charter schools negotiate many aspects of the contract with the teachers only in that school, I don't think things will change all that much -- which, of course, given our lack of knowledge on charter schools, we can't be sure is good or bad.

Charter School Teacher Retention

Debra Viadero reported last week on a paper by Dave Stuit and Tom Smith finding that charter school teachers are 230% more likely to leave the field (an updated version actually pegs it at 237%) at the end of the year than are teachers in traditional public schools, based on data from 2003-04.

Given that both writers are from Vanderbilt and that I find the topic interesting, I thought I'd look into it further. It's a conference paper -- not a finished paper that's been peer-reviewed and published in a journal -- so I'm not going to get into significant detail. While I'm sure it's not perfect and that it will undergo further revisions, I will say that the methodology seems pretty straightforward and that I trust both of the writers to investigate things rigorously and interpret them correctly, so my guess is that when the final version is released most of these figures will remain about the same. That said, let's get to it.

The most striking finding was that charter school teachers were more than twice as likely to leave the field as are teachers in traditional public schools -- the raw numbers are 14.1% and 7.0% (please note that the 230% estimate is based on log-odds ratios calculated using Hierachical General Linear Models controlling for clustering in schools*). They also calculated that charter school teachers were 113% as likely to transfer to another school.

They also found turnover nearly twice as high in schools that were new start-ups versus schools that had been converted to charter schools -- which makes sense because a new start-up would have more growing pains and instability than would a converted school.

But the most important question is why charter school teachers were more likely to leave the profession. Based on survey responses from those who left, charter school teachers were about twice as likely to report leaving for better salary and benefits (46%/22%), dissatisfaction iwth school (51%/24%), or due to school staffing action (40%/20%), and more than three times as likely to report leaving to pursue additional coursework outside education (27%/8%). They were also about one-third as likely to report leaving due to retirement (14%/38%).

In short, it seems that more teachers are leaving to move on to bigger and better things. This makes sense if we consider how many charter schools pluck people to serve as almost missionaries for a few years (think KIPP and their use of TFA fellows). I would expect that charter schools employ more people who are teaching as a way to give back to the community for a few years and fewer people that want to make a career out of teaching than are traditional public schools. Indeed, if we look at the demographics, charter schools employ more people under the age of 30 (34%/20%) and fewer over the age of 50 (18%/29%) than do traditional public schools. Teachers are also much less likely to be certified (67%/90%), and one would imagine that investing the time and effort into getting certified means that a teacher both intends to stay longer and will now be more likely to stay longer.

I've only just scratched the surface of all the paper examines and discusses, but I think the difference in attrition between sectors is worth thinking through. I could see the interpretation that this means charter schools need to work harder to retain teachers. I could also see an argument that attrition will hold back charters from achieving the success they could. But my main interpretation would be that charters tend to employ different types of teachers than do traditional public schools -- and these teachers, put simply, are less likely to stay in the profession for a long time (which could be exacerbated by the amount of time and effort required to teach in some of these charter schools). And I think the largest question it raises is about the replicability and scalability of charter schools.

If we continue to expand the number of charter schools and replicate those that have been more successful, will they continue to operate by employing teachers that don't make a career out of teaching? If so, how many people are out there that are both capable of being good teachers and want to teach for a few years?


*there was some confusion over how the 230% figure (now 237%) was derived on the Inside School Research site. After consultation with the author it's been explained that it was done thusly: a log-odds ratio of 3.368 was calculated. An odds ratio of 1 would mean that teachers were equally likely to leave the profession from both sectors. A ratio of 3.368 means that those in charter schools were 3.368-1 (=2.37, move the decimal point so that it turns into 237) percent more likely to leave.

Is it Easier to Close a Charter School?

Closing a traditional public school is rarely an easy task. Cities with declining populations have had one heck of a time closing neighborhood schools -- I can't count how many times a planned school closing has spawned a community uprising. And when school board members are elected by district it's even tougher -- what're the odds of getting re-elected if you let a school in your district be shut down despite raucous opposition from a group of concerned citizens?

While traditional public schools are often built to remain a permanent (at least for the foreseeable future) fixture of the community, charter schools are necessarily designed to be opened and closed with much greater frequency. The market theory underlying charter schools dictates that only the strong shall survive. So, in order for charters to live up to the promise of this theory they must be shuttered with greater frequency than other schools.

The expectation that charter schools will be closed unless they meet certain expectations should make them easier to close, but I'm not sure that it does. On the one hand, there's not usually a neighborhood or large alumni base rallying to save a charter school on its way out. On the other hand, charters to attract the students of more socially active parents. Take this article (hat tip: Alexander Russo) in yesterday's Fresno Bee, for example. The local KIPP school there is on the verge of closing, and parents are up in arms.

Granted, the school's not being shuttered for low performance -- it's the second highest-performing middle school in the district based on the current grading system -- but rather a set of complex circumstances. It seems that the school has defaulted on a construction loan that they thought would be repaid by a state grant. Meanwhile, the state won't give them the grant until the city signs off on it. And the city won't sign off on it until the school fixes a number of problems that were noted in a recent report -- including charges of corporal punishment by the principal (who has resigned), the hiring of uncredentialed teachers and the lack of fingerprinting before hiring staff.

Nevertheless, I wonder whether closing a charter is easier, on average, than closing a traditional public school. I have to believe that more people arise when a neighborhood school is threatened, but which would you fight harder: the closing of your neighborhood school or the closing of a school you had chosen yourself and for which you had to submit yourself to three hours of interviews, woken up at dawn every day to drive your child to school, and signed a contract that you would read with your child every night?