Thoughts on Education Policy

Education and the Declining Median Class

noreply@blogger.com (Corey Bunje Bower) — Mon, 26 Mar 2012 12:03:00 +0000

A column a while back by David Brooks and numerous other reviews of Charles Murray's new book raise an issue I've been meaning to write about: the growing gulf between different classes (or "social tribes," as Brooks labels them) in the U.S. -- not just in earnings, which we hear a lot about, but in both achievement and a number of behaviors related to achievement. As Brooks writes about the White population Murray discusses:

There are vast behavioral gaps between the educated upper tribe (20 percent of the country) and the lower tribe (30 percent of the country) . . . Roughly 7 percent of the white kids in the upper tribe are born out of wedlock, compared with roughly 45 percent of the kids in the lower tribe. In the upper tribe, nearly every man aged 30 to 49 is in the labor force. In the lower tribe, men in their prime working ages have been steadily dropping out of the labor force, in good times and bad. People in the lower tribe are much less likely to get married, less likely to go to church, less likely to be active in their communities, more likely to watch TV excessively, more likely to be obese.

The NY Times followed up with a story about the achievement gap growing between rich and poor students in the United States. The article was based largely on two chapters from a book that was released last fall by the Russell Sage Foundation (which, as I've previously mentioned, I highly recommend).

In one chapter, Sean Reardon from Stanford finds that "the achievement gap between children from high- and low-income families is roughly 30 to 40 percent larger among children born in 2001 than among those born twenty-five years earlier" and that the gap between students from families in the 90th and 10th percentiles of income is now "nearly twice as large" as the gap between blacks and whites (the opposite was true 50 years ago).

Since I've spent my entire post-college life either trying to teach low-income students or researching low-income students, the chapter jumped out at me when I first saw a preview of it last summer. The fact that the achievement gap between rich and poor was actually growing (while the black-white achievement gap has essentially plateaued the past 20 years), jibed with what I've read about, and seen in, both our schools and our society.

But then, when I was preparing to discuss the chapter with my students last fall, I took a closer look at the trends. And this is what I discovered: the real growth in the gap between rich and poor isn't actually between the richest and the poorest, it's between the richest and those in the middle.

I've pasted one chart from the Reardon chapter to show what I mean. The gap between those at the 50th percentile and those at the 10th percentile -- the middle-poor gap, if you will (represented by the dashed line) has been fairly steady. The gap between those at the 90th percentile and those at the 10th percentile, meanwhile -- the rich-middle gap, if you will (represented by the solid line) -- has grown rapidly, from under a quarter of a standard deviation for children born in the 1940's to almost three-quarters of a standard deviation for children born in the 21st century.

I actually missed that point when I first skimmed the chapter, but to me it's maybe the most pressing policy issue of the next few decades: how will middle-income Americans work and live?

As somebody who studies the effects of poverty, I'm pre-disposed to believe that poverty, and the low performance of children living in poverty, poses the largest educational problem. And there's no doubt that that problem is large. Even if the poorest kids aren't falling further behind the middle-income students, they're still far behind where they ought to be and farther behind the wealthiest kids than before.

But if the middle of the income distribution falls far behind as well, that could leave us in really serious trouble. This chart of wage growth that EPI offered in response to Brooks' op-ed is one way of looking at the issue. The wages of the working poor are awful, but it's the hourly wages of the middle-income folks that have seen the lowest growth in the last 20 years. Indeed, the median household made only slightly more in 2010 than it did in 1978 ($49K vs. $46K -- Table H-6) when adjusting for inflation.

As the table below indicates, from 1991-2010, median income by education level declined considerably for the nearly 60% of the population with a high school diploma, some college, or an associates degree, held steady for the 20% or so with a bachelor's degree and increased for the 10% or so with a master's or professional degree (to reduce visual clutter, I leave off the 13% or so of Americans who did not reach, or did graduate from, high school and the 1% who earned a doctorate degree*). Now, education levels increased a bit during the time -- so part of the explanation may be that there were more people in each of the higher groups -- but not by enough to change the fact that Americans with median levels of education are earning less now than they were 20 years ago.

Median income by education level. Source: US Census, Table H-13

It's hard to imagine a burgeoning economy in any country that sees no real income growth for those at the middle of the income distribution. But Murray, Brooks, and others (including liberals as well) also note other worrying trends concerning health, marriage, childbirth, etc.

Indeed, another recent NY Times article reported that the majority of babies born to women under 30 are now born out of wedlock. But more important are the different rates by social class: "About 92 percent of college-educated women are married when they give birth, compared with 62 percent of women with some post-secondary schooling and 43 percent of women with a high school diploma or less". And while the number of babies born out of wedlock has risen for all three groups, it's the middle group that's seen the starkest increase: in 1990 only 11% of children born to women in their 20's with some college were born out of wedlock, but by 2009 that number had more than tripled to 34%.

Again, the problem is more pervasive among the least-educated women (the comparable number is 51%), but the largest change is in the middle of the income distribution.

Now, you may have noticed that I titled the post "Median Class" instead of "Middle Class," and that's because what a lot of people define as "middle class" isn't actually comprised of people in the middle of the income distribution. People often talk about college-educated adults belonging to the middle class, but fewer than 30% of adults have a 4-year degree. Depending on which model of social class one uses, those in the middle of the income distribution (those who fall right around median) are usually classified as lower-middle-class or working class. And I want to be clear that I'm talking about those who fall right around the median of the income distribution.

It seems to me that a large part of the challenge is economic. Jobs that pay high wages to employees without high levels of educational attainment are fast disappearing (one could write "high wage/low skill" jobs as shorthand, but I don't possess the skills for most of these jobs so that seems inaccurate to me). We still have a not-insignificant number of jobs in construction, trucking, the trades, manufacturing, and so forth that pay fairly well, but the number of those types of jobs has declined dramatically -- largely due to globalization and/or technology -- in recent decades. And it seems unlikely that this trend will dramatically reverse. In other words, it seems unrealistic to expect anywhere near 70% of the population to find stable employment with decent wages without a 4-year degree.

The result, it seems to me, is that instability and low wages are no longer the domain of only the poorest Americans. And it seems reasonable to assume that the growth in the rich-middle achievement gap is due, at least in part, to the spread of this job instability to the median earners.

In short, it seems like there's a growing bulge in the middle of the income and education distributions that is lost. Fewer and fewer can make a good living without a college degree. They're falling further behind the wealthy academically. And to make matters worse (to be intentionally and melodramatically blunt), recent reports say they're increasingly divorced, fat, and lazy as well.

That's worrying. But what really worries me is that I don't see an easy solution. Brooks' integration idea may be a small step in the right direction, and improving our educational system would certainly be another. But neither seems likely to prevent the problem from getting worse in the next 20 or so years. It seems unlikely we can do any, yet alone all, of the following in a short period of time:

1.) dramatically increase the number of stable, high-paying jobs for those without college degrees
2.) dramatically** increase the number of college-educated adults while also increasing the number of stable, well-paying jobs for those with degrees accordingly
3.) reverse social trends and encourage more two-parent households, more civic engagement, less obesity, etc.

But I hope I'm wrong. I hope 20 years from now I'm writing about the resurgence of the median class and not about the spread of poverty to children of middle-income households. Either way, I think recent data indicate we need to adjust our focus when we discuss ways to boost performance of the lowest achievers. If we want to focus research and policy on those lagging behind, we need to broaden our scope beyond just the 10 or 20% lowest-income Americans. Those in the middle aren't doing that much better.

*median income increased slightly from 1991-2010 for those with less than a 9th grade education -- from $20,640 to $21,254, decreased from $27,375 to $24,787 for those who attended, but did not graduate from, high school, and also slightly decreased for those with a doctorate as well -- from $121,693 to $119,825.

**"dramatically," in this case, does not mean the 10 percentage point increase we've seen over the past 25 years for adults over age 25 and especially does not the 10 percentage point increase we've seen over the past 35 years for those aged 25-29

Thoughts on "Educational Productivity"

noreply@blogger.com (Corey Bunje Bower) — Wed, 14 Mar 2012 08:22:00 +0000

Last week, Matthew Ladner produced a stunning chart showing an "implosion" in our nation's educational productivity. Productivity here seems to be defined as ratio of per-pupil expenditures on public education to average NAEP test scores. The former has tripled since 1970 while the latter has essentially remained flat for the upper grades. I'm not sure of the impetus behind that particular post, but the Bush Center has written a similar post that makes all the same mistakes.

Before I delve into those mistakes, I'll point out that they've also created a nifty website that allows people to compare students' standardized test scores in any district's to the scores of other students in the state, nation, and world.*

So, what's wrong with comparing spending to achievement? Seems straightforward. And the graph is certainly compelling. But, alas, the statistics that seem the most straightforward are often the least useful. Among other issues:

1.) Spending and test scores are on different scales. Spending can multiply almost infinitely while the test scores have a ceiling. In the chart on the site, the average 17 year-old scored 306 out of 500 on the NAEP math test in 2008. Which means that even if every kid in the country earned a perfect score the next time around, the average score would only increase about 63%. Since school spending has tripled, the ratio of spending to achievement would still be far greater now than it was 40 years ago.

2.) Why would we assume that it takes the same level of effort for a school to get a student to earn a certain score now as it did in 1970? A zillion factors other than education spending influence achievement levels. If parenting ability, economic circumstances, living conditions, and such increased dramatically then we shouldn't think it's miraculous if scores increase with no additional school spending. Similarly, if societal conditions worsen in some way, then it would necessarily mean that more effort is necessary to achieve the same scores. I have no idea whether it's now harder or easier to get the average 17 year-old to score 306 on the NAEP now than it was in 1970, but we'd need to know that answer to accurately measure educational productivity.

3.) Why would we assume that the same level of spending is commensurate with the same level of effort on behalf of districts now as it was in 1970? The economic and social context of schooling is dramatically different. Perhaps most importantly, the number of women in the workforce -- particularly in fields outside of education -- has exploded. Simple economics dictates that it must cost more to buy the services of an equally qualified teacher.

4.) Test scores were not the sole goal of that increased spending. Surely, we also aimed to increase the number of high school and college graduates (I don't have the HS stats handy, but almost twice as many 25-29 year olds have bachelor's degrees now as did in 1970 (though it's increased by only about a third since 1975)). I think a reasonable argument could be made that it's increasingly costly to get each additional student to graduate (i.e. moving the HS graduation rate from 50 to 60% is easier than moving it from 80 to 90%), so we might not expect the same returns per dollar on those measures. And, surely, we also aimed to improve many other skills (e.g. critical thinking, physical/emotional health, social skills, art appreciation, etc.) that aren't measured by the math and reading tests listed.

So, should Ladner's alarming chart worry us? I wouldn't dismiss it out of hand. I wouldn't be the least bit shocked if our returns to effort and spending have decreased the past 40 years. But we can't tell whether productivity has decreased, remained steady, or increased by looking at that chart. The most compelling figures he presents are the large increases in non-teacher staff in schools, but some unknown number of support staff are certainly invaluable so even that doesn't prove all that much.

And, by the way, those same four problems apply to any international comparison of a simple spending : test score ratio. Were we to completely eliminate schools, culture, society, and a myriad of other contextual factors would still produce kids who scored much higher and lower on tests in different countries; it would be harder and easier and cheaper and more expensive to change that in different countries; and each country emphasizes different outcomes to a different degree.

I'll be the last person to argue that our nation's schools are just fine -- we face countless problems with a nearly infinite number of solutions -- so please don't interpret my criticism as an argument for the status quo. I hate the status quo. But also realize that Ladner's chart gives us exactly zero information about what ails our schools.

*The "world" here is 25 developed nations. On another note, here's a fun game: they don't seem to have compiled the list of the top-performing districts, so go see which ones you think might rank highest. So far, I've found:

-The average student in Chappaqua, NY outscores 89% of students internationally in reading and 82% in math

-The average student in Chatham, NJ outscores 88% of students internationally in reading

-The average student in Brookline, MA outscores 77% in math.

Is Teacher Quality Really Causing the Achievement Gap?

noreply@blogger.com (Corey Bunje Bower) — Sat, 25 Feb 2012 16:04:00 +0000

Yesterday, the NY Times released the value-added scores of thousands of teachers over the past five years. Before and immediately after the release, people seemed to mostly argue the merit of the decision to release the data. But I have a substantive question about the data.

What caught my eye was that, according to the NYT analysis of this particular set of scores, good teachers are evenly distributed between high-poverty and low-poverty schools. From the article:

there was no relationship between a school’s demographics and its number of high- or low-performing teachers: 26 percent of math teachers serving the poorest of students had high scores, as did 27 percent of teachers of the wealthiest.

The LA Times reported a roughly a similar situation in LA when they released teachers' scores a couple years ago. Which is really quite shocking in a number of ways. Most notably, researchers and practitioners have long assumed that lower-poverty schools had worse teachers than higher-poverty schools -- past studies have repeatedly found that teachers in high-poverty schools are less experienced, turn over at a much higher rate, score lower on achievement tests, attend less selective colleges, etc. Accordingly, at least part of the theory of action behind the teacher quality movement has been that giving low-income students teachers who are as good as or better than those in higher-income schools would significantly narrow the achievement gap.

But these two measures of teacher quality indicate there may be no major differences between low- and high-poverty schools, while we know that large gaps in achievement still exist between low-income and high-income students. Which means at least one of two things.

1.) Differences in teacher quality are not a major driver of the achievement gap.

2.) These value-added scores are not a good measure of teacher quality.

I don't think anybody seriously doubts -- or at least that anybody serious doubts -- that some teachers are much better than others and that the best teachers can make a large difference. But if quality teachers, according to these value-added measure, are roughly evenly distributed between high- and low-poverty schools in LA at the same time that we see differences between high- and low-income students growing, then improving the quality of teachers (again,as indicated by these value-added measures) in high-poverty schools seems unlikely to close the achievement gap. Either other factors influence achievement far more, the effects of quality teachers on students are much less direct than many assume, or what we're measuring isn't what matters.

In short, these data indicate that we need to broaden our focus beyond teacher quality and/or re-evaluate the way we're currently measuring teacher quality.

How Education Research is like Football Research

noreply@blogger.com (Corey Bunje Bower) — Wed, 08 Feb 2012 11:14:00 +0000

In the fall, I wrote about an instance in which outsiders may be needed in education reform. Today I'll give you an example of how outsiders can also be dangerous (though this one pertains more to research).

Perhaps the largest change in educational research over the past decade or so has been the sizable increase in large-scale quantitative research, a fair amount of which is conducted by researchers outside of ed schools. Like any change, this has resulted in both positives and negatives. But one thing that worries me is that I consistently notice the people who are most worried about statistical rigor in quantitative analyses (both inside and outside of ed schools) tend to be less concerned with understanding the context and processes of schooling.

And that's incredibly dangerous.

Methodology, statistics, and technical skills are very, very important in the development of good research. But without a proper understanding of how schools work and what is actually happening on the ground, one can't expect to ask the right questions. And if one fails to ask the right questions, it really doesn't matter how complex and rigorous their analysis is because the answers to those questions are meaningless.

Here's one example of how such a process can unfold -- it's completely unrelated to ed policy, but I still think it's illustrative. The Freakonomics Blog posted a brief discussion yesterday of the ending to the Super Bowl. The post said two things (paraphrasing, of course):

1.) Isn't it amazing that the coaches of both teams realized that the Giants scoring a touchdown with about a minute left was actually a better outcome for the Patriots? The Patriots' coaches tried to let the Giants run the ball into the endzone while the Giants' coaches instructed their players not to score a TD. These counter-intuitive behaviors are an excellent example of game theory properly implemented.

2.) But then the Giants failed to take game theory into account when attempting their two point conversion. Wouldn't it have been much better for them to run time off the clock instead of trying to score to go up 6 points instead of 4? They might've been able to kill 20 seconds by running the ball 95 yards backwards and around in circles, and certainly being up 4 with 40 seconds left is better than being up 6 with a minute left. Why didn't the coaches think of this?

There's some clever thinking going on here. Yes, this is an interesting application of game theory. And, yes, running 20 seconds off the clock would've been a better strategy. So the application of economic theory to the situation is exemplary. In a short space, there's a cogent analysis and a provocative question. But there are two fatal flaws.

1.) Both coaches did not apply game theory. Tom Coughlin, the Giants' coach, said he preferred that the team take the guaranteed six points to running down the clock. So let's hold off on patting him on the back for correctly applying game theory.

2.) More importantly, the clock doesn't run during two-point conversions. The Giants could've run around in circles for ten minutes, and there still would've been exactly 59 seconds left on the clock.

So, what we have here, is a smart professor who's well-trained in economic theory and statistics. This training has allowed him to make an important insight about a football game and ask an interesting question. Except that he doesn't actually seem to know much about the rules of football or the context of the situation. Which has rendered his question moot.

And I see the same thing (in a much less dramatic and much less foolish) way happening in education research. Smart people with training in other fields and disciplines and serious methodological credentials come into the field and find some low-hanging fruit ripe for picking. At first, this seems like a great idea. We can never have too many smart, well-trained researchers in education. And the eye of the outsider can be sharp. But then the research starts and we realize that somebody can be smart and well-trained but, at the same time, fail to truly understand how schools work and the contexts under which students, teachers, principals, schools, etc. operate. And then we get smart, well-trained people asking the wrong questions (or interpreting their findings in silly ways). And that neither advances the field nor helps us improve our educational system.

Let's bring the analogy back to football. Let's say that Football was a field in many Universities. Grad students train under faculty who work for Schools of Football and/or Departments of Football Policy, Football Leadership, Football Teaching & Learning, Football Evaluation, Football Foundations, Football Studies, and so on. And most of the research on football is conducted by faculty and grad students from these schools and departments. There's no reason why an economist shouldn't do a study on the costs and benefits of attending school on a football scholarship; why a psychologist shouldn't conduct a study on the impact of playing football on one's personality; or a sociologist shouldn't conduct a study of the impact of playing football on one's social capital. But in order to do these studies well, they first need to understand how the game of football is played, what a player does on the field, how much he practices, and so on. Otherwise they're just chucking their theories against a wall and hoping one sticks somewhere.

So, to all the smart economists, psychologists, sociologists, etc. out there who wish to conduct the research on education: Welcome, we'd love to hear your insights and figure out if we can apply your theories and methods to help us advance our field and improve our schools. But before claiming that you've solved a problem none of us have been able to for the last 100 years, take some time to learn how schools operate. Read a massive and wide-ranging stack of literature. Go visit some schools. Talk to people who work in schools and education departments. Talk to people who study those who work in schools and education departments. Then begin your research.

At the very least, that should save you the embarrassment of asking students how many touchdowns they need to score in order to hit a home run on their fourth grade reading test.

The Logistics of "Thinning Out" Bad Teachers

noreply@blogger.com (Corey Bunje Bower) — Tue, 07 Feb 2012 10:38:00 +0000

Nick Kristof recently wrote another column calling for more high-quality teachers based on the latest paper on value-added measures of teacher quality. There's a whole lot to discuss about both the column and the research paper, but let me focus for a minute on one small part of it.

Near the end of the column, Kristof writes that "If we want to recruit and retain the best teachers, we simply have to pay more — while also more aggressively thinning out those who don’t succeed. It’s worth it." Recruiting, retaining, paying (and training, which is left out of this sentence) are all complex endeavors, but the "thinning out" part of the equation is often taken for granted.

Here's my question for Kristof: even if (and that's a big if) we can find a fair, accurate, and agreeable way to identify and dismiss the worst teachers, how many teachers are we actually going to dismiss in such a scenario?

The first question would obviously be whether we need to fire the bottom 5%, 10%, 25% or some other number. That's up for discussion.

But the logistical question then, is how many teachers among the bottom X% 1.) can be readily identified and 2.) are planning on teaching again next year. This will differ greatly by school and district, but in some places, this is going to be a very small number.Why? Let's take a look at what the research says.

First, research consistently finds that it takes 3-5 years for a teacher to reach their potential. So a good number of the lowest-performing teachers are simply going to be novices who will be better teachers next year. We don't want to fire a first-year teacher who was in the bottom X% if we have reason to believe they'll be a really good teacher in a couple of years. That would be incredibly counterproductive.

Second, research has consistently found that value-added measures of teacher effectiveness bounce around considerably from year to year -- particularly for teachers who teach a small number of students (e.g. a 4th grade reading teacher with 18 students versus an 8th grade math teacher with 150 students). At least one paper has found that averaging scores over three years provides a much better, and more stable, estimate of teacher performance than does any single-year estimate.

Third, a number of recent papers have found that, at least in the first few years, many of the least successful teachers exit teaching. This makes sense -- if you start a new career and find yourself completely overwhelmed, you're not likely stay very long.

Fourth, teacher attrition is exceedingly high in many high-poverty schools. The general consensus is that about half of urban teachers leave the field within their first 3 years.

So, what does this mean? We probably don't want to fire a whole lot of teachers in the first 3-5 years of their career because a.) they're still learning and improving; b.) we can't be that sure who the worst teachers are anyway; and c.) a good portion of the catastrophically bad teachers are self-selecting out of the field anyway. If we give discount the first two years, when teachers are still learning their craft, and then take three more years to compute accurate value-added scores, it would only be teachers who'd taught for 5+ years who would really be ripe for firing due to low value-added scores.

Which means that the main herd we're trying to thin is the teachers who've made it through those first few years, reached their potential, and for whom we have accurate value-added estimates. But how many teachers is that? When I looked at high-poverty NYC middle schools a few years, I found that in the average school, only 1/3 of teachers had 5 or more years of experience.

Let's say that we're very confident in our ability to recruit and retain teachers who are better than our current teaching force and so we decide to fire all below average teachers (a full 50%) -- which would be a far more aggressive plan than any I've seen proposed. First, the majority of these below average teachers are novices who are still improving and for whom we don't have particularly good estimates of ability. Given that the majority of struggling beginning teachers either improve or self-select out of the profession, let's estimate that 2/3 of all teachers in their first 5 years are identified as below average teachers. This would mean that only 1/6 of all teachers in their sixth year and beyond are below average teachers. And since only 1/3 of teachers are in their sixth year or beyond, this would mean that only 1/18 of all teachers would both have 5+ years of experience and be rated below average. This is a little under 6% of all teachers.

The average school in my sample had 72 teachers. So, that's the equivalent of firing four teachers. And that's under an extremely aggressive scenario. Besides, now that you've rid your school of the chaff, who, exactly, do you want to fire next year? And if you want to argue that we could be more aggressive and fire some of the novice teachers, that would mean there'd be fewer low-performing experienced teachers (since teachers tend to be roughly equally effective pre- and post-tenure). So, for now, let's stick with the assumption that, under an aggressive plan, we'd fire four teachers this year in the average school.

Now, other districts have far more experienced teachers. And it might make more of a dent there. But a good number of our poorest-performing schools and districts are quickly churning through teachers too fast for firing low performers to make much of an impact. Certainly, we should make every effort to rid our schools of the worst teachers (by increasing the performance of, and/or dismissing the lowest performers) -- I don't think anybody seriously disputes that notion. Or, at the very least, I don't think anybody serious disputes that notion. But will firing the lowest performing 6% of teachers in high-poverty NYC schools make a difference? It's possible. But let's be reasonable -- it's not going to make much of a difference.

So, yes, let's work harder to rid our schools of the worst teachers. But let's not pretend it will be easy to do. And, perhaps more importantly, let's not hold our breath while we wait to see if that bullet is actually silver. In most places, other problems loom far larger.

Evaluating the Evidence on Non-School Interventions

noreply@blogger.com (Corey Bunje Bower) — Mon, 06 Feb 2012 09:50:00 +0000

I've been meaning to finish writing this piece for six weeks, and now I finally have. Enjoy.

One of the most e-mailed articles in the NY Times shortly before Christmas was this piece by Helen Ladd and Edward Fiske on social class and educational achievement, in which the authors call for more non-school interventions ("education policy makers should try to provide poor students with the social support and experiences that middle-class students enjoy as a matter of course"). Overall, I thought it was a pretty good piece, but two things in particular struck me.

1.) That they build an argument for focusing on what happens outside of schools and then their first recommendation is to expand pre-schools.

2.) The recommendations after the pre-school discussion are fairly vague.

While the first is interesting, I'm more intrigued by the second -- and I wonder to what extent it's because they want to recommend that we change 30 things they can't possibly list in the limited space and to what extent it's because they're not sure exactly what to address.

Which begs the question: what do we know about which non-school programs will make a difference? One particularly promising young scholar has argued that we don't yet know enough (you'll get the joke if you click on the link) to draw many conclusions on the topic.

The authors are certainly right that "Large bodies of research have shown how poor health and nutrition inhibit child development and learning" and they could've included numerous other factors at the family and neighborhood level. Since we know that these social factors and environmental conditions are causally related to academic performance, trying to ameliorate their impact on low-income children makes all the sense in the world. But, at the same time, I have yet to find (after extensive searching) a whole lot of evidence that we've been able to successfully do this in ways that rigorous research has found subsequently improved academic performance. And Russ Whitehurst argues the point even more strongly, writing in a recent report that "There is no compelling evidence that investments in parenting classes, health services, nutritional programs, and community improvement in general have appreciable effects on student achievement in schools in the U.S."

Let's take a look at the few programs they do mention in the piece. When I search Google Scholar for research on the programs they name, this is about all I can find on the East Durham Children's Initiative, Syracuse's Say Yes to Education program, Omaha's Building Bright Futures, and Boston's Citizen Schools. Only the last one links the program to any educational outcomes, and it appears to be an internal report. If there's evidence in peer-reviewed academic journals that these programs have improved students' academic performance, I've yet to see it (note: this is not to say that any of these four aren't working, just that we don't yet have really good evidence that they are).

At this point, some of you may be saying "you forgot about the Harlem Chidren's Zone!". That's certainly the most-cited example of social policy impacting academics. But there's a funny thing about that. As far as I can tell, only one study has linked HCZ to academic outcomes. And one thing that recently caught me eye is a chapter by Roland Fryer and others in the new Duncan/Murnane book on inequality and schools (highly recommended, btw). In particular, I find it interesting how they've changed their tune on HCZ the past couple years.

In 2009, Fryer put out an NBER working paper with PhD student Will Dobbie arguing that the HCZ had effectively closed the black-white achievement gap. The paper got all sorts of play in the press, with David Brooks claiming it proved once and for all that the "no excuses" schools were all that we needed and some of the Broader, Bolder folks replying that, no, it proved once and for all that community resources made the difference.

Shortly thereafter, I asked Geoffrey Canada which it was when he visited Vanderbilt -- he said that we needed both and that it was a "terrible, phony debate" to try and separate them. Nor could Dobbie and Fryer definitively separate them; in the introduction, they write (emphasis theirs) "We cannot, however, disentangle whether communities coupled with high-quality schools drive our results, or whether the high-quality schools alone are enough to do the trick." (p. 4)

But now they've updated the paper and, according to Fryer's Harvard info page, it's been accepted at the American Economic Journal: Applied Economics. This is from the abstract: "We conclude with evidence that suggests high-quality schools are enough to significantly increase academic achievement among the poor. Community programs appear neither necessary nor sufficient."

This would go nicely with the new book chapter (here's a slightly different version) in which they write, on the first page:

The evaluation of the Harlem Children's Zone allows us to conclude that a high-quality school coupled with community-based interventions does not produce better results than a high-quality school alone, offering further evidence that school investments offer higher social returns than community-based interventions.

That seems like a rather sweeping statement to make based on one preliminary estimate of one program's effects but, nonetheless, their findings do put the burden of proof back on those supporting the Broader, Bolder position.

The closest thing I've seen to a collection research citations indicating that we do have evidence that community-based interventions can work is David Kirp's recent book, but even that involved a good deal of cherry-picking and mostly discussed small programs not explicitly linked with local schools.

So, where does this leave us? As I wrote above, we have plenty of evidence that a wide range of experiences associated with living in poverty negatively impact kids' academic performance. And we have plenty of reason to believe that altering these experiences could, potentially, improve kids' academic performance. But I, and others, would argue that we have precious little empirical evidence that social policy has (or will) alter kids' lives in ways that will subsequently improve their grades, test scores, graduation rate, attainment, etc. So I find it a bit odd that Ladd and Fiske conclude by writing

But let’s not pretend that family background does not matter and can be overlooked. Let’s agree that we know a lot about how to address the ways in which poverty undermines student learning. Whether we choose to face up to that reality is ultimately a moral question.

I'd make a different pitch if I were they. I'd write something more along the lines of this: Let's not pretend that family background and living conditions don't matter and can or should be overlooked. Let's agree that we know a lot about how poverty undermines student learning and how large this impact is. And let's agree that we urgently need more research on ways to address the links between poverty and education. The Promise Neighborhoods and other initiatives deserve our full attention and support in the short-run and can potentially provide that will help us better address the problem in the long run.

Of course, twice as many words with half the certainty is a really bad formula for an op-ed. And there's no quicker way to frustrate policymakers than to write "more research is needed."

But, at the same time, I'm not sure it's helping their cause to claim that we know how to solve the problem. If I'm in charge of a new Promise Neighborhood, my immediate reaction would be "We do? Great!" Quickly followed by asking "which factors should I aim to address and which programs do we know are best to address these?" I don't know the answer to that, and I've yet to hear from anyone who does.

So, in the end, I'd say there's about as much empirical evidence that social policy will close the achievement gap as there is that charter schools, merit pay, and vouchers will close the gap. That is, very little. So if we insist on arguing for an either/or approach, this leaves us at a standstill. Both sides can yell that the other side's evidence is weak. Which doesn't seem particularly productive to me.

As a researcher, this seems like an excellent argument to conduct a lot more research on the links between social policy and academic performance (as well as on in-school interventions). Were I a policymaker, I'd want to avoid putting all my eggs in one basket. We know the status quo doesn't work, but we can't really say for sure what else would be better. That seems like a golden opportunity for policymakers and researchers to work together and experiment (literally) with a wide variety of reforms -- the former would get to hedge their bets and look prudent and open-minded while the latter would get to conduct groundbreaking research on a crucial issue.

In sum: Do we have conclusive evidence that a particular set of non-school interventions will close the achievement gap? No, we don't. So let's not claim we do. But, let's also vow to keep searching for it.

Top 50 Endowments Per Pupil

noreply@blogger.com (Corey Bunje Bower) — Tue, 31 Jan 2012 22:25:00 +0000

I'm sure I missed a couple schools (I'm human, I didn't include public schools, and there may be some ultra-small schools I missed), but these are all the schools I could find who are able to spend at least $10K per student per year (assuming a 5% endowment spending rate) while glancing through the rankings and endowment data.

Rank	School	Endowment	Enroll	End/Stu	5%/stu	USNWR
1	Princeton	$17,109,508	7,802	$2,192,964	$109,648	Res 1
2	Yale	$19,374,000	11,701	$1,655,756	$82,788	Res 3
3	Harvard	$31,728,080	19,627	$1,616,553	$80,828	Res 1
4	Pomona	$1,700,454	1,560	$1,090,035	$54,502	LAC 4
5	Swarthmore	$1,508,483	1,524	$989,818	$49,491	LAC 3
6	MIT	$9,712,628	10,566	$919,234	$45,962	Res 5
7	Amherst	$1,641,511	1,795	$914,491	$45,725	LAC 2
8	Grinnell	$1,500,219	1,655	$906,477	$45,324	LAC 19
9	Williams	$1,784,305	2,083	$856,603	$42,830	LAC 1
10	Stanford	$16,502,606	19,535	$844,771	$42,239	Res 5
11	CIT	$1,772,369	2,175	$814,882	$40,744	Res 5
12	Rice	$4,451,452	5,879	$757,178	$37,859	Res 17
13	Cooper Union	$607,135	910	$667,181	$33,359	RCN 2
14	Wellesley	$1,499,872	2,411	$622,095	$31,105	LAC 6
15	Berea	$978,735	1,613	$606,779	$30,339	LAC 27
16	Washington &Lee	$1,218,132	2,173	$560,576	$28,029	LAC 12
17	Dartmouth	$3,413,406	6,141	$555,839	$27,792	Res 11
18	Notre Dame	$6,259,598	11,992	$521,981	$26,099	Res 19
19	Richmond	$1,877,193	3,618	$518,848	$25,942	LAC 27
20	Chicago	$6,575,126	12,781	$514,445	$25,722	Res 5
21	Bowdoin	$904,215	1,762	$513,175	$25,659	LAC 6
22	Smith	$1,429,527	3,113	$459,212	$22,961	LAC 19
23	Claremont McKenna	$543,236	1,278	$425,067	$21,253	LAC 9
24	Emory	$5,400,367	13,381	$403,585	$20,179	Res 20
25	Trinity (TX)	$962,829	2,417	$398,357	$19,918	RUW 1
26	Duke	$5,747,377	14,983	$383,593	$19,180	Res 10
27	Bryn Mawr	$671,103	1,755	$382,395	$19,120	LAC 25
28	Wash U	$5,280,143	13,820	$382,065	$19,103	Res 14
29	Northwestern	$7,182,745	19,389	$370,455	$18,523	Res 12
30	Berry	$752,544	2,087	$360,586	$18,029	LAC 121
31	Middlebury	$907,668	2,532	$358,479	$17,924	LAC 5
32	Hamilton	$657,529	1,861	$353,320	$17,666	LAC 17
33	Columbia	$7,789,578	22,283	$349,575	$17,479	Res 4
34	Haverford	$402,730	1,177	$342,167	$17,108	LAC 10
35	Colby	$611,441	1,825	$335,036	$16,752	LAC 21
36	Vassar	$814,130	2,446	$332,841	$16,642	LAC 14
37	Penn	$6,582,029	19,842	$331,722	$16,586	Res 5
38	Carleton	$653,465	2,020	$323,498	$16,175	LAC 6
39	Macalester	$654,465	2,033	$321,921	$16,096	LAC 25
40	Harvey Mudd	$243,125	773	$314,521	$15,726	LAC 18
41	Davidson	$509,583	1,742	$292,528	$14,626	LAC 11
42	Denison	$654,584	2,275	$287,729	$14,386	LAC 49
43	Brown	$2,496,926	8,695	$287,168	$14,358	Res 15
44	Lafayette	$658,146	2,414	$272,637	$13,632	LAC 40
45	Vanderbilt	$3,414,514	12,714	$268,563	$13,428	Res 17
46	Mount Holyoke	$602,481	2,345	$256,922	$12,846	LAC 29
47	Cornell	$5,059,406	20,939	$241,626	$12,081	Res 15
48	Colgate	$693,436	2,903	$238,869	$11,943	LAC 21
49	Oberlin	$699,895	2,974	$235,338	$11,767	LAC 24
50	Holy Cross	$606,074	2,899	$209,063	$10,453	LAC 29

There seem to be four tiers here:

1.) Three schools can spend over $80,000 per student in a given year
2.) Eight schools can spend between $40K and $55K per student
3.) 13 schools can spend between $20K and $40K per student
4.) 26 schools can spend between $10K and $20K per student

I was curious whether national research universities or liberal arts colleges had more resources per student but, outside the top three, the're pretty evenly mixed.

Endowments of Top 25 Research Universities

noreply@blogger.com (Corey Bunje Bower) — Tue, 31 Jan 2012 22:12:00 +0000

I already posted the figures for top 25 liberal arts colleges, so I might as well post these as well.

USNWR	School	Endowment	Enroll	End/Stu	5%/stu	Rank
1	Harvard	$31,728,080	19,627	$1,616,553	$80,828	3
1	Princeton	$17,109,508	7,802	$2,192,964	$109,648	1
3	Yale	$19,374,000	11,701	$1,655,756	$82,788	2
4	Columbia	$7,789,578	22,283	$349,575	$17,479	15
5	Chicago	$6,575,126	12781	$514,445	$25,722	10
5	CIT	$1,772,369	2,175	$814,882	$40,744	6
5	MIT	$9,712,628	10,566	$919,234	$45,962	4
5	Penn	$6,582,029	19,842	$331,722	$16,586	16
5	Stanford	$16,502,606	19,535	$844,771	$42,239	5
10	Duke	$5,747,377	14,983	$383,593	$19,180	12
11	Dartmouth	$3,413,406	6,141	$555,839	$27,792	8
12	Northwestern	$7,182,745	19,389	$370,455	$18,523	14
13	Johns Hopkins	$2,598,467	21,092	$123,197	$6,160	21
14	Wash U	$5,280,143	13,820	$382,065	$19,103	13
15	Brown	$2,496,926	8,695	$287,168	$14,358	17
15	Cornell	$5,059,406	20,939	$241,626	$12,081	19
17	Rice	$4,451,452	5,879	$757,178	$37,859	7
17	Vanderbilt	$3,414,514	12,714	$268,563	$13,428	18
19	Notre Dame	$6,259,598	11,992	$521,981	$26,099	9
20	Emory	$5,400,367	13,381	$403,585	$20,179	11
22	Georgetown	$1,160,291	16,871	$68,774	$3,439	24
23	Carnegie Mellon	$1,017,338	11,618	$87,566	$4,378	23
23	USC	$3,517,173	36,986	$95,095	$4,755	22
25	Wake Forest	$1,058,250	7,162	$147,759	$7,388	20

The six columns above are the school's US News & World Report ranking, the school name, the endowment size (in thousands), the school's enrollment (as reported by USNWR), the endowment per student, a ratio of 5% of the endowment per student (which is roughly how much a school has to spend per student solely from tapping into the endowment), and the rank (within this sample) of the endowment per student.

I dropped public schools because I wasn't sure which endowments listed were for one particular campus versus which ones were for a system of schools. As expected, there's a strong correlation between endowment size and ranking (-.68 between endowment and ranking and -.70 between per pupil endowment and ranking).

Notable overachievers include Penn and Columbia, which both have longstanding reputations of excellence given their placement in both the Ivy League and large, historically prominent American cities, and Johns Hopkins, which has some very prominent programs in medicine and hard sciences.

Notable underachievers include Rice, which is located in Houston, which has only recently become one of the largest cities in the nation, and Notre Dame, which has strong ties with the Catholic church.

Endowments of Top 25 Liberal Arts Colleges

noreply@blogger.com (Corey Bunje Bower) — Tue, 31 Jan 2012 09:19:00 +0000

The release of the latest endowment data piqued my curiosity. One can look at them a million ways, but I'm particularly interested in liberal arts colleges and decided to compare the endowment figures to US News & World Report rankings.

As you can see in the table below, the higher-ranked schools generally have both larger endowments and more endowment money per student (correlation of -.53 between endowment and USNWR rank and -.58 between endowment/student and UWNWR rank), which is, of course, not a surprise (both because they money allows schools to afford items that increase prestige and because the endowment itself is used by UWNWR as one component of their rankings).

Before I get into the discussion, let me display the data. In the six columns are the USWNR ranking, the school name, the endowment size (in thousands), the school's enrollment (as reported by USNWR), the endowment per student, a ratio of 5% of the endowment per student (which is roughly how much a school has to spend per student solely from tapping into the endowment), and the rank (within this sample) of the endowment per student.

USNWR	School	Endowment	Enroll	End/Stu	5%/stu	Rank
1	Williams	$1,784,305	2,083	$856,603	$42,830	5
2	Amherst	$1,641,511	1,795	$914,491	$45,725	3
3	Swarthmore	$1,508,483	1,524	$989,818	$49,491	2
4	Pomona	$1,700,454	1,560	$1,090,035	$54,502	1
5	Middlebury	$907,668	2,532	$358,479	$17,924	12
6	Bowdoin	$904,215	1,762	$513,175	$25,659	8
6	Carleton	$653,465	2,020	$323,498	$16,175	17
6	Wellesley	$1,499,872	2,411	$622,095	$31,105	6
9	Claremont McKenna	$543,236	1,278	$425,067	$21,253	10
10	Haverford	$402,730	1,177	$342,167	$17,108	14
11	Davidson	$509,583	1,742	$292,528	$14,626	20
12	Washington & Lee	$1,218,132	2,173	$560,576	$28,029	7
12	Wesleyan	$589,811	3,215	$183,456	$9,173	23
14	Vassar	$814,130	2,446	$332,841	$16,642	16
17	Hamilton	$657,529	1,861	$353,320	$17,666	13
18	Harvey Mudd	$243,125	773	$314,521	$15,726	19
19	Grinnell	$1,500,219	1,655	$906,477	$45,324	4
19	Smith	$1,429,527	3,113	$459,212	$22,961	9
21	Bates	$231,451	1,725	$134,174	$6,709	24
21	Colby	$611,441	1,825	$335,036	$16,752	15
21	Colgate	$693,436	2,903	$238,869	$11,943	21
24	Oberlin	$699,895	2,974	$235,338	$11,767	22
25	Bryn Mawr	$671,103	1,755	$382,395	$19,120	11
25	Macalester	$654,465	2,033	$321,921	$16,096	18

I should first note that I dropped the US Naval and Military Academies (which tied for 14th) from the sample -- no endowment data was available, and I'm not convinced they belong in this group anyway.

In no particular order, here are the things that stand out to me:

1.) As I said above, the wealthiest tend to be ranked the highest. No surprise. Five institutions in particular far out-distance the rest in terms of resources per student, four of which are ranked 1-4.

2.) Given that all of these schools are in the same general arena in terms of prestige, there's quite a bit of variability in terms of financial resources -- two schools have less than $10K per student to spend while five have over $40K.

3.) Notable underachievers (high resources, low ranking) include Grinnell, Smith, Washington & Lee, and (arguably) Wellesley. Two of these are single-sex institutions, which may limit applications (raising acceptance rate) and/or increase alumni giving relative to other factors. Washington & Lee is one of only two schools located in the South and has a reputation as being more conservative than the other schools. I'm not sure how to explain Grinnell other than possibly the Iowa location. As I mentioned above, exactly five schools have more than $40K to spend per student (the same five that have endowments exceeding $1.5 billion), and four of those rank 1-4. Grinnell ranks 19th.

4.) Notable overachievers include Carleton, Davidson, Wesleyan, and Bates. I'm not sure I have a great explanation for these other than the possibility that their reputation is larger than their endowment (an argument I don't find super compelling). More on this below.

5.) An interesting factoid: Macalester's endowment is exactly $1 million more than Carleton's. Macalester is 45 miles north of Carleton, enrolls 13 more students, and is 19 spots behind in the rankings.

Were I to do more than muse about summary statistics, I'd wonder why some schools earn much higher ratings relative to their financial resources than others. To the extent that a given school's goal is to enroll the best students and earn the highest ranking, some are drastically more successful/efficient than others.

Of course, I'd expect that a large part of the explanation is that some of these schools place far more emphasis on the factors USNWR measures and/or the USNWR rankings themselves. I'd also imagine that, among other things, allocation of these resources, admissions practices, location, reputation, campus offerings, and -- in various ways -- performance of campus leaders, faculty, and staff influence the success/efficiency of a school (as measured by USWNR relative to endowment resources).

Lastly, I'd wonder if the endowment, say, 20 years ago might be more strongly correlated with the ranking than the current endowment. Or at least if some the low endowment/high rank schools had higher endowments relative to their peers 20 years ago and some of the high endowment/low rank schools had lower endowments relative to their peers 20 years ago. This certainly would've influenced the faculty they hired 20 years ago, the perception of the college by students' parents, and probably the perception of the school by people in academia, at foundations, working as college counselors, etc. In other words, schools that have dramatically expanded recently may be thought of as more resource-rich than they currently are and schools that have seen recent dramatic growth in endowment may be thought of as more resource-poor than they current are.

I'm no expert on higher ed or philanthropy, so I can really only guess about these numbers. All I know is that I find them interesting and that there are probably some good explanations for the patterns we see. If anybody knows more about these topics than I, I'd love to hear your explanations.

Update: I neglected to note that along with allocation of resources, I'm sure the allocation of the endowment matters as well -- if large chunk of the endowment is restricted to, say, maintaining the hockey rink or seminary that it will affect the school differently than if a large chunk is restricted to, say, financial aid or even unrestricted uses.

I also neglected to mention that not only do some schools emphasize the ratings and try to game the system, but some might even just flat out lie.

The Wrong Question About Teacher Pay

noreply@blogger.com (Corey Bunje Bower) — Wed, 04 Jan 2012 01:24:00 +0000

The NY Times posted an online debate today that asked whether teachers are overpaid. The following is their prompt, verbatim.

In the private sector, people with SAT and GRE scores comparable to those of education majors earn less than teachers do. Does that mean teachers are overpaid? Or that public schools should pay more to attract top applicants who tend to go into higher-paying professions?

I'd argue that this is the wrong question. If current teachers earn more than similarly achieving (as measured by the SAT and GRE) private sector employees, that means they must earn the same as private sector employees who are higher achieving.

Given the recent focus on teacher quality, shouldn't the question then be why these higher-achieving graduates choose to do something other than teach despite earning no more than current teachers earn?

Moving on from the "Search for Universals"

noreply@blogger.com (Corey Bunje Bower) — Thu, 01 Dec 2011 07:47:00 +0000

As I've delved deeper into the field of education research, I've grown increasingly frustrated by two large problems I see in that research. While listening to Malcom Gladwell's TED talk on consumer choice, something he said helped crystallize the second issue (I'll explore the first in another post).

In the talk, Gladwell discusses the shift of food science (starting with Prego pasta sauce) from trying to create the "one best" product for all to creating multiple products that fit specific groups. He compares that trend to cancer research, arguing that "the great revolution in science of the last 10-15 years" is "the movement from the search for universals to the understanding of variability".

If he's right, then it seems ed policy researchers missed a memo somewhere (to be fair, it's not just ed policy -- I see the same problem in lots of policy literature).

The vast majority of the policy research I've read, seen presented, or heard discussed focuses exclusively, or at least mainly, on the average effect of a particular variable, intervention, or policy. Over the years, we've developed increasingly sophisticated methods to more accurately estimate these average effects. But I rarely hear people discuss the differential effects of the variable, intervention, or policy of interest.

In other words, we keep trying to figure out if different policies "work," but we define "work," as making a statistically significant difference for the average student, teacher, principal, school, district, or state. If we instead asked for whom a given policy works, we'd likely find that it works very well for some while harming others.

In the talk, Gladwell mentions a food scientist hired to create the "one best Pepsi" who conducts taste tests of the product with varying amounts of sugar. To his surprise, no one, clear winner emerges. Some people prefer only a little sugar, some like a medium amount, and some like a lot. He then has an epiphany and realizes that there's no such thing as the "one best Pepsi" (which eventually leads to the creation of a wide variety of Prego sauces that target different audiences).

I'd argue that education policy is almost exactly the same. For example, imagine a new math curriculum. How do we decide if it works? The gold standard of research would dictate that we randomly assign, say, 100 classrooms to use the new curriculum and 100 to use the old one. We'd then compare the average scores at the beginning and end of the year of the treatment and control groups. If kids in the treatment group score significantly higher, on average, than the control group then that curriculum earns a gold star.

We spend increasing amounts of time trying to figure out which math curricula will yield the largest achievement gains across the students who use them. But it seems exceedingly unlikely that "one best math curriculum" truly exists. Some states, districts, schools, teachers, and/or students will do best with curriculum A, some with curriculum B, and some with curriculum C.

Wouldn't our time be better spent figuring out for which students curriculum A would be best and for which students curriculum B would be best (and why)? And we could say the same about all of the largest issues in education policy -- teacher training, teacher pay, charter schools, and so on. In later posts, I'll explore how our research and policy would differ if we aimed to understand and account for variability rather than simply finding the one best policy.

I've long thought it odd that we spend most of our lives being told not to stereotype and make generalizations while the most educated people in the country strive to make the largest generalization possible in their research. Most (though not all) researchers focus incessantly on "generalizability" (one of those words we use in the field but the spellchecker won't recognize): if we can generalize your findings to 10 million students across the country, you're likely to publish the paper in a better journal than if we can only generalize your results to students in one school.

As it turns out, I was right to think that odd. Greater generalizability matters in many ways. But, at some point, we start missing the point -- and for exactly the same reasons teachers and parents across the country are telling our kids not to generalize. Everybody is different.

Every student is different. Every teacher is different. Every principal is different. Every school is different. Every district is different. Every state is different. The best policy, on average, will help one and hurt another. Until we understand this variability, we're doomed to fail.

Reaction to the CMO Study

noreply@blogger.com (Corey Bunje Bower) — Mon, 07 Nov 2011 09:49:00 +0000

I don't disagree with the larger point behind Andrew Rotherham's argument regarding the recent study on Charter Management Organizations (CMOs) that we need to "live with some lemons" in order to see other schools succeed (and subsequently learn from those success stories). I certainly wouldn't want my kid to attend one of the lemon schools (the educational equivalent of NIMBYism), but the larger argument that we sometimes have to take a step backward to take two steps forward is reasonable.

I, do, however disagree with two things he writes in the column:

1.) He writes that the study disproves charter critics' argument that "the good ones can’t be replicated to serve enough kids to really make a difference" by making it "clear that it is indeed possible to build a lot of schools that are game-changers for a lot of students". He mentions that KIPP has 100 or so schools now and continues to perform well. That's good. That's a lot of schools. What KIPP has done is really impressive.

But when charter critics say that high-flying charters can't be scaled up, they don't mean that we can't create 100 good charters. They mean that we can't create 10,000 good schools run exactly the way KIPP and other schools are currently run. In particular, KIPP and other top-performing schools tend to rely heavily on 20-something TFA members and TFA alums who are high-achieving, idealistic, often single, and willing to work long hours for relatively little pay for a few years. There's nothing wrong with doing that. If I opened a charter school tomorrow, I'd probably do the same thing. But it is something that can't be replicated in every high-poverty school across the country. The fact that KIPP has opened 100 schools absolutely does not change the fact that there aren't enough teachers who fit this mold to run all schools like this.

That's not to say that we can't copy many other innovations and features of the KIPPs of the world, just that building 100 schools run one way doesn't mean we can run every other school in the country (or even just all the high-poverty schools) the exact same way. The "enough to make a difference" description of the argument is arguably a straw man since even one school "makes a difference" -- Rotherham should instead focus on implications for systemic reform rather than beating up straw men.

2.) I also find it really odd that the study itself and some of the articles on the study seem to emphasize that the differences in achievement gains between CMO-run charters and traditional public schools were not statistically significant, while Rotherham writes that "The study found that, in general, students at charter-network schools outperform similar students at traditional public schools, although sometimes not by very much" (and he's not the only one who said something like this).

I understand that the way these results are interpreted are different in academia and policy. Of the eight point estimates they calculate, one is borderline significantly positive, three others seem like they could maybe be positive (point estimate larger than the standard error), three are trivial, through technically positive, and one is trivially negative. So, I guess that's kinda (sorta) positive. On the other hand, of the 22 CMOs they examine, 11 have positive coefficients in both math and reading, 2 are neutral in both, and 9 are negative in both.

In academia, that means that there were no significant differences -- I can't imagine trying to construe that as a positive world. In the policy world, it could mean that "there are likely to be very small, positive differences," but to say they "outperform" other schools "in general" seems rather strong to me -- especially since he added "sometimes not by very much". There were no areas in which the average CMO school outperformed the average non-charter by a lot, so that seems like an odd thing to say.

Of course, I hesitate to even point out these two things (they just struck me as odd while I was reading his column) because they threaten to distract from what I think should be the main discussion surrounding this study and other research on charters. The argument about whether or not charters are better or not is, quite frankly, silly. Some are. Some aren't. The end. I want to know why some are better and what we can learn from those that are better. And, similarly, why some are worse and what we can learn from them. As long as we regard "charter" as a word that magically makes a school either evil or, well, magical, we won't really be learning all we can to help educate our students.

Do Teacher Quality Initiatives Impact the Wrong Teachers?

noreply@blogger.com (Corey Bunje Bower) — Tue, 25 Oct 2011 06:34:00 +0000

Let me illustrate my point by first telling three anecdotes.

1.) Believe it or not, back in elementary school I was just about the model student. I was quiet and respectful in class, did all my homework on time, scored high on tests, wrote good reports, and won numerous awards. When teachers fretted about the performance of their students, I don't think my name came up too often.

Which is why my first day of fourth grade sticks out. I'd been assigned to a teacher new to our school, and I didn't know what to expect. I hoped she'd be nice and was a little worried when she took the opportunity to sternly lay down the law and do her best to discourage any disobedience. As my Mom tells it, I came home in tears that day, sobbing "She's sooo strict!"

2.) A couple weeks ago, I was the teacher fretting about my class. As I prepared interim grade reports, it was even more evident to me that a number of students weren't putting forth the effort I was hoping they would (and had become accustomed to seeing from students in my class). As I puzzled over this, part of me wanted to read my class the riot act. I settled for hoping that the sub-par grade reports and a few words of motivation would kick them into gear.

A day later, I got a tentative knock on my office door. A student was worried about the report. I looked at the student's grades for the semester and quickly ascertained that there was no need for concern here. We spoke for a while and I assured the student that earning the highest quiz grades in the class indicated a strong likelihood that the end-of-term grade would be pretty high if present efforts were maintained.

3.) A good friend of mine is a model teacher. You know that teacher that arrives at school before dawn, gives up lunches, nights, weekends, and breaks to tutor students, chaperon dances, re-make that lesson plan for the 20th time, and do whatever else is necessary (and, usually, unnecessary)? That's my friend. Were I the principal, I'd promptly resign and insist my friend take the job.

A couple years back, the school district where my friend teaches implemented a new teacher evaluation program involving lots of new checklists and observations and other bells and whistles. Ever since, my friend has been an absolute wreck. Every conversation inevitably, and repeatedly, turns to the strong likelihood that my friend will no longer be employed in the near future. I assure my friend that the new system is designed to ensure the district keeps the model teachers and that the worrying is unnecessary, but to no avail. I don't know if the constant anxiety has negatively impacted my friend's teaching or not, but it's certainly impacting quality of life.

so, what do these three have in common?

In the first, the teacher (rightfully) wanted to scare the worst students straight and push the mediocre ones to do better. But it was the best student (I'd like to think) who was mortified, not the worst ones. Many years later, I found out my Mom had relayed my reaction to the teacher, who had sighed, shaken her head, and said something like "it's always the wrong ones who get scared."

In the second, I (rightfully, I sure hope) wanted to scare the worst students straight and push the mediocre ones to do better. But the only reaction I got was from possibly the best student in the class -- the one who doesn't need to spend any time fretting about what the end of term report card will say.

In the third, the district (rightfully, I think) wanted to scare the worst teachers straight (and/or just fire them) and push the mediocre ones to do better. I can't say how the other teachers responded, but the model teacher I know is the one who's been scared, despite being straight as an arrow to begin with.

what does this mean?

Is it possible that our attempts to scare teachers straight are only scaring the ones who were already doing things the right way? After all, the ones who care the most about their performance are the ones who are most likely to take the new initiatives to heart.

Whether one threatens to fire teachers, rolls out a new evaluation system, publishes value-added scores, implements a new incentive pay system, or whatever else, I wonder who will be most responsive? It seems likely that it's those who were already the most responsible.

so what?

If we can expect those who care the most to react the strongest to accountability and evaluation initiatives, then we need to change the way we frame and present these initiatives. We can't just assume that a few threats will scare the stiffs straight when the stiffs aren't even paying attention. And we don't certainly don't want to scare off the best and the brightest.

I'd argue, we need to take a more nuanced and targeted approach when pursuing these types of efforts. Let's first make sure that those who are doing the right thing are recognized and thanked for their efforts. Those who aren't recognized and thanked, and don't seem interested in being recognized and thanked, may be the ones we need to threaten, encourage, or hold accountable.

I recently had a conversation with a friend who manages organizational change in another field. I relayed the story of my friend the model teacher and the subsequent anxiety. My non-teacher friend quickly dismissed the anecdote, noting that all organizational change elicits fear and anxiety among employees. It seems to me that teachers might be more anxious than others, but I'm inclined to agree with that point: all organizational change probably does elicit fear and anxiety among employees. But is it the right employees who are scared and anxious?

When Outsiders are Needed

noreply@blogger.com (Corey Bunje Bower) — Tue, 18 Oct 2011 08:43:00 +0000

I've written about the push to include more outsiders in education (here and here, for example), and often focused more on the negatives than the positives of doing so. So, today, let me take a brief moment to highlight one of the negatives of not allowing outside perspectives into education.

As regular readers know, I used to teach at a middle school in the Bronx that was shut down a few years back. While shuttering the school (and subsequently opening three new, smaller, schools inside the building) was certainly no panacea, it's hard for me to believe it could possibly have made the situation any worse.

Not everything at my school was a disaster (the most notable exception to me was that a good portion of the teachers were at least very devoted if not also very skilled), but the list of negatives far exceeds the list of positives. "Dysfunctional" would be a fair (maybe even kind) assessment of the day-to-day operations of the school.

That's the background for this snippet of conversation between two veteran teachers from a few weeks ago:

Teacher 1: "The more i visit schools, the more I see we were doing this right. [Our school] should have never closed."
Teacher 2: "You are so right [Teacher 1]. I still get angry about it...like it was our fault!"

I'm not a psychologist, but it seems pretty clear to me that the two teachers are (still) unable to dispassionately evaluate our school. This would align with the split in reactions to the announcement the school would close that I witnessed: the newer teachers in the building (myself included) mostly seemed to say things like "good riddance . . . I'll find a better position somewhere else," while the vets struggled with the decision and where to go next (and were suddenly filled with nostalgia for a school they'd ostensibly detested the week prior). They obviously had a much deeper connection to the school than did us newbies, but they also seemed to interpret evaluations of the school as implicit evaluations of their own personal performance.

I'm sure there are a million good reasons for them to feel this way, but the policy-relevant point is that those feelings prevented them from seeing all sides of the situation. If any attack on the school becomes a personal attack against them, it seems unlikely that they'd ever be able to embrace radical change in a school that clearly (to me, anyway) needed just that.

So, in this case, I'd argue that outsiders were needed to do that. I left before the new schools were up and running, so I have no idea if the outsiders' solution really helped, but I think the recognition that the school wasn't working was a valuable contribution regardless.

In short: while outsiders frequently intrude where they're not needed, this was an instance where they were.

I'm back

noreply@blogger.com (Corey Bunje Bower) — Tue, 18 Oct 2011 08:25:00 +0000

Not that I ever officially left, but between dissertation, teaching, revising manuscripts, job hunting, etc. blogging kept falling to the bottom of my to-do list. I don't expect that to-do list to grow much shorter in the next six months, but I've also noticed how much more engaged I am in ed. policy issues outside of my research-focus when I'm blogging regularly. So I'm going to blog regularly. Not every day. And usually shorter pieces than I've written in the past. But I'm back.

Why No Outrage over Principal Quality?

noreply@blogger.com (Corey Bunje Bower) — Tue, 16 Aug 2011 20:15:00 +0000

As teachers around the country start to head back to work, I'm starting to hear the same thing I hear every year from teachers I know: "my new principal is horrible". But it seems like I never read anything in the news about principal quality: instead, everything seems to focus on teacher quality. Bad teachers are absolutely a problem, but is it possible that bad principals are actually the larger problem?

Here in metro Nashville, a quarter of principals are new this year. I'm not an expert on school leadership, but it's hard for me to imagine that most principals, like teachers, do more than tread water their first year. While the new principals learn the ropes, the school climate hangs in the balance: too many moves in the wrong direction may result in teachers, staff, parents, and students becoming disillusioned by, or just flat out leaving, the school.

The research says that teacher quality explains a greater percentage of variance in student achievement on a yearly basis than does principal quality, which makes sense given the direct relationship one has with their teacher versus the mostly indirect relationship one has with their principal. But in the longer-term, might a bad principal have a larger negative effect on a student than a bad teacher? After all, a bad teacher can ruin a classroom, but a bad principal can ruin a school.

The Middle Ground in the Discipline Debate

noreply@blogger.com (Corey Bunje Bower) — Tue, 19 Jul 2011 08:19:00 +0000

A new report being released today apparently finds that 60% of students in Texas were suspended or expelled at least once between 7th and 12th grades. As the NY Times reports, that's a huge number (though I don't quite understand why only 31% were suspended out of school -- apparently half of the kids received in-school suspensions instead (supposedly that's less severe, but does that seem like a worse punishment to anybody else?)). Everyone interviewed in the article seems outraged at the number, and rightly so -- there's simply no way that 60% of students really cause serious problems in schools.

That said, simply reducing such punishments is no answer either. My school, for example, was under a good deal of pressure to reduce suspensions (word on the street was that our first principal resigned under pressure largely because the rate was deemed too high). The result was that a few students got away with ludicrous behaviors, significantly reducing what the vast majority of students learned while simultaneously frustrating teachers in a building that already had a serious attrition problem.

So, I empathize with all those who are outraged by the sky-high numbers in this report. We should certainly try to spend less time punishing, and more time teaching, our students. But I also empathize with all those students and teachers whose learning and teaching are unnecessarily inhibited on a daily basis by a few students acting out. I completely agree that we need to reduce the number of punishments meted out, but that can't be the only goal -- we simply cannot sacrifice student learning in the pursuit of less distressing numbers.

District Choice -- For Cities

noreply@blogger.com (Corey Bunje Bower) — Mon, 11 Jul 2011 16:47:00 +0000

Here's an interesting story to follow from the suburbs of Pittsburgh. It seems that the majority of the residents of the tiny borough of Rosslyn Park have signed a petition asking that their community be part of the Chartiers Valley School District rather than the Carlynton School District.

Why? A number of issues seem to be at play, but it seems that the main driver is that Chartiers Valley is, in many ways, a better district and has lower tax rates. Given that 34 of 70 school-aged children residing in Rosslyn Park attend private or parochial schools, it's possible that the latter is actually more important than the former.

This isn't without precedent, but I can't say I've ever heard (or considered the possibility) of towns switching school districts. Granted, this only applies to towns that are part of a multi-town school district -- which eliminates this as a possibility in an awful lot of places -- but it seems plausible that this could become a growing trend. Based on the information in the article, I think if I lived in Rosslyn Park I'd want to switch districts too. But I wonder if this were to catch on whether it would just be another way for parents to send their kids to more segregated schools.

Asking the Right Question About Charter School Skimming

noreply@blogger.com (Corey Bunje Bower) — Mon, 11 Jul 2011 06:32:00 +0000

This NYT article seems incomplete, but I like it for one simple reason: people talk all the time about charters cherry picking or cream skimming kids, but never seem to ask the right question . . . this article does. There's a ton of evidence that most charters do not take the highest scoring students (see, for example, this chapter from this new book) and those data are used as evidence that charters don't skim.

Case closed, right? Do charters skim? No, they don't.

But that's the wrong question. The issue shouldn't be whether charters take the highest scoring students, it should be whether they enroll the best-behaved and/or most motivated students (and then nudge out those who are unruly and/or unmotivated).

In other words, we should be asking if charters enroll kids who are better students instead of asking if they're enrolling students who previously earned higher scores. Why? Once you get a class or school full of motivated, attentive, and polite students it's a heck of a lot easier to teach them. And a heck of a lot easier to see large gains in test scores.

I have yet to see any rigorous analysis of the extent to which charters do, in fact, enroll or retain better students. Instead, I read a lot of anecdotes like the one from the NYT article I linked to above. Were I to hazard a guess, it would be that there's at least one charter out there that enrolls/retains substantially better students than the surrounding schools.

Even if I'm right, whether or not that's a good thing or a bad thing is a whole separate discussion. But let's start that discussion by asking the right questions.

If Teachers are Impossible to Fire . . .

noreply@blogger.com (Corey Bunje Bower) — Wed, 29 Jun 2011 05:48:00 +0000

. . . then why are so many so petrified of losing their jobs? In conversation after conversation with teachers from a wide variety of schools around the country this continues to stand out to me. And I'm not quite sure what to make of it.

What are the contexts of these conversations? They mostly arise in the following two situations:

1.) A new administrative regime comes in (at either the school level or higher) or the current regime hands down a new directive. Teachers scramble to re-do their bulletin boards, do more test prep, fill out more paperwork, or whatever else they think they need to do to cover their behinds. This certainly doesn't apply in all situations, since I've also seen teachers ignore new directives, essentially refuse to implement new curricula, etc.

2.) More worryingly, I've seen it time and time again when teachers are aware of wrongdoing by other people in the building or district -- particularly when it involves a direct supervisor. I often seemed to be the only one in my building willing to report the unethical behavior I witnessed -- possibly because I had the luxury of knowing I wasn't trying to teach in the same district again the next year. I was recently speaking with a colleague who has witnessed outrageously unethical behavior by the principal at his school. I encouraged him to report this, and the response I received was "I need my job too much . . . [my principal] is waaaaay too dangerous. I'm scared to death of him".

I have far too many anecdotes to fit into one blog post, but I can't tell you how many conversations I've had with panicked teachers anxious about their job status. Given that almost all of these teachers were tenured at the time of the conversation, the anxiety in their voices doesn't jibe with the current rhetoric about teacher labor markets. It seems to me that there are three possible explanations for this (not including the possibility that my perceptions are skewed):

1.) Teachers are, indeed, almost impossible to fire -- but they don't realize that. I suppose it's possible that teachers perceptions are off, but it seems unlikely that their that off-base.

2.) Teaching as a profession tends to attract a lot of people-pleasers who are afraid to stand up for themselves. This may not be entirely without merit -- I'd feel comfortable saying that most teachers I know or have met are more interested in helping others than causing trouble, but this seems like only part of the explanation at best.

3.) There's a dangerous lack of trust in too many schools and school systems. I don't want to be alarmist or paint with too wide of a brush, but this strikes me as the most plausible explanation of the three. If teachers don't trust their supervisors to be fair and ethical, it stands to reason that they'll constantly worry about their jobs regardless of whatever protections they have.

Is worrying about one's job always a bad thing? Of course not; sometimes a little panic can boost productivity. But when it results in the proliferation of unethical or downright abusive behavior, I start to worry about all the worriers. And when policies aim to increase the worry-level of teachers, I worry about the potentially negative consequences for our schools and students.

To paraphrase the old milk commercials: Trust. It does a school good.

"For Me and Not for Thee"

noreply@blogger.com (Corey Bunje Bower) — Mon, 27 Jun 2011 19:54:00 +0000

Yesterday's Economic Scene Column by David Leonhardt captures my biggest objection to the "not everybody should go to college" argument. He concludes the column by writing that:

I don’t doubt that the skeptics are well meaning. But, in the end, their case against college is an elitist one — for me and not for thee. And that’s rarely good advice.

First, some context. Fewer than one-third of 25-29 year-olds have earned a four-year degree, and even fewer adults from older generations have done similarly. And evidence abounds that students from high-SES families are far more likely to obtain a college diploma (to the extent that high-achieving high school students from poor families are less likely to earn a diploma than are lower-achieving students from wealthier families). The last stat that I read was that 67% of students at the top 200 or so colleges come from families ranking in the top quartile economically while only 10% come from households ranking in the bottom half. So if it's true that too many people are attending college, that probably means too many kids of high-SES parents are attending college.

But it seems that those who most forcefully demand that fewer people attend college and/or that more people should pursue other options possess college degrees themselves -- and plan on sending their own children to college. Depending on how one looks at it, that makes many of these arguments either elitist or hypocritical.

What if a Principal Allows Teachers to be "Bad"?

noreply@blogger.com (Corey Bunje Bower) — Fri, 24 Jun 2011 17:02:00 +0000

The rhetoric about "bad teachers" may never go away -- in part some teachers will always perform poorly, act irresponsibly, and so on (just as there are poor performers and irresponsible people in all professions and fields). That said, what bothers me most about the rhetoric is that it continually oversimplifies the problem. Too many commentators seem to assume that bad people magically pop up in schools to torture principals and belittle children. But reality is more complex.

One situation that has arisen in numerous anecdotes I've heard from teachers is that a Principal will allow selected teachers (often their friends) to behave irresponsibly or worse. Examples include showing up late, parking illegally, dressing inappropriately, eating meals with the Principal instead of teaching, leaving other people in charge of their class while they run errands, and, in at least one instance, abusing children.

Are "bad teachers" a problem in these schools? Absolutely -- and they should be dealt with -- but not following the script we normally read (teacher is bad, principal wants to fire him/her, union steps in). In these cases, the story I hear is that a "bad" principal allows a few teachers to do as they please while the rest of the teachers stew in outrage and cower in fear.

Did the Principal in these situations make these teachers bad? It's not quite that simple. But these Principals have certainly negatively impacted the performance of a few teachers while subsequently damaging the climate and performance of the school as a whole.

Situations like these are why I worry more about the extent of the damage done by irresponsible Principals than I do about the damage done by irresponsible teachers.

Unintended Consequences of College "Promise" Policies

noreply@blogger.com (Corey Bunje Bower) — Fri, 24 Jun 2011 03:53:00 +0000

The Quick and the Ed had a good piece today on the emerging research on the college "promise" policies enacted by various districts around the country (I believe Kalamazoo was the first and Pittsburgh is the largest, but I might be mistaken). They're all a little different, but they essentially guarantee to pay for some portion of college for all students in the district who meet a certain benchmark.

The piece notes research indicating that more students in some of these districts are enrolling, and remaining, in college. That's likely a good thing. But what I don't see mentioned is one of the potential unintended consequences of the policy -- grade inflation. If kids have to meet a certain GPA standard to receive the funds, it's likely that teachers will hesitate before giving a student a 'C' or below knowing that they could be costing the kid the opportunity to attend college -- and that districts that want to be a success story would likely pressure teachers not to give kids low grades in order to bump up their college attendance rates. I don't know in how many places this is happening, but I know it's not zero -- so keep your eyes open for this when more reports on the effectiveness of these promise policies roll in.

Family Values Don't Matter Anymore?

noreply@blogger.com (Corey Bunje Bower) — Tue, 21 Jun 2011 00:01:00 +0000

One thing that continues to intrigue me is the flip in the attitudes of some conservatives toward the power of schools. When the Coleman Report was released, at least one prominent conservative concluded that it meant that "schools make no difference; families make the difference" and that education is "all family". But for some reason now the attitude is that the poor results of African-American males are "nothing that a good school wouldn’t fix" as Peter Mayer writes on the Fordham Institute blog.

Mayer writes in response to an Ed Week article about the poor performance of black boys, arguing that if we spent less time on teaching them about character, drugs, pregnancy, jobs, and dealing with emotional problems and more time on teaching them content that we wouldn't have this problem. He concludes by arguing that:

African American children, like most children, would do much better later in life if school taught them how to read and write – and, hopefully, a little history and science, art and math along the way – instead of being served up what has become a steady and distracting and unhealthy diet of paternalism and fries.

I'm struck by four things:

1.) I won't belabor the point here, but we have plenty of evidence that non-school factors impact academic performance. Simply ignoring emotional problems, drug use, pregnancy, and other problems that start outside of school won't change that fact.

2.) I'm not even sure that high-poverty schools spend more time on character education, drug prevention, and sex ed than do other schools. If anything, particularly given the incentive structure of NCLB, I'd guess that higher-poverty schools spend more time on reading and math than do others.

3.) Regarding the paternalism quote, I find it interesting that a number of the high-performing charter schools Fordham continually praises were examined in a book written by David Whitman and published by Fordham back in 2008. What are these schools doing right? Whitman argues that, above all else, all of these high-performing schools are highly paternalistic.

4.) Perhaps most striking to me is the dismissal of pregnancy as something worth addressing. Given that around three-quarters of African-American children are now born out of wedlock and that evidence indicates that both single-parent households and teenage mothers negatively impact the performance (on average, of course) of students, this seems like an excellent time to call for a renewed focus on family values. It seems to me that one could build a pretty strong case that a combination of marriage, delayed child birth and personal responsibility could go a long way toward improving the results of African-American males, so I find it odd that most conservative commentators instead seem to call for better schooling instead.