Tuesday, December 14, 2010

The Times' Tenure Math Problem

The NY Times today profiles some fairly reasonable, and minor, changes to the way tenure is granted to teachers in NYC (principals a.) have to describe why they're giving a teacher tenure and b.) are able to select from four ratings that can award, delay, or deny teachers tenure).

On the surface, those make sense, and the article seems to do a pretty good job describing them.  But I have a problem with the math on which the article is premised.  The second sentence of the article states that "virtually every new teacher earns" tenure, and then later cites a statistic that "6,400 teachers who were eligible for tenure, 234 teachers were denied it, or roughly 3.7 percent".

The problem here is that a "new teacher" and a teacher "eligible for tenure" are two very different things.  Writing that 96% of those eligible are awarded tenure is very different from writing that "virtually every new teacher earns" tenure.  Why?  To start with, NYC has a high attrition rate -- particularly among new teachers.  Nationwide, around 50% of teachers in urban schools leave within their first three years.  Let's say that that statistic is precisely correct for the teachers that entered NYC schools in 2007; the correct calculation would then be that 46% of new teachers who began teaching in 2007 were awarded tenure in 2010 -- which looks like very different.  In other words, the statement that "virtually every new teacher earns" tenure is demonstrably false.

Now, the 46% statistic wouldn't be all that helpful in judging the rigor of the tenure process because new teachers leave for all sorts of reasons.  Some leave because their principals threaten to fire them if they don't; some leave because they simply prefer a different district or profession; some leave because they're simply unhappy; and some leave for other reasons or a combination of reasons (e.g. family relocation, pregnancy, health, etc.).

So, leaving aside the silly -- but important -- syntax error by the Times, what would actually be the actual best way to compute how many teachers earn tenure?  There are a few different ways depending on exactly what the question is that one wants to answer, but I think you'd have to start by defining a population of teachers who want to earn tenure at some point during their teaching career (in NYC that would mean they want to teach in the city for more than 3 years).  Exactly who counts in that group is up for debate: for example, would we include a teacher who wants to make it a career during year 1 but finds that he/she just doesn't like teaching during year 2?  There's no clear answer.  I would certainly include, however, any teachers who leave explicitly because they believe they will receive a negative rating if they don't -- something that happens all over the place but never seems to get mentioned in the press and only rarely in academic work (I've heard economists reference "the dance of the lemons," to describe teachers who transfer to other schools to avoid being fired).  The estimate would shed some light on how many unsuccessful teachers -- teachers who might not receive tenure if they went through the process -- are leaving prior to starting their 4th year.

Whatever parameters one set, the answer to how many teachers tried to get tenure and didn't would be somewhere between the 46% estimate above and the 96% statistic cited by the NY Times.  The more practical questions to answer, however, would really be in estimating how many people would be valuable as tenured teachers but don't end up teaching more than 3 years for whatever reason and how many are not particularly valuable as tenured teachers but earn tenure anyway for any number of reasons (my guess is that the former group is larger than the latter, though there are also probably quite a few in the former who wouldn't stay in teaching almost no matter what), and then figuring out how to get more in the former group to stay and more in the latter group to leave.

At any rate, I don't think there's much disagreement that tenure is too easy to earn in NYC (personally I'd argue for a more meaningful process that's awarded a little later in one's career, and matched by some added distinction that's meaningful to teachers), so I don't want readers to think that this critique of the Times' math problem is really a veiled attack on any who would dare insinuate that all teachers aren't perfect.  But the Times should know better than to make such silly mistakes.  And I hope that readers know better than to accept such superficial calculations as gospel.

Do "virtually all new teacher earn" tenure?  No.  Do almost all teachers who go up for tenure review earn tenure?  Yes.  But those are two very different statistics -- and neither really answers the more important questions of how many excellent teachers don't receive tenure and how many poor teachers do.

Thursday, December 9, 2010

Remember: Not Everyone Prioritizes Achievement

The commentariat seems to agree on two things regarding American schools: that they're not good enough and that students need to be learning more.  Read a random blog or watch a random talk show and you'll likely see or hear something about more rigorous curricula, longer hours, longer school years, or some sort of argument that our students need to be pushed harder.

And you're twice as likely to see or hear these things after the results of an international assessment are released (e.g. the latest PISA results this week) and the U.S. again ranks behind a lot of countries.

But we in the education world can sometimes forget that there are other things that matter too.   It would be wise for those driving this commentary to remember that not everybody shares their priorities and keep in mind that many people in our country have no interest in spending more time in school.  I'd divide this constituency into three different groups:

1.) People who absolutely detest school and can't wait for the bell to ring.  Witness this cartoon comparing school to jail . . . with the only difference being that the disgusting food offered in jails is always free.  For kids who believe school is tantamount to jail or torture, it's unclear how exposing them to more misery will improve their lives.  In other words, it may be more important to find a type of schooling that they only hate a little bit than to simply pile on more.

2.) Families with students enrolled in elite, high-pressure schools (usually private or suburban).  Witness the number of parents excited by the new movie "Race to Nowhere" that chronicles the downside of such schools.  Indeed, the most obvious sign of the difference between classes when it comes to schooling may be the broad push for more rigorous schooling for low-income students while high-income parents increasingly worry about the amount of stress their children endure while running five clubs, taking 4 AP classes, and waking up early for SAT prep class on Saturday before going out to volunteer at the local animal shelter.

3.) Those whose priorities lie elsewhere -- e.g. sports.  Witness the saga of Urban Meyer, the University of Florida's highly successful football coach, who stepped down yesterday -- at the age of 46 -- because he wanted to spend more time with his family.  Did the press release say that he wanted to help them with their homework or volunteer in their classrooms?  No.  It said that "after spending more than two decades motivating and celebrating the young men I've been so proud to coach, I relish the opportunity to cheer for my three terrific kids as they compete in their own respective sports".  For other families it's music, art, drama, religion, or any of a million other activities that evoke more passion.

 There's some overlap, of course, between these groups.  But an awful lot of students/parents/families fall into at least one of these groups.  This certainly doesn't mean that we shouldn't examine longer days/years or such reforms to boost our national achievement levels.  But it does mean that we shouldn't assume that these types of changes would automatically be popular or successful.  And it illustrates some of the trade-offs of these types of reforms.

In a number of the highest scoring countries, many kids spend countless hours in "cram schools" after school and on weekends -- essentially doing test prep.  American kids, on the other hand, are more likely to go to football practice, piano lessons, or their part-time job after school.  There's no objective answer as to which practice is better, but time is finite and kids can't do everything.  And many, many parents would prefer that their kids choose from the latter list.

So, by all means, we should be examining ways in which we can boost our national achievement.  But boosting these test scores isn't the only thing that the citizens of our country care about.  As such, any reform designed to boost these scores will not automatically be accepted by all.  In short, a more productive discussion would weigh the costs and benefits of emphasizing different areas rather than simply assuming that everybody wants higher test scores at whatever cost.

Are Superintendant Searches a Thing of the Past?

After NYC hired an outsider (Joel Klein) to run their schools, Pittsburgh decided to hire an outsider as well (Mark Roosevelt).  Both leaders were controversial, but can claim some successes.  And both were generally liked more by policy wonks than by teachers or parents.

When NYC decided to replace Klein this year, the process was done behind closed doors and -- as far as anyone can tell -- exactly one candidate was interviewed (update: apparently, at least one other person may have been offered the job).  The end result is a prospective new chancellor (Cathleen Black) that may be even more strongly disliked by parents and teachers.  When Pittsburgh needed to replace their superintendent this year, meetings were held behind closed doors and exactly one candidate was interviewed.  The announcement was made today that the board will hire Deputy Superintendant Linda Lane as the next leader of Pittsburgh Public Schools on Monday.  It remains to be seen how parents and teachers will react to the news, but it seems unlikely that the move was made to assuage their concerns.

There, are, however, three major differences:

1.) Lane has spent almost 40 years working as a teacher and administrator in public schools, while it was unclear whether Black had even ever set foot in a public school when her hiring was announced.

2.) While Bloomberg was evasive and seemed to insinuate that others were interviewed for the position, the Pittsburgh Board of Education is being upfront about it.  Said the board president: "What would be the point of spending the money to do the search when what we wanted was right here?"

3.) Many in the press and public turned on Bloomberg after his announcement.  Judging by the glowing article in the Pittsburgh Post-Gazette, however, Lane is apparently already beloved by the press.  It will be interesting to see if the public feels the same way.  Despite the glowing review, it does seem somewhat unlikely that a board of ed that didn't even feel the need to pretend that they were conducting a thorough search would emerge completely unscathed.

Thursday, November 18, 2010

Value-Added vs. Batting Averages

Stephen Sawchuck reports that a new report on value-added scores compares them to measures in other fields, including batting averages in baseball.  I think the batting average analogy is a good one.  Here's why:

When we look at a player's batting average (the number of hits divided by the number of at bats), we are provided with some information that reflects how good they are at baseball.  The value of this information increases when the player has more at bats and plays for more years.  And the larger the gap is between two players' batting averages, the more certain we can be that one player is better than the other.

But, at the same time, a player's batting average in one given year gives us only a small amount of information about that player's ability.  Averages vary year to year: players suffer injuries, become distracted, grow older, switch teams, play with different teammates, and so on.  For example:

Who's a better shortstop: Alexei Ramirez or Derek Jeter?  The former hit .282 this year while the latter hit .270.

Who's a better first baseman: Aubrey Huff or Ryan Howard?  The former hit .290; the latter .277.

Batting average doesn't do a good job of measuring power, speed, defense, leadership, or any of a million other desirable characteristics.  And for that reason, teams have moved beyond batting average when evaluating and signing players.  It seems like every year a new stat bubbles up that measures a player's ability in one facet of the game.

Adam Dunn is a career .250 hitter who plays awful defense, but will make millions of dollars next year because he'll probably hit 40 home runs yet again.

Mike Cameron, a .250 lifetime hitter, signed a 2-year, $15 million dollar contract last year as a 36 year-old.  His power (about 25 HR per season recently) didn't hurt, but the main motivation was his defense.

When we look at a more sophisticated (though still flawed) computation of how many wins a player added over an average replacement player (Wins Above Replacement, or WAR), we see that Jose Bautista, who hit .260 this year, ranks sixth.  The fifh-ranked player, Adrian Beltre, has a lifetime average of .275.

At the same time, nobody would argue that batting average is meaningless -- especially over longer periods of time.  There are plenty of hall of famers with career averages over .300, but none that I know of with averages of .220.

In other words, the batting average analogy is an excellent one for value-added scores because they have four very important things in common:

1.) Anybody who tells you that it is totally meaningless is totally wrong.
2.) When you see large differences over long periods of time between two people, you can be pretty sure that the one with the larger number is better.
3.) At the same time, that number in and of itself gives us very little information about how good somebody is, particularly over a shorter period of time.
4.) There are other skills not measured by the number that need to be taken into account when evaluating the person.

Unlike baseball, we don't have better statistics in evaluation.  We don't have something like on base percentage, yet alone change in student motivation over replacement teacher.

What does this mean for schools and value-added scores?  If we designed the perfect evaluation system, given current knowledge and tools, value-added scores would have to be included.  The only really compelling reason to leave them out is the potential for misuse by people who don't understand #'s 2-4 above.  Partly for that reason, it's important that value-added scores be only one component of how a teacher is evaluated.  And principals should be looking for large differences over long periods of time, not tiny year-to-year differences when making hiring/firing/tenure decisions.

Opponents of value-added scores would be better off arguing that they can be useful -- but only in some circumstances -- while proponents should be looking for supplemental measures to boost the meaningfulness of evaluations.  And both sides should keep in mind that the majority of teachers don't teach subjects tested by state tests anyway, so in addition to measuring a teacher's impact on students other than on standardized tests (e.g. motivation, attitude, self-control, creativity, interpersonal skills, etc.) we should also keep in mind that we need better ways to evaluate a teacher's impact on students in other subjects beyond 3rd-8th grade English/math as well.

In short, value-added scores can aid our evaluations of teachers . . . but only a little bit, and only if used properly.

Tuesday, November 16, 2010

"The Shadow Scholar"

In case you missed it, The Chronicle of Higher Education recently published an article written by "Ed Dante" (a psuedonym) about his/her experience writing papers for desperate/lazy college students.  I'd put "The Shadow Scholar" in the "must read" category for all those interested in higher education, and in the "well worth your time" category for everybody else.

The author makes decent money (on pace for 66K this year) writing papers for students and says that a student has never complained about getting caught.  The process and stories are interesting, but the author makes two main points:

1.) A lot of students are absolutely awful at writing and more should be done about this.
2.) A lot of assignments are ridiculous and drive students to look for other solutions (including purchasing papers)

I wholeheartedly agree with the first and half agree with the second.

The comments are almost as interesting as (though more tedious than) the article itself.  Between the author, the commenters, and my own head, there seem to be some competing hypotheses as to why a not insignificant number of students cheat in one way or another, particularly on written assignments.  In no particular order, they are:

1.) Professors assign meaningless/useless/stupid assignments
Why?
-large class sizes prevent individualized assignments
-laziness
-following conventions/norms
-more concerned with research than teaching

2.) Students don't know how to write
Why?
-they're non-native speakers
-they didn't learn how to write during K-12 schooling
-their college profs don't take the time to teach them how to write

3.) Students are unethical
Why?
-they don't understand what's ethical and what's not
-ethics and morals aren't emphasized enough in school
-a few bad apples make everyone look bad
-lots of pressure to get good grades; little pressure to behave ethically
-they don't value the assignments they're asked to complete or the courses they must take

4.) Unethical people make it easy to cheat
Why?
-some people care more about money than ethics
-people think they're doing others a service since they think the assignments are stupid


I don't think any of these fully explain the situation, but I do think they all shed a little bit of light on it.  In short, there are plenty of people and institutions at whom we can point fingers . . . but wouldn't it be more productive if we instead focused on fixing the problem?

Thursday, November 11, 2010

More on NYC's New Chancellor

The NY Times has a piece on Bloomberg's secretive search for the next schools chancellor.  A Times blog follows up with a mildly amusing post asking if anybody was actually interviewed during this search (and if they were, or know someone who was, to write in the comments section). 

The secrecy surrounding the search is interesting, but seems like it could be a red herring -- the ability of Ms. Black to lead the schools is ultimately more important than exactly how she was picked . . . with the caveat that if the process makes her appear illegitimate, it could undermine her authority.  Witness, for example, this online petition that a teacher friend was asking people to sign asking the state not to grant a waiver allowing her to serve as chancellor despite no educational background.

It's somewhat surprising to me that she's been greeted with this much skepticism given how many district leaders there are out there with no educational backgrounds, including her predecessor, Joel Klein.  It will be interesting to see if she's able to win over her detractors or if the next three years will be one long battle.

Wednesday, November 10, 2010

NYC's New Chancellor

The shocking news of the evening is that Joel Klein has stepped down as chancellor of NYC's schools (and accepted a job with News Corp.) and been replaced by former magazine exec Cathleen Black.  I knew nothing about Cathleen Black before a few hours ago, and have no idea whether she'll be a good leader for NYC's schools or not, so I'll leave the guessing to others.

But what I found most interesting about this is that she (a.) has no background in education and (b.) is described by the NY Times as someone who "earned a reputation in publishing as a tough-minded chief executive who never left her employees guessing what she wanted".

It's almost cliche at this point to hire a superintendent who is both an outsider and "tough".  I have no idea whether she'll still be described as "tough-minded" in years to come, so we'll have to wait and see on that one, but the absence of an educational background is worth more examination.

On the one hand, it's easy to see why somebody from the business world is attractive to a mayor or board of ed.  If your goal is to transform the school system as fast as possible, you want somebody who will come in and get the job done without much of a fuss.  And given that the vast majority of educational leaders have extensive backgrounds in education and that our system still has many problems, it would be easy to dismiss such a background as unnecessary.

And I don't think that such a view would be entirely without merit.  Ed. Schools probably catch more flak than they deserve, but it's pretty clear that many of them don't offer very high-quality or practical programs.  And there's a certain amount of value in bringing in somebody that hasn't been a part of the system -- there's an Icelandic expression that's relevant here: "the eye of the outsider is sharp".

At the same time, there's no reason that an outsider couldn't be somebody with experience outside of NYC instead of just outside of schools.  Having the negotiating skills or steely resolve necessary to stare down the union may or may not produce positive results, but it's also possible that experience as a teacher might more readily convince the union's constituents that new chief wants to work with them instead of against them.

All in all, I can't think of another field in which expertise or experience is treated with less regard than it is in education -- probably largely because virtually every resident of this country spent a good portion of their lives attending school.

Of course, if this trend continues, at some point in the future we'll read about some rebel mayor deciding to pursue the outside-the-box strategy of hiring somebody with a lot of educational experience to serve as superintendent since none of the business whizzes have yet found the magic cure-all for schools.

update: The NY Times has a forum on whether a district leader needs to have a background in education to succeed.  The arguments largely mirror those outlined above.  Clara Hemphill asks “would the mayor have named a publishing executive as head of the Department of Health?”  Richard Kahlenberg points out that a background as a teacher might make a leader more readily accepted by the rank-and-file.  And Marcus Winters and Neil McCluskey of the conservative Manhattan and Cato Institutes argue that an outsider perspective can more readily advance new ideas, especially since the old ideas that education folk have been trying have yet to cure all that ails our schools.

Friday, November 5, 2010

Today's Random Thoughts

*One college professor writes that their teacher prep program is losing many of their brightest prospective history teachers because the students are skeptical that they'll have much over the content of their courses and turned off by the battering teachers take in the press.  I tend to be skeptical of those who claim that there's a seismic shift -- with every single person who's creative abandoning teaching -- but I don't doubt that there is some serious dissatisfaction among some and, correspondingly, at least a small amount of attrition . . . I would expect it more from early-career teachers than from prospective teachers, so I find this piece both somewhat surprising and somewhat worrying.  Since we seem to be focusing a lot of time and effort lately into proposals that will attract and retain better teachers, we should probably investigate these types of claims before deciding which policies to pursue (hat tip: EP).

*A note to all the worried parents and college students out there: what the student chooses to major in doesn't really matter all that much when it comes to earnings later in life . . . in fact, it might be better to study what one is passionate about than simply picking what one deems most practical.

*Speaking of worried parents and teenagers, this NY Times/Chronicle of Higher Ed joint piece has a graphic that demonstrates fairly well the rise in the number of colleges that high schoolers have applied to over the past 20 years, though they miss out on the most explosive growth -- people applying to 10, 20, 30, or more colleges.  When I was in HS, it was typical for ambitious kids from well-to-do families to apply to about 8 or so; my understanding is that it's now typical for kids at private and upper-class public schools to apply to 20 or more schools (which the common app and online applications have made much easier to do).  In one sense, this is a savvy thing for kids to do -- with ever-declining acceptance rates, why not make sure you have every base covered?  But, at the same time, unless they're getting fee-waivers, that's a lot of money we're talking about for college applications.

*A group of researchers at Cornell write about ways to redesign lunch lines that will result in kids eating healthier food.  I always appreciate it when people find little tweaks that cost almost nothing, but get big results by nudging people in the right direction . . . but I have to wonder if simply doing these types of things wouldn't be abdicating our responsibility to teach kids about nutrition and responsibility.  If they don't take chocolate milk b/c it's behind the white milk and don't take ice cream b/c it's in a covered container, that's good for today -- but what will they do when they go to the grocery store as adults?

Thursday, November 4, 2010

I'm Still Alive

Apologies for the sudden dead air, but I've been swamped with dissertation and job-related activities. I'll try to update at least once each week over the next few months, but I can't make any promises.

Quick note while I'm here: everybody seems to be wondering if yesterday's election results will bring us compromise or gridlock for the next two years. If it's the former, education might be a logical starting point . . . I can't think of another major issue where the parties are closer to agreeing.

Friday, October 8, 2010

Small Schools and Extracurriculars

I've written this week (here and here) about the decision of Pittsburgh's superintendent to step down.  One of the reforms he's implemented over the past 5+ years is move the district away from large, comprehensive high schools and toward smaller, specialized high schools.  It's a trend anybody in NYC would be familiar with (where the average size of a middle school has shrunk by about 15% since 2004).  In the past I've referred to this as the "charterization" of schools.  As is clear in NYC and other places, charters tend to be much smaller and have more specialized names (think "Knowledge is Power Program" instead of "PS 100" or "Washington High"), and draw from beyond the local neighborhood.

In a lot of ways, I think smaller schools are a good thing (or at least have the potential to be a good thing).  I've heard one charter operator explain that he believes in schools with about 500 students because that's what all the top private schools are like -- and I think he makes a good point.  A small school makes it easier for everyone in the building to know each other and, subsequently, for schools to be more aware of the dynamics of the school environment and the needs of individual students.

But there's one major drawback to the smaller schools that doesn't seem to get much press: extracurriculars.  A lot of students define their experiences in high school (and, for that matter, college) by their experiences on athletic teams, in school plays, in the marching band, and so on.  When a friend mentions an old classmate and you can't quite put a face to the name, what do they say?  They don't say "he really liked history" . . . they say "she played the clarinet" or "he was on the swim team".  Everybody takes classes in high school, but far fewer people make pottery or play a leading role in the school play.  Accordingly, some students -- at least to some extent -- care more about their favorite extracurricular activity (or activities) than they do about classes.  Witness this excellent Sports Illustrated piece from last year about a school in Ohio that cut all sports and started hemorrhaging athletes.

So, what do we do to ensure that kids at these new smaller schools get to experience the activity (or activities) that might make them love high school?  There are a few options.

1.) We can continue to create schools that are focused around interests and activities -- arts academies and the like -- and hope that kids with similar interests enjoy similar activities.  Though I'm skeptical that athletic leagues will want to allow the new basketball academy to join its ranks.  And I'm not sure what happens when the quarterback at the football academy wants to join the glee club or the soprano at the choir academy wants to play volleyball.

2.) We can combine schools for the purposes of extracurriculars.  This is apparently Pittsburgh's plan, at least for football.  The biggest problem with that is that it robs the power of the school play or the Friday night football game to bring the community together (both the school community and the surrounding neighborhood).  If two separate schools field one football team, are they supposed to have two separate pep rallies?  And if two small towns merge their schools, in which town do they play football?  (Here's an excellent article on this phenomenon, again in the areas surrounding Pittsburgh.)

3.) Schools can boost rates of participation -- in various ways -- so that a smaller school can support more activities.  A number of private schools (I think, historically, this is something all-boys schools tended to do) mandate that every student play on a certain number of athletic teams each year.  Or schools could allow some clubs to meet during class time (I took a journalism class in jr. high in which we also wrote the school newspaper).  Or they could just emphasize clubs and activities more to students, parents, and teachers.

4.) We can go back to large, comprehensive, neighborhood high schools ("back" meaning in urban areas -- that's still still the norm in the suburbs and only somewhat possible in rural areas).  NYC might have the highest concentration of (relatively) small, specialized, non-neighborhood high schools in the country.  But it was announced this week that a new HS in Queens will be a large, comprehensive one.  “People want one large comprehensive school. You don’t want a bunch of boutique schools, a dance school, a school for lawyers,” said the DOE rep.

5.) Kids can just learn to do without.  Historically, the vast majority of schools in the United States have been quite small (which, of course, doesn't mean that the majority of students have attended small schools).  And many might deem other things (e.g. math skills) more important anyway.  Besides, there's always the possibility that local organizations (the theater, library, YMCA, little league, etc.) might sponsor activities if a school doesn't.  Of course, that would then remove one more enjoyable activity from students' school experience.

Extracurricular activities are extraordinarily important to many students, parents, schools, and communities and we do ourselves a disservice when we ignore this during our debates and discussions about education.

Thursday, October 7, 2010

More on Roosevelt's Departure

I wrote yesterday that I didn't understand why Mark Roosevelt was stepping down as Superintendent of the Pittsburgh Public Schools (and in mid-year, no less). And I still don't.

For those who are unfamiliar with Roosevelt and Pittsburgh, here's a little bit of background info: Roosevelt came to Pittsburgh a little over 5 years ago with no experience as a teacher, principal, or district administrator.  Which isn't to say was completely unqualified or without talent or relevant experience.  He had a law degree from Harvard, worked on education issues as a state legislator in Massachusetts (where he later ran for Governor), and served as a CEO and in various other leadership positions.  He then completed a fellowship with the Broad Foundation (not sure of the exact name of the program, but it was a 10 month program that dealt with leadership in urban schools) right before his appointment.

When he came to Pittsburgh, the district was most famous for its contentious (and that's being kind) board of ed meetings -- so contentious that a large number of foundations had pulled money from the district and warned that they'd better straighten up if they wanted to get it back.  At the same time, the city of Pittsburgh has fewer than half the residents that it did 60 years ago.  100 years ago it was the 8th largest city in the country and, I believe, had more corporate headquarters than any American city outside NYC (that might not be exactly correct, but the point is that it was a major center of industry).

Which means two things for the city and district: 1.) that they had far fewer students, but not far fewer schools, and 2.) that there's a lot of foundation money, per capita, in the city.  Which meant two things for Roosevelt: 1.) He needed to shrink or close schools, and 2.) he needed to make the foundations happy.

He closed 22 schools in all, and reconfigured many more.  Whether or not this was the correct move, one must admit that it took political courage and saved the district money -- and that it also upset a lot of parents living near schools that were closed or changed.  He's also been a big hit with the foundations.  The major Pittsburgh newspaper, the Post-Gazette, ran two stories on Roosevelt today (here and here), both of which are full of praise from all sorts of people.

Which is why it surprised me that he was leaving.  Yes, some parents and teachers didn't like him.  But the foundations loved him, the board of ed loved him, the mayor loved him, and the union barely put up a fight (they recently agreed on a 5-year contract).  I really have no insight into why he left -- I don't know the man personally and am unfamiliar with the day-to-day operations of his office -- so my best guess, given his previous frequent career-hopping, is that he was telling the truth when he said that he simply needed a new challenge.

The gist of what I've read and heard about him over the years is that he has many of the same strengths and weaknesses of a typical businessman.  He obviously got along well with the foundation people and the board of ed (a definite plus compared to previous regimes), but nobody would confuse him for a community organizer or former teacher.  The result was an administration that seemed to run smoothly and attracted lots of outside dollars, but didn't spend much time asking parents or teachers for advice.

Saying he ran the schools like a business would be cliche and overly simplistic, but he clearly wanted to streamline things.  He closed schools, narrowed curricula, and hired principals who would do things his way.

From the second article in the Post-Gazette: "In the classroom, Mr. Roosevelt sought to make teaching more consistent, including instituting a managed curriculum that required teachers to use certain materials and to present them at a certain pace. The recent teacher survey showed that some teachers think they have too little role in decision-making."


In the lower grades, the district closed many of the lowest-performing schools and moved the former students to K-8 "Accelerated Learning Academies" with extended hours and years.  In the upper grades, the vo-tech high school, and many other vo-tech programs, were closed in the name of academic rigor, scripted curricula were implemented, and he began opening small, specialized schools.  He leaned heavily on paid outside help to accomplish these reforms.  RAND evaluated all the schools, America's Choice provided the procedures and curricula for the ALA schools to follow, Kaplan wrote new HS curricula (and then was asked to leave so the district could re-write them themselves with some help from Pitt), and an outside firm was hired to run an alternative school for students with consistent discipline or attendance problems.  In the long run, he might be best remembered for starting the "Pittsburgh Promise," a fund that aims to pay full college tuition for qualifying district students who attend PA state colleges.  The fund got off to a rocky start, but is now much closer to its $250 million goal thanks, in part, to Roosevelt's relationship with local foundations.


In the end, he upset a number of parents and teachers.  Though the union rarely criticized him, there was a sharp divide between the attitudes of elementary and secondary teachers -- with the new contract being passed due to overwhelming support from the former and despite heavy opposition from the latter.  The lack of opposition from the union led to the election of some non-slate candidates in the last election who'd promised to be less docile.


And I'll be the last to argue that at least some of this anger was warranted.  I've written in the past that his new grading policy was poorly implemented and his newer system, though a little better, didn't work the way it was supposed to, that teacher morale was suffering under his leadership, and that some of his rhetoric was unhelpful.  But I'll be the first to admit that Pittsburgh could do a heck of a lot worse than 5+ years of more or less smooth sailing with the foundations, the board, and the union to go along with a steady stream of new ideas (some good and some bad).


But love him or hate him, one has to admit that it's somewhat irresponsible of him to step down abruptly in the middle of such turbulent change (and, personally, I think leaving mid-year reflects poorly on him -- but you're entitled to your own opinion).  This is the first year of a new $80 million dollar program (funded by Gates and the federal govt.) to re-vamp teacher pay and evaluation, numerous schools and students are slated to be reconfigured and moved next year, and he has a million other balls up in the air.  I think everybody can agree that he's done some good things and some bad things (though they certainly won't agree which things were bad and which were good), but I think everybody can also agree that he didn't finish the job.  It may or may not prove to be a good thing for the city/district that he didn't finish the job, and there may or may not be some very good reason(s) underlying the decision, but I'm somewhat disappointed in him personally for being willing to pick up and leave before the job is done.

Why?  In the midst of the effusive praise lavished on him by people of all different stripes was this quote from a leader of a local parents group: "Our group would ask that all initiatives be placed on hold until a new superintendent is found . . . We don't want all this money, expense, experimentation done if a new superintendent is going to take it in a different direction."  I don't know how many months it's going to take to find a new superintendent or how many of the current reforms will continue under his/her leadership, but the changeover seems pretty likely to result in a lot of uncertainty, hesitation, and delays.  Which can't be what Roosevelt thinks is best for the district.

Wednesday, October 6, 2010

Pittsburgh Superintendent Stepping Down

Forget the conflicting rumors over whether Ron Huberman is or is not stepping down as head of the Chicago school system, this is actually bigger news.  While Huberman is just getting his feet wet as chief of the Chicago schools, Mark Roosevelt dove in head-first a long time ago in Pittsburgh.

Apparently, he's resigning to become the head of Antioch College when it re-opens next fall -- though the college says there are a few more hurdles he has to leap before they'll be ready to make that decision.

But the timing of the move makes it both newsworthy and surprising.  And not just because he's jumping ship in the middle of the school year (December 31st, to be exact).  Few teachers would ever consider abandoning their students mid-year, so it seems somewhat tasteless for a Superintendent to do it (or maybe it just proves that teachers are more important, since the district will likely operate about the same in January as it did in December).

Roosevelt is in his sixth year leading Pittsburgh's schools (which is a fairly long time for an urban superintendent) and was, seemingly, right in the middle of his master plan for reform.  During his time he's implemented massive reform -- closing 22 schools and restructuring countless others, founding the "Pittsburgh Promise" (guaranteed college tuition for district grads that attend in-state public schools), and winning a $40 million grant from the Gates Foundation to overhaul teacher evaluation and pay last year, among many other things.  While he's upset more than a few teachers and parents, the school board is in his pocket, there are no national headlines about his brusque personality, and the mayor isn't going to be replaced anytime soon.

In other words, he pretty much had free reign to mold the district as he saw fit in the coming years (and his pay was recently upped to $240K, with which one can live like a King in eminently affordable Pittsburgh, as part of a new five year contract).  He was on top of the world.  And now he's leaving.  What?  What am I missing?

Considering that he had never worked in schools prior to his job in Pittsburgh (he was a lawyer/politician/businessman), and that he's faced so little opposition from the Board of Ed, it's hard to believe that he's burnt out (though, I suppose, not impossible).  He's originally from Massachusetts and attended Harvard, so there's not a blindingly obvious tie between him and Antioch (in Ohio).  He's never been a college administrator before, so he's not returning to his previous profession.  It's hard to believe that Antioch would be courting him if a giant scandal was about to be exposed.  So, why's he leaving?  You got me.  He's changed positions and professions quite a few times, so maybe he was just ready for a new challenge.

But considering that he was right in the middle of major reform in Pittsburgh, it seems like an awfully odd time to walk away.

It also exemplifies one of the problems with education.  Every time people start to adjust to a wave of reform, somebody new comes along and demands something else.  But people still wonder why teachers and school systems are so resistant to change.

It will be very interesting to see who gets hired as his replacement and which reforms he/she continues . . . stay tuned.

Tuesday, October 5, 2010

Today's Random Thoughts

-my latest pet peeve: people who complain about schools hiring people during a recession.  Why are we so bitter that people are finding gainful employment?

-The entire American educational system is not in crisis.  The entire system could be improved, yes, but only one portion (the same portion attended by children from low-income families) is truly in crisis.  That's essentially what Nicholas Lemann wrote in the New Yorker, but for some reason Joel Klein and others felt the need to write in and point out that a number of schools really are in crisis.  Am I the only one who fails to see an actual disagreement here?

-Not sure of the exact statistics, but Linda Perlstein writes that Macke Raymond told her that most charter school lottery "losers" actually end up at other charters, private schools, or move rather than attend the local public schools.  I'd like to see an actual statistic (e.g. is "most" 60% or 90%), but it sounds plausible.

-Albert Shanker was, apparently, full of good ideas.  I always wondered when I was teaching why a few exemplary lesson plans weren't made available for teachers on every topic so that they could get ideas and adapt them to their classrooms.  Wouldn't that make a lot more sense then asking every teacher to reinvent the wheel every time they teach?

-Another pet peeve: people who fundamentally misunderstand statistics and argue that a correlation doesn't exist because there's an exception to the rule.  Kevin Carey is far too smart to make that exact argument, but he falls into the same type of trap by pointing out that students in DC became poorer (it's unclear if this is true, but it's certainly plausible) over the past couple years at the same time that their test scores increased . . . which apparently means that income doesn't perfectly correlate with a child's test scores (?) or something like that.  Of course, we already knew that the relationship between poverty and academic performance is complex, that there's not a perfect correlation between income and test scores because a million other factors also impact test scores, and that children living in urban poverty are capable of improving their test scores.  So I'm unclear on exactly what this proves other than that people still don't understand that the exception doesn't disprove the rule and that a correlation doesn't mean that two things always move in lockstep.

-On a more positive note, I stumbled across this Freakonomics quorum on how to close the achievement gap the other day.  It's 2 1/2 years old, but it's still fairly interesting.

Can a School Transform a Neighborhood?

Mike Petrilli had an interesting post over at Flypaper yesterday, in which he writes that Arne Duncan's argument that "the only way you change communities is by having great public schools in those communities" is preposterous.

I say "interesting" for two main reasons:

1.) I'm not really that surprised, since Mike Petrilli is probably the most contrarian writer over there, but it's still somewhat surprising to see somebody from Fordham -- a group that has labeled Richard Rothstein and Charles Murray "defeatists" for doubting the efficacy of schools -- arguing that there's something schools can't do.

2.) I think it's a generally interesting question.  Can a school, or a group of schools, transform a neighborhood?  There's no question that the effects of poverty and the performance of students and schools are heavily intertwined, but it's yet to be seen exactly how much changing one can change the other.

The debate on the links between neighborhoods and schooling has generally been focused more on the reverse hypothesis: i.e. can fixing a neighborhood fix its schools (or, at the societal level, can eradicating poverty close the achievement gap)?

Petrilli quotes colleague Jamie O'Leary saying that "schools need to improve despite the neighborhoods; improving poor neighborhoods is beyond the capacity (or purpose) of public schools."  That's an interesting take.  I can see why, in the short run, the last thing somebody running a school would be worried about was whether their school was transforming a neighborhood.  But, in the long run, if good schools can't improve neighborhoods then what's the point of good schools?  If closing the achievement gap didn't reduce poverty, then what, exactly, would be the point of closing the achievement gap?

Sure, helping kids succeed academically is nice.  But if it doesn't translate into a better job and higher standard of living, doesn't it ring kind of hollow?  If there was no achievement gap in this country, but just as much poverty; just as much crime; just as much despair; and just as much suffering, would it really be a significant better place to live?

Petrilli challenges readers to name a single community that has been transformed as the result of the performance of the local school.  Off the top of my head, I cannot name such a community (which doesn't mean it hasn't happened), but I can offer some thoughts on how, hypothetically, such an outcome would occur:

First, Superman shows up and turns the local schools in a down and out neighborhood into the best schools in the area.  As a result, two things happen: 1.)kids learn more; and 2.) homes in that neighborhood become more desirable.  As kids learn more, they become more likely to do their homework and less likely to loiter around or otherwise terrorize the neighborhood after school.  As word spreads that the neighborhood has good schools and docile teenagers, interest in the neighborhood increases and gentrification begins.  Homes previously in disrepair are renovated, abandoned factories are turned into hip lofts, and vacant lots are filled with fancy new townhomes.  Home prices (and rents) increase.  Stable families with more money and higher achieving kids move in.  Unstable families with less money and lower achieving kids move out.  The parents in the neighborhood take pride in their local schools and band together with the new arrivals to form a community association.  This association bands together to clean up the neighborhood, raise money for community center, ballfields, and community garden, and demand more amenities from their local politicians.  With all the kids now attending school during the day and attending tutoring or playing on the newly constructed athletic fields after school, crime continues to drop and housing prices continue to rise.  The kids in the neighborhood go on to attend college and get high-paying jobs.  Some move back to their still-improving neighborhood, raising the average income and education levels of the neighborhood even further, and join the community association.  Their kids attend the local schools and are excellent students who stay out of trouble, do their homework, and volunteer in the community.  By this point, the neighborhood is a happy, healthy place to raise children.

Which isn't to say that all is right with the world: some of the former residents who were forced out by price increases caused by the gentrification are no better off.  Other neighborhoods still have problems.  But not this one.  This one's been fixed as a result of the prowess of all the local schools.  And all thanks to Superman, who decided to leave Lex Luther alone and make them the talk of the town.

Ok, so that's obviously the dream scenario.  But the general gist is plausible.  If nothing else, better schools could certainly help spur gentrification.  Whether or not that really improves a neighborhood (since so many of the residents would be forced out) is up for debate.  But, in the long run, better educated children returning to a neighborhood to raise their children would certainly have a positive impact on both a neighborhood and its local schools.  On the other hand, one could argue that in this scenario that it's really the improving neighborhood that sustains the quality of the schools over time.  And, at the same time, that it would've been more efficient to simply improve the neighborhood and watch the local schools soar (instead of waiting for Superman to turn them around).

I've been involved in more than one discussion of whether fixing schools or fixing neighborhoods is the more effective way to reduce poverty.  Before I started teaching, and before I started grad school, I had that debate in my head.  I decided to enter education, and then decided to study education policy, because I sided with the former position: that fixing schools was the most realistic and efficient way to improve the lives of low-income children -- and subsequently improve our nation.  After spending some time studying the effects of poverty on academic performance, I now find myself sitting on the fence.  I'm not convinced we really know the answer to the question.  I have little doubt that, when taken to the extremes, both are true.  If we were able to actually transform a neighborhood (which likely would require extraordinary amounts of time, effort, and money), the local schools would certainly be better -- and I think if we were able to fix our school system (again, requiring a lot of time, effort, and money) then it would go a long way toward improving our worst neighborhoods.  On the one hand, fixing a school (though certainly not easy), has to be easier than fixing a neighborhood, but fixing a neighborhood (if possible) has to have larger effects.  I suspect that some combination of reforms have to be undertaken at both levels in order to fix both, but I digress . . .

The fixing neighborhoods versus fixing schools debate can be another post (or a hundred posts) for another time (maybe then I'll use language less simplistic than "fixing"), but I'll end by asking a different set of questions:

1.) Can the local schools become, and remain, excellent without first improving the neighborhood?

2.) If so, can the local schools become, and remain, excellent without subsequently altering the neighborhood in which they're located?

3.) Can anybody point to one example where a poor, crime-ridden, and disorderly neighborhood housed and sustained excellent neighborhood schools for multiple decades with no significant changes to the community before, during, or after this time period?

Who's The New Teacher?

While everyone else was off watching "Waiting for 'Superman'" this weekend, I stumbled upon Tony Danza's new show where he becomes a teacher in Philly.  I was initially highly skeptical of a show where a former star actor becomes a teacher at an urban school to make it back on tv and show the world how great he is.  But after watching the first episode, I was left feeling more sorry for Tony Danza than anything else.

I don't know what was in the 99% of the footage that was edited out, but two things seem quite apparent to the viewer as the school year starts out: 1.) Mr. Danza isn't a very good teacher; 2.) He spends a lot of time dwelling on worries and self-doubt, both about his teaching abilities and other things in his life.

I'd also add that he comes across as a genuinely enthusiastic and charismatic person -- somebody who could probably be a great teacher, though it's not obviously apparent that he has the knowledge base requisite to teach 10th grade English.  That said, these were a few of my thoughts as I watched the episode:

*Wow, the Asst. Principal (Ms. DeNaples) is incredibly rude to him.  Are you an administrator in an urban school that wants to make new teachers quit?  Treat teacher the way she does.

*The Principal, on the other hand, is tough but fair.  She informs that he'll get "all the support" he needs, but she also says that he has to know that "if this doesn't work, you're out of here".

*The kids are pretty blunt.  One says flat-out: "he's a lot older than the videos I saw of him."  Another interrupts his first(?) day of class to ask if he's nervous and points out that he's sweating through his shirt.  This alleviated some of my concerns that that the students would be in awe of him and the tv cameras and act like docile angels.

*It seems like in every camera shot his mentor teacher (not sure what his exact title was) was there.  They seemed to be constantly talking over how Danza's lessons had gone (not well).  Most teachers in Danza's situation wish they had an ever-present mentor like that.  On another note, the fact that his mentor was at least 20 years younger than him was somewhat amusing.

All in all, it wasn't wholly unrepresentative of what life is like as a new teacher in an urban school.  I'm still skeptical of his motives, how it will affect the kids (he apparently lives in LA, so it seems unlikely he's planning on doing this for more than one year), and the way the footage is edited.  I also could find no indication that he was teaching more than the one English class in the show or on the website (that would make life an awful lot easier).  I found the show neither incredibly riveting nor incredibly aggravating to watch.  I'm not going to boycott it, but I'm also not going to schedule my life around the airtime to ensure I catch every remaining episode.

If I do watch a few more, though, I'll be watching to see how they frame the middle and end of the year.  Right now they're clearly clearly portraying him as a struggling teacher.  Will the focus of the rest of the series be on his personal struggles and crises, on the difficulties of teaching in an urban school, or be made into a cheesy "Danza conquers Philly" tale of success and redemption?  Is Danza the "Superman" for whom we're waiting?

p.s. Linda Perlstein had a good take on the show a couple weeks back that's worth reading

update: Just came across this article on the series in the LA Times (which, by the way, has the worst website of any online newspaper I've read recently -- it looks more like the National Enquirer than a major national newspaper).  Apparently Danza taught last year, only taught two classes per day, was only filmed teaching until January, and had to have the teaching coach sit in on every lesson b/c he wasn't full certified.  And it mentions a "casting call" for the students in his class -- not quite sure what that means.  It also sounds like he ended up considering the year a success despite the rocky start.

Friday, October 1, 2010

How to spend $100 million

Mark Zuckerberg's $100 million dollar gift to Newark schools has recieived a lot of attention.  This despite the fact that it amounts to only about a 2% increase in spending if spread across Newark's schools.  In other words, I think we can all agree that if his money simply gets added to Newark's general funds that not much will happen (even after accounting for the $100 million in matching funds that the grant requires).

So, how should Zuckerberg, or any other philanthropist, spend their money if they actually want to change schools?

Robert Pondiscio writes a lot of really smart things, but I don't see much merit in his idea that they money be used to set up an "X Prize" in education rewarding, for example, the first urban district to close the achievement gap in NAEP scores for 3 consecutive years.  I have two reasons for believing that the money would sit in the bank for an awful long time.  First, I don't buy the argument that motivation is a humongous problem in education.  There are certainly assorted unmotivated teachers, principals, adminstrators, etc. scattered throughout the country, but I don't think motivation levels are at a point where a little extra money can raise them enough to produce large gains.  Second, if dramatically altering the structure of a school district enough to close the achievement gap were an easy thing to accomplish (requiring just a little extra blood, sweat, and tears) then somebody would have done it by now.

Andrew Rotherham doesn't push the X Prize idea, but gives Zuckerberg and others a few tips in his TIME column yesterday.  I think they're mostly pretty good points (though I think the notion that one must necessarily make everybody hate them to get anything done is overplayed).  But if I had extra money lying around that I wanted to use to revolutionize education, I'm not sure I'd be much closer to figuring out how to use it after reading that column.

Maybe it's because I'm not fabulously wealthy, or maybe it's because there are so few large-scale replicable successes backed by empirical evidence in our education system, but I genuinely have no idea what I would do with that kind of cash.  But I think there's one golden rule that anybody who donates to education must keep in mind: about a million different things impact educational performance over a number of years, and changing one of those things for a short period of time isn't usually going to have that much of an effect.  In other words, it's really, really hard to change a child's (yet alone a school's or city's) educational trajectory.

Rotherham writes that philanthropists should "go big or go home."  Indeed, paying for large-scale interventions is enormously expensive.  The Harlem Children's Zone, for example, has a budget of over $75 million this year alone.  That's for about 10,000 children -- one-quarter of Newark's 40,000 students.

Which would lead me to consider two possible approaches:

1.) Focus on something small and do it well.  Maybe that's a single school or neighborhood or a single activity or topic.  I'd much rather sponsor, say, an intensive and well-run debate program that students attend daily than buy a whole bunch of math software, reading books, and training videos that are used occasionally.  Similarly, I'd rather provide 1,000 kids with high-quality health care, after-school acitivities, counseling, cooking classes, and mentors than provide 40,000 kids with, say, tutoring if they want to attend.  Not because I don't care about the other 39,000 kids, but because I'd rather make the difference in the lives of 1,000 kids and then try to find more people and money to scale up than simply throw money at 40,000 and not really accomplish anything.

2.) Do something that will leverage action.  This is where Pondiscio's not completely off-track with X Prize idea.  I don't think the Race to the Top money will do very much, but the competitive grant process certainly spurred an awful lot of legislative changes.  But I'd rather incentivize actions than results.  Teachers, principals, and superintendents are already motivated to help kids learn more, but they may not be motivated to adopt a new training program, curriculum, salary structure, evaluation system, etc. in a timely fashion.

Tuesday, September 28, 2010

Exceptions are Exceptional

Many people noted the article on the turnaround of Brockton HS in yesterday's NY Times.  But I didn't notice anybody critiquing the headline.  The piece is titled "4,100 Students Prove 'Small is Better' Rule Wrong".

Yipes.  Regardless of whether or not small schools really outperform larger ones* or whether or not Brockton has really turned around, the headline is absurd.  Exceptions don't disprove rules in social science.  Exceptions are, by definition, exceptional.  Can we please take that into account when we discuss education?

Someone driving home safely while drunk doesn't prove that it's not dangerous to drink and drive.  A sole survivor of a plane crash doesn't prove that plane crashes don't kill.  A child from a troubled neighborhood graduating from Harvard doesn't prove that growing up in a troubled neighborhood doesn't impact one's educational performance.  And one large school excelling doesn't mean that small schools aren't generally better.

update: GothamSchools outdoes the Times yet again by linking to the article with a far more responsible phrase -- "Brockton High School in Massachusetts dispels the myth that only small schools can improve."


*the evidence that small schools actually are better is, to my understanding, weak at best -- but that's not relevant to the point I'm making

Thursday, September 23, 2010

The Primary Purpose of Merit Pay

The most popular opinion of the last few days seems to be that the primary purpose of merit pay is to re-shape the teacher labor force by attracting and retaining better teachers. The notion that performance incentives would motivate teachers to perform better in the classroom has been implicitly or explicitly derided as silly and/or unimportant.

Did I miss something? Maybe I need to do some archival research, but I could've sworn that before the release of the results there weren't many merit pay proponents making this argument. But since learning of the lack of effect on standardized test scores in the Nashville experiment, it seems to be the only one I hear.

After learning of the results, Rick Hess wrote that

The second school of thought, and the one that interests serious people, is the proposition that rethinking teacher pay can help us reshape the profession to make it more attractive to talented candidates, more adept at using specialization, more rewarding for accomplished professionals, and a better fit for the twenty-first century labor force.

and the Washington Post quotes Eric Hanushek saying

The biggest role of incentives has to do with selection of who enters and who stays in teaching - i.e., how incentives change the teaching corps through entrance and exits . . . I have always thought that the effort effects were small relative to the potential for getting different teachers. Their study has nothing to say about this more important issue.

and Tom Kane writes:

the impact of the specific incentive they tested depends on what underlies the differences in teacher effectiveness–effort vs. talent and accumulated skill. I’ve never believed that lack of teacher effort–as opposed to talent and skills–was the primary issue underlying poor student achievement gains. Rather, the primary hope for merit pay is that it will encourage talented teachers to remain in the classroom or to enter teaching.

the Obama administration's official position seems to align with that too.  Here's how the same Washington Post article described their repsonse:

While this is a good study, it only looked at the narrow question of whether more pay motivates teachers to try harder," said Peter Cunningham, assistant U.S. education secretary for communications and outreach. "What we are trying to do is change the culture of teaching by giving all educators the feedback they need to get better while rewarding and incentivizing the best to teach in high-need schools, hard to staff subjects. This study doesn't address that objective.

Maybe I'm wrong and there are more people that would've agreed with these four statements a few days ago than I think, but there were certainly more than a couple people arguing that performance incentives would increase teachers' motivation, improve their classroom performance, and subsequently increase the academic performance of their students.  I've had conversations with people who've directly told me that lack of motivation is a huge problem in teaching and that providing proper incentives would fix this.

Without more research, I can't tell you whether people have conveniently changed their mind about the primary purpose of performance pay or whether those who believe it should be used primarily to alter the teacher labor force are now simply stepping to the forefront while those who believed in its motivational potential are shrinking into the background.  But I'd guess that it's a little of both.

On the plus side, might everyone now agree that teacher pay should be re-fashioned with the primary goal being to encourage the recruitment and retention of excellent teachers?  Do I hear a consenus emerging?  I guess time will tell . . .

Wednesday, September 22, 2010

Nashville Incentive Pay Experiment: Results Wrap-Up

The National Center on Performance Incentives released the final report on the Nashville performance pay experiment (known as POINT) yesterday.  The press release is available here, and the full report is available here.

The study involved 296 middle school math teachers in Nashville who were assigned to either a treatment group (eligible for bonuses of 5, 10, or $15,000) or a control group and then tracked for three years.

The main result was that students assigned to bonus-eligible teachers did not perform any better than students assigned to treatment group teachers.  The lone exception were 5th grade students in years 2 and 3 of the study, but the gains did not persist through the end of 6th grade.  The main portion of the executive summary reads as follows:

POINT was focused on the notion that a significant problem in American education is the absence of appropriate incentives, and that correcting the incentive structure would, in and of itself, constitute an effective intervention that improved student outcomes.
By and large, results did not confirm this hypothesis. While the general trend in middle school mathematics performance was upward over the period of the project, students of teachers randomly assigned to the treatment group (eligible for bonuses) did not outperform students whose teachers were assigned to the control group (not eligible for bonuses).

Prior to implementation, researchers calculated that 55% of teachers would be able to obtain bonuses if their students answered only an additional 2-3 questions correct on the state test.  Across the three years of the study, 33.6% of bonus-eligible teachers earned a bonus in one or more years.

This means that most teachers did not dramatically improve, even for one year.  It also means that the tendency of test scores to bounce around significantly did not result in different random groups of teachers receiving a bonus each year.  About two-thirds of teachers in the treatment group never received a bonus.  And about two-thirds of the teachers who received a bonus earned one in multiple years.

Of potentially more import are the results from the teacher interviews and surveys.  These will continue to be analyzed in the coming months, but right now the main takeaway point is that teachers in the treatment group really didn't report changing much at all.  About 80% of teachers reported that they were already working as hard as they could before the incentives were implemented and were therefore unable to work any harder afterward.

There are two potential bright spots, however, for merit pay proponents:

1.) The project ran smoothly (e.g. the right teachers received the right bonuses at the right time) and didn't suffer any major backlash.  That this was truly a partnership between union, university, district, and other groups probably helped in this regard.

2.) It's unclear right now, but the bonuses may have had a small effect on the patterns of teacher attrition for those in the bonus group.  27 teachers in the control group left to teach in another district while only 15 teachers in the treatment group did.  The numbers are small, so conclusions are hard to draw, but the most popular criticism of the study by performance pay advocates seems to be that it didn't shed any light on how performance pay might affect teacher recruitment and retention.

There was also no evidence that treatment group teachers were successful in gaming the system by having more kids (or lower performing kids) removed from their class, preventing new students from transferring into their classes, focusing more efforts on math instruction at the expense of other subjects, or helping their students cheat on the state tests.

What does it mean?

I wrote before that this was one of the most important studies this decade.  That said, it's still just one study.  And we have to be careful about how we extrapolate from the results of any one study.  The main question the study was set up to answer was whether offering performance incentives to individual teachers would result in their students performing better on standardized tests.  The study offered no evidence that this was the case, and little reason to believe that this would be the case for similarly designed incentive systems.

Why is this so?  The main reason seems to be the lack of any major changes by teachers.  But there are a couple of other possibilities.  Just because teachers reported that they didn't change anything doesn't necessarily mean that they didn't change anything.  So it's not out of the realm of possibility that at least some of the teachers made changes but that these changes didn't yield subsequent results.  It's also possible that, despite the fact that teachers received the project fairly well, they wanted to prove the working hypothesis wrong -- that is, that teachers eligible for a bonus, at least on some level, resented the fact that somebody thought they'd teach better if they were offered a carrot, and decided to not work harder despite the carrot looking awfully delicious.  At the same time, the control group teachers could've worked a little harder to prove that they were plenty motivated to teach solely because they wanted their students to succeed.  While the answer is likely a little of all of the above, I tend to think the most likely scenario is that teachers simply weren't all that much more motivated by the prospect of a bonus fairly far down the road (teachers were paid the following November).

What it doesn't mean is that all merit pay schemes forevermore are doomed to abject failure.  We don't know if different types of bonuses awarded in different ways (shorter time spans, group awards, non-monetary awards, etc.) might have a larger effect, and we know little about how performance pay affects the long-term make-up of the teacher labor force.  At the same time, it does call into serious question the application of the overly simplistic homo economicus model used by economists.  My gut feeling is that economists tend to view teacher motivation and the teaching profession in an overly simplistic manner guided too much by basic economic theories and not enough by the literature on the sociology of teaching or the psychology of motivation.

Where do we go from here?

NCPI has another study utilizing team-based incentives that should be out at some point over the next couple of years.  In addition, other non-randomized studies of the myriad incentive systems that have sprouted all over the country the past couple years are certainly underway as well.  I'm not sure why the NCPI researchers chose to claim that this was "the first scientific study of performance pay ever conducted in the United States," since the definition of "scientific" is hotly debated, but the literature base will certainly continue to grow in the coming years regardless.

In addition to the continuing analysis of the data from this study, there are plenty of opportunities for other researchers and funding agencies to examine questions surrounding the impact of merit pay on the teacher labor force, on school culture, on student outcomes other than test scores, and many other areas.

In the meantime, it's important that merit pay opponents not claim that this study proves once and for all that merit pay does not, and will not ever, work in schools.  If nothing else, it appears likely that better teachers got paid more money than worse teachers -- which is arguably an improvement on the current system.  And, at the same time, it's important that merit pay proponents not claim that this study is meaningless and that we should recklessly proceed with merit pay in schools at breakneck speed.

Merit pay has grown considerably more popular among teachers over the past decade, so it's eminently possible that districts and unions can work together to design and implement better, more nuanced merit pay systems that might have a better chance of success.  I don't think anybody would argue that the status quo is the perfect system, so simply saying that merit pay won't work isn't a solution.  But these results indicate that we should proceed with caution.  The assumption that teachers will work harder for financial incentives is now a dangerous one to make and should be made with caution.  As such, the performance pay systems that will undoubtedly continue to emerge should begin with a more nuanced and informed understanding of the practices and motivations of current and prospective teachers.  We can only hope that an informed and open-minded approach from both sides will eventually result in compensation systems that attract and reward good teachers in ways that current teachers find meaningful and fair.


For more information, see my previous posts on the experiment:


Part 1: Background Info
Part 2: What to Look For
Part 3: Why it Matters
Part 4: What We Can Learn
Live-Blog of Results

Tuesday, September 21, 2010

Live-Blogging the Release of the Nashville Performance Pay Experiment Results

Today marks the release of the National Center on Performance Incentives' report on the Nashville performance pay experiment (known as POINT).  The full report will be available on their website after the discussion.  In the meantime, I'm going to provide you with some snapshots of what's being said and written.  Please check out my previous posts on this study as well.  A live, streaming, video of the press conference is online here.  The press release (which is pretty good) is now available here.  The full report is available here.

Previous posts:
Part 1: Background Info
Part 2: What to Look For
Part 3: Why it Matters
Part 4: What We Can Learn

12:50pm: The press conference has begun.  I'm going to begin by posting the summary statement from the executive summary of the report:

POINT was focused on the notion that a significant problem in American education is the absence of appropriate incentives, and that correcting the incentive structure would, in and of itself, constitute an effective intervention that improved student outcomes.

By and large, results did not confirm this hypothesis. While the general trend in middle school mathematics performance was upward over the period of the project, students of teachers randomly assigned to the treatment group (eligible for bonuses) did not outperform students whose teachers were assigned to the control group (not eligible for bonuses).


Before you get excited, or disappointed, about this, bear in mind what I've written before -- the most important things we can learn from this study aren't what happen to test scores, they're insights into teacher behavior from the interviews and surveys.  Keep checking back for more details on this, and other, topics.

Also, please keep in mind that this study does not definitively prove either that merit pay systems are a bad idea or a good idea.

1:00pm: The number of teachers who received bonuses remained steady throughout (40, 41, 44), but the number of eligible teachers declined significantly (143, 105, 84) -- meaning over half of the teachers received a bonus for being above the historical 80th percentile of teachers in the last year.  This could mean that less successful teachers tended to leave the study -- whether by switching subjects, schools, or careers.  Or it could mean that the tests became easier and more teachers were rewarded.  I would say that it could mean that it took three years for the incentives to have an effect, but the treatment group did not outperform the control group -- even in the final year.

from the executive summary: "attrition of teachers from POINT was high. By the end of the project, half of the initial participants had left the experiment."

The report says that the differences in attrition -- and reasons for attrition -- between control and treatment groups was not statistically significant.  But it reports that 27 control group teachers, and only 15 treatment group teachers, left the study because they'd switched districts during the experiment.  I'd like to know more about this and whether this might be evidence that incentives made teachers slightly more likely to remain in the Metro Nashville school system.

about teacher attrition, from the report:


Teachers who left the study tended to differ from stayers on many of the baseline variables. Teachers who dropped out by the end of the second year of the experiment were more likely to be black, less likely to be white. They tended to be somewhat younger than teachers who remained in the study all three years. These dropouts were also hired more recently, on average. They had less experience (including less prior experience outside the district), and more of them were new teachers without tenure compared to teachers who remained in the study at the end of the second year. Dropouts were more likely to have alternative certification and less likely to have professional licensure. Their pre-POINT teaching performance (as measured by an estimate of 2005-06 value added) was lower than that of retained teachers, and they had more days absent. Dropouts completed significantly more mathematics professional development credits than the teachers who stayed.Dropouts also tended to teach classes with relatively more black students and fewer white students. They were more likely to be teaching special education students.

I'm going to need a little time to digest that, but the next table demonstrates that treatment and control group teachers, in all three years, did not differ in terms of effectiveness (as measured by tests) in the years prior to the experiment -- meaning that more effective teachers didn't seem more likely to stay if they could possibly earn bonuses.  Treatment group teachers, however, were six percentage points less likely to leave the middle school in which they started during the three years.

1:03pm: There were, however, positive effects for 5th grade teachers in the 2nd and 3rd years, though the effects did not persist until the end of 6th grade.  The 5th grade teachers had the same students all day for multiple subjects, so it's possible that they shifted focus to math instruction or that they simply knew their students better and were able to get better results.  The center did analyze results on other subject tests and found no differences that would indicate teachers ignored these subjects and only focused on math.

33.6% of the treatment group received a bonus in at least one year (out of 152, 16 won once, 17 twice, and 18 thrice).  Analysis done by the researchers prior to the experiment found that 55% of teachers were within a few more correct questions per student of attaining scores that would earn them a bonus.

1:08pm: Here are some other interesting tidbits from the executive summary, report, and press conference (which is now in Q&A):

-80% of teachers reported that they were already working as hard as they could and didn't change their effort due to the opportunity to earn an incentive.

-from the executive summary: "The introduction of performance  incentives in MNPS middle schools did not set off significant negative reactions of the kind that have attended the introduction of merit pay elsewhere. But neither did it yield consistent and lasting gains in test scores. It simply did not do much of anything."

-from the report: "From an implementation standpoint, POINT was a success. This is not a trivial result, given the widespread perception that teachers are adamantly opposed to merit pay and will resist its implementation in any form."

-I didn't mention this before, but the placement of teachers in control/treatment groups -- and whether or not they received a bonus was, officially, confidential.  The center didn't distribute this information to principals or teachers and participating teachers signed statements saying that they wouldn't tell anybody their status.  It looks like this was at least moderately successful, as about 75% of teachers reported that they didn't know if anybody had won a bonus in their school.

-There were no differences in student attrition between groups -- meaning that there's no evidence that bonus-eligible teachers were more likely to get problem students removed from their class.  There were also no differences in students enrolling late, and students who left treatment teachers' classes were no lower scoring.

1:12pm: Dale Ballou responds to a question by saying that test scores bounced around a lot before the experiment, making it "difficult to extrapolate" from the data prior to the start of the experiment and tell whether teachers were doing better post-bonus than pre-bonus.

1:16pm: From the report, here are the results from the teacher survey.  In short, nothing huge or shocking (note TCAP is the TN state test):

There are few survey items on which we have found a significant difference between the responses of treatment teachers and control teachers. (We note all contrasts with p values less than 0.15.) Treatment teachers were more likely to respond that they aligned their mathematics instruction with MNPS standards (p = 0.11). They spent less time re-teaching topics or skills based on students’ performance on classroom tests (p = 0.04). They spent more time having students answer items similar to those on the TCAP (p = 0.09) and using other TCAP-specific preparation materials (p = 0.02). The only other significant differences were in collaborative activities, with treatment teachers replying that they collaborated more on virtually every measured dimension. Data from administrative records and from surveys administered to the district’s math mentors also show few differences between treatment and control groups. Although treatment teachers completed more hours of professional development in core academic subjects, the difference was small (0.14 credit hours when the sample mean was 28) and only marginally significant (p = 0.12). Moreover, there was no discernible difference in professional development completed in mathematics. Likewise, treatment teachers had no more overall contact with the district’s math mentors than teachers in the control group.

1:20pm: One question I really want answered (that I don't see in the report) is how the bonus winners and losers differed (experience, school type, etc.), if they differed at all.  A questioner just asked something along those lines, and the response was that they've just begun to look at that, but that they're wary to draw too many conclusions about this because the sample sizes will get smaller the more focused the question is.

1:27pm: not sure this is quite verbatim, from the panel, but it's a good point nonetheless: "just because we didn't find an effect from doing incentives this way doesn't mean you wouldn't find results by doing it another way."

1:43pm: The questions aren't really helping to advance the discussion all that much -- in part b/c there are too many speeches and not enough questions.  The line at the microphone is now too long for me to get a question in before the end of the session, but here's what I'd ask:

1.) There are two assumptions behind performance pay that this study was designed to test: one is that teachers will respond to the opportunity to earn incentives by changing their practice, and two is that these changes in practice will result in better student performance.  The report says that there were very few differences in reported teacher behavior between treatment and control groups, but is there enough data to draw any conclusions about the performance of those who did report making changes?

I had a second question, but it now completely escapes me.  I'll try to find an answer to at least that question, at least.  That's going to do it for now, but expect a more concise summary of the report's findings in the next 24 hours or so.

Wait, just kidding: here are the other four questions that are rattling around in my brain right now.  I ran into one of the researchers in the hallway and asked him the question above in addition to the first three below.  The answer was basically that they're good questions, and ones they plan on looking into more -- the focus for today was completing the analysis of the trends in test scores.

2.) What, if any, differences were there between bonus winners and losers (e.g. were winners more experienced, teaching in better schools, teaching higher/lower performing students, teaching a different subject, etc.)?

3.) What, if any differences, were there between the reactions of bonus winners and losers (e.g. more likely to stay/leave, more/less critical of experiment, more/less enthusiastic, more/less likely to report subsequent changes in behaviors)?

4.) I think 15 treatment group and 27 control group teachers left the district during the experiment.  Can we consider that statistic (and maybe any relevant survey/interview questions) evidence that teachers might be less likely to leave an urban district if they can earn a bonus there but not elsewhere?

5.) A much higher percentage of the treatment group teachers (over half, compared to less than one-third) won bonuses in the final year.  Is that simply because the less experienced teachers were much more likely to leave their assignments, because the test changed, or something else?

Now that's (really) all for now.

Shame on Rick Hess

Rick Hess' post on the Nashville performance pay experiment yesterday received a lot of attention.  In the post, Hess, a merit pay proponent, argues that the results of this experiment will "tell us nothing of value".

I found the post somewhat surprising (as did many others, I think).  And I'm somewhat ashamed to admit that, as I read it, my inner cynic said "I wonder if Hess knows that the results weren't spectacular and he's just trying to discredit them ahead of time?".  But I quickly shook that off, since I tend to give people the benefit of the doubt.  Hess raises a lot of good points, and he seems sincere when he warns readers about the dangers of either side making wild claims based on the results.

Well, I was wrong.  Wrong to trust Rick Hess.  Re-read his post again, but this time be aware that Hess already knew the results of the experiment -- and that the results weren't what he wanted.

That's right.  Hess already knew the results.  I was informed of this by a reliable source familiar with the project -- who also tells me that Hess has spent the last few years lauding the researchers and chomping at the bit to rub the positive results in the faces of unions across the country.  And now that they didn't turn out the way Hess had hoped, he's decided to pretend that he didn't know the results and tell everybody that the results don't matter.

Regardless of one's point of view, that's simply dishonest and disgraceful.  Shame on Rick Hess.

Now, that's not to say that Hess is the only one being dishonest.  I'm sure that there are plenty of people on both sides of the debate that have been, and/or will be, dishonest to some extent.  So I don't want this post to be construed as vindication that opponents of merit pay are always victims and proponents are always evil liars. 

But, at the same time, I find this post particularly galling.  Hess has made a name for himself lately by being somewhat independent-minded on a number of issues.  Sometimes I agree with him, sometimes I disagree -- but he always has something interesting and insightful to say.  In short, he gave me good reason to hold him to a higher standard than most policy wonks who too often seem incapable of seeing an issue from multiple sides.  Indeed, a good portion of what he wrote yesterday is useful and insightful (though he takes it too far), and that's what made the post so remarkable.  But now I don't know what to believe.  A number of his points should still stand (for example, examining the ways in which merit pay affect recruitment and retention is important), but I don't know which ones are sincere and which ones are not.

I respect honesty.  I respect people who seek to advance the discussion of education policy.  Today, I do not respect Rick Hess.