Thursday, November 18, 2010

Value-Added vs. Batting Averages

Stephen Sawchuck reports that a new report on value-added scores compares them to measures in other fields, including batting averages in baseball.  I think the batting average analogy is a good one.  Here's why:

When we look at a player's batting average (the number of hits divided by the number of at bats), we are provided with some information that reflects how good they are at baseball.  The value of this information increases when the player has more at bats and plays for more years.  And the larger the gap is between two players' batting averages, the more certain we can be that one player is better than the other.

But, at the same time, a player's batting average in one given year gives us only a small amount of information about that player's ability.  Averages vary year to year: players suffer injuries, become distracted, grow older, switch teams, play with different teammates, and so on.  For example:

Who's a better shortstop: Alexei Ramirez or Derek Jeter?  The former hit .282 this year while the latter hit .270.

Who's a better first baseman: Aubrey Huff or Ryan Howard?  The former hit .290; the latter .277.

Batting average doesn't do a good job of measuring power, speed, defense, leadership, or any of a million other desirable characteristics.  And for that reason, teams have moved beyond batting average when evaluating and signing players.  It seems like every year a new stat bubbles up that measures a player's ability in one facet of the game.

Adam Dunn is a career .250 hitter who plays awful defense, but will make millions of dollars next year because he'll probably hit 40 home runs yet again.

Mike Cameron, a .250 lifetime hitter, signed a 2-year, $15 million dollar contract last year as a 36 year-old.  His power (about 25 HR per season recently) didn't hurt, but the main motivation was his defense.

When we look at a more sophisticated (though still flawed) computation of how many wins a player added over an average replacement player (Wins Above Replacement, or WAR), we see that Jose Bautista, who hit .260 this year, ranks sixth.  The fifh-ranked player, Adrian Beltre, has a lifetime average of .275.

At the same time, nobody would argue that batting average is meaningless -- especially over longer periods of time.  There are plenty of hall of famers with career averages over .300, but none that I know of with averages of .220.

In other words, the batting average analogy is an excellent one for value-added scores because they have four very important things in common:

1.) Anybody who tells you that it is totally meaningless is totally wrong.
2.) When you see large differences over long periods of time between two people, you can be pretty sure that the one with the larger number is better.
3.) At the same time, that number in and of itself gives us very little information about how good somebody is, particularly over a shorter period of time.
4.) There are other skills not measured by the number that need to be taken into account when evaluating the person.

Unlike baseball, we don't have better statistics in evaluation.  We don't have something like on base percentage, yet alone change in student motivation over replacement teacher.

What does this mean for schools and value-added scores?  If we designed the perfect evaluation system, given current knowledge and tools, value-added scores would have to be included.  The only really compelling reason to leave them out is the potential for misuse by people who don't understand #'s 2-4 above.  Partly for that reason, it's important that value-added scores be only one component of how a teacher is evaluated.  And principals should be looking for large differences over long periods of time, not tiny year-to-year differences when making hiring/firing/tenure decisions.

Opponents of value-added scores would be better off arguing that they can be useful -- but only in some circumstances -- while proponents should be looking for supplemental measures to boost the meaningfulness of evaluations.  And both sides should keep in mind that the majority of teachers don't teach subjects tested by state tests anyway, so in addition to measuring a teacher's impact on students other than on standardized tests (e.g. motivation, attitude, self-control, creativity, interpersonal skills, etc.) we should also keep in mind that we need better ways to evaluate a teacher's impact on students in other subjects beyond 3rd-8th grade English/math as well.

In short, value-added scores can aid our evaluations of teachers . . . but only a little bit, and only if used properly.

Tuesday, November 16, 2010

"The Shadow Scholar"

In case you missed it, The Chronicle of Higher Education recently published an article written by "Ed Dante" (a psuedonym) about his/her experience writing papers for desperate/lazy college students.  I'd put "The Shadow Scholar" in the "must read" category for all those interested in higher education, and in the "well worth your time" category for everybody else.

The author makes decent money (on pace for 66K this year) writing papers for students and says that a student has never complained about getting caught.  The process and stories are interesting, but the author makes two main points:

1.) A lot of students are absolutely awful at writing and more should be done about this.
2.) A lot of assignments are ridiculous and drive students to look for other solutions (including purchasing papers)

I wholeheartedly agree with the first and half agree with the second.

The comments are almost as interesting as (though more tedious than) the article itself.  Between the author, the commenters, and my own head, there seem to be some competing hypotheses as to why a not insignificant number of students cheat in one way or another, particularly on written assignments.  In no particular order, they are:

1.) Professors assign meaningless/useless/stupid assignments
Why?
-large class sizes prevent individualized assignments
-laziness
-following conventions/norms
-more concerned with research than teaching

2.) Students don't know how to write
Why?
-they're non-native speakers
-they didn't learn how to write during K-12 schooling
-their college profs don't take the time to teach them how to write

3.) Students are unethical
Why?
-they don't understand what's ethical and what's not
-ethics and morals aren't emphasized enough in school
-a few bad apples make everyone look bad
-lots of pressure to get good grades; little pressure to behave ethically
-they don't value the assignments they're asked to complete or the courses they must take

4.) Unethical people make it easy to cheat
Why?
-some people care more about money than ethics
-people think they're doing others a service since they think the assignments are stupid


I don't think any of these fully explain the situation, but I do think they all shed a little bit of light on it.  In short, there are plenty of people and institutions at whom we can point fingers . . . but wouldn't it be more productive if we instead focused on fixing the problem?

Thursday, November 11, 2010

More on NYC's New Chancellor

The NY Times has a piece on Bloomberg's secretive search for the next schools chancellor.  A Times blog follows up with a mildly amusing post asking if anybody was actually interviewed during this search (and if they were, or know someone who was, to write in the comments section). 

The secrecy surrounding the search is interesting, but seems like it could be a red herring -- the ability of Ms. Black to lead the schools is ultimately more important than exactly how she was picked . . . with the caveat that if the process makes her appear illegitimate, it could undermine her authority.  Witness, for example, this online petition that a teacher friend was asking people to sign asking the state not to grant a waiver allowing her to serve as chancellor despite no educational background.

It's somewhat surprising to me that she's been greeted with this much skepticism given how many district leaders there are out there with no educational backgrounds, including her predecessor, Joel Klein.  It will be interesting to see if she's able to win over her detractors or if the next three years will be one long battle.

Wednesday, November 10, 2010

NYC's New Chancellor

The shocking news of the evening is that Joel Klein has stepped down as chancellor of NYC's schools (and accepted a job with News Corp.) and been replaced by former magazine exec Cathleen Black.  I knew nothing about Cathleen Black before a few hours ago, and have no idea whether she'll be a good leader for NYC's schools or not, so I'll leave the guessing to others.

But what I found most interesting about this is that she (a.) has no background in education and (b.) is described by the NY Times as someone who "earned a reputation in publishing as a tough-minded chief executive who never left her employees guessing what she wanted".

It's almost cliche at this point to hire a superintendent who is both an outsider and "tough".  I have no idea whether she'll still be described as "tough-minded" in years to come, so we'll have to wait and see on that one, but the absence of an educational background is worth more examination.

On the one hand, it's easy to see why somebody from the business world is attractive to a mayor or board of ed.  If your goal is to transform the school system as fast as possible, you want somebody who will come in and get the job done without much of a fuss.  And given that the vast majority of educational leaders have extensive backgrounds in education and that our system still has many problems, it would be easy to dismiss such a background as unnecessary.

And I don't think that such a view would be entirely without merit.  Ed. Schools probably catch more flak than they deserve, but it's pretty clear that many of them don't offer very high-quality or practical programs.  And there's a certain amount of value in bringing in somebody that hasn't been a part of the system -- there's an Icelandic expression that's relevant here: "the eye of the outsider is sharp".

At the same time, there's no reason that an outsider couldn't be somebody with experience outside of NYC instead of just outside of schools.  Having the negotiating skills or steely resolve necessary to stare down the union may or may not produce positive results, but it's also possible that experience as a teacher might more readily convince the union's constituents that new chief wants to work with them instead of against them.

All in all, I can't think of another field in which expertise or experience is treated with less regard than it is in education -- probably largely because virtually every resident of this country spent a good portion of their lives attending school.

Of course, if this trend continues, at some point in the future we'll read about some rebel mayor deciding to pursue the outside-the-box strategy of hiring somebody with a lot of educational experience to serve as superintendent since none of the business whizzes have yet found the magic cure-all for schools.

update: The NY Times has a forum on whether a district leader needs to have a background in education to succeed.  The arguments largely mirror those outlined above.  Clara Hemphill asks “would the mayor have named a publishing executive as head of the Department of Health?”  Richard Kahlenberg points out that a background as a teacher might make a leader more readily accepted by the rank-and-file.  And Marcus Winters and Neil McCluskey of the conservative Manhattan and Cato Institutes argue that an outsider perspective can more readily advance new ideas, especially since the old ideas that education folk have been trying have yet to cure all that ails our schools.

Friday, November 5, 2010

Today's Random Thoughts

*One college professor writes that their teacher prep program is losing many of their brightest prospective history teachers because the students are skeptical that they'll have much over the content of their courses and turned off by the battering teachers take in the press.  I tend to be skeptical of those who claim that there's a seismic shift -- with every single person who's creative abandoning teaching -- but I don't doubt that there is some serious dissatisfaction among some and, correspondingly, at least a small amount of attrition . . . I would expect it more from early-career teachers than from prospective teachers, so I find this piece both somewhat surprising and somewhat worrying.  Since we seem to be focusing a lot of time and effort lately into proposals that will attract and retain better teachers, we should probably investigate these types of claims before deciding which policies to pursue (hat tip: EP).

*A note to all the worried parents and college students out there: what the student chooses to major in doesn't really matter all that much when it comes to earnings later in life . . . in fact, it might be better to study what one is passionate about than simply picking what one deems most practical.

*Speaking of worried parents and teenagers, this NY Times/Chronicle of Higher Ed joint piece has a graphic that demonstrates fairly well the rise in the number of colleges that high schoolers have applied to over the past 20 years, though they miss out on the most explosive growth -- people applying to 10, 20, 30, or more colleges.  When I was in HS, it was typical for ambitious kids from well-to-do families to apply to about 8 or so; my understanding is that it's now typical for kids at private and upper-class public schools to apply to 20 or more schools (which the common app and online applications have made much easier to do).  In one sense, this is a savvy thing for kids to do -- with ever-declining acceptance rates, why not make sure you have every base covered?  But, at the same time, unless they're getting fee-waivers, that's a lot of money we're talking about for college applications.

*A group of researchers at Cornell write about ways to redesign lunch lines that will result in kids eating healthier food.  I always appreciate it when people find little tweaks that cost almost nothing, but get big results by nudging people in the right direction . . . but I have to wonder if simply doing these types of things wouldn't be abdicating our responsibility to teach kids about nutrition and responsibility.  If they don't take chocolate milk b/c it's behind the white milk and don't take ice cream b/c it's in a covered container, that's good for today -- but what will they do when they go to the grocery store as adults?

Thursday, November 4, 2010

I'm Still Alive

Apologies for the sudden dead air, but I've been swamped with dissertation and job-related activities. I'll try to update at least once each week over the next few months, but I can't make any promises.

Quick note while I'm here: everybody seems to be wondering if yesterday's election results will bring us compromise or gridlock for the next two years. If it's the former, education might be a logical starting point . . . I can't think of another major issue where the parties are closer to agreeing.