Limitations of Research and the Headlines that Ignore Them

I'm in the middle of finals week, so I only have a couple minutes, but I think this is worth looking at for a second.

A grand headline in the NY Times today reads "Study Suggests Math Teachers Scrap Balls and Slices."  The article summarizes a study being published today that found people learned math better when taught abstractly (think: formulas) than when taught using manipulatives (the balls and slices to which the headline refers).

The finding is extremely interesting. And I saw it echoed and trumpeted all over the place. And then I read the article. It was a short study on one kind of problem done with college kids. Meaning that we have virtually no idea, based on this study, whether this is true for elementary school kids who learn this way for 6 years.

This is no critique of the authors -- there's value to their study -- but, for the life of me, I can't figure out why everybody seems to think this means that we should stop using manipulatives forevermore. This sheds little, if any, light on that issue.

I'm disappointed in the NY Times, and I'm disappointed in everybody else who parroted these results without explaining what they really meant.

(disclaimer: finals are making me a bit grumpier than usual)

Update: Thanks, Dr. Dorn

What's Wrong with Ed Policy

Most of the time I can't imagine focusing on anything but education policy for the rest of my life. But, at certain times, I wonder if it's worth it. Petty arguments and ideological venom seem to be poisoning the field, and I'm not sure if I can take it. Here's the latest example of what's wrong with education policy. We'll follow it step by step.

Step 1. In the wake of the latest evidence on vouchers in Milwaukee, Sol Stern (a noted supporter of vouchers for many years) writes that "markets in education may not be a panacea," that there has been no "Milwaukee Miracle," and that we should consider other options.

Step 2. Jason Riley writes a piece in the Wall St. Journal in defense of vouchers and choice, arguing that "seeking panaceas and miracles is setting the bar for success unreasonably high."

Step 3. Letters to the editor pour in (here and here), including some from teachers who argue that the biggest problem is that their kids come to school far behind.

Step 4. George Clowes writes in to call these letters condescending and suggests that we should blame teachers instead of students or families ("Instead of blaming social conditions for student failure, let's just ask teachers to do their job: Educate students").

Step 5. Greg Forster revels in this "smackdown" on Jay Greene's blog and compares Clowes to Mr. T.

Notice that the level of discourse (not to mention civility) devolves with each step. By the end we're left with one side rooting against the other. How is this productive? It's not.

If the field of education policy becomes nothing but half the participants fighting to defeat the other half of the participants, then I have no interest in participating. Being childish and immature can be fun, but it has no place in education policy. If it's not possible to make an honest attempt to find solutions to problems without a large group making knee-jerk judgments regardless of what you find, then count me out.

I see no utility in rooting for one solution or blaming one group instead of another. In other words, can't we focus on attacking the problem instead of each other?

Is America's Entire Education System Failing?

Bashing our education system has been a popular sport since well before I was born. Indeed, if I didn't think there were serious problems I wouldn't be studying education policy. Nonetheless, I sometimes wonder if people go too far. After reading Bob Herbert's op-ed in the Times today I decided I should take a little time to wonder if our education system is really that bad.

Though standardized tests have their shortcomings, they're usually the best way to compare the performance of different countries' school systems. Pasted above is a chart I made last semester of the reading ("Literacy") scores from just the G-8 countries (basically the 8 wealthiest countries in the world) so that we're comparing apples to apples. The scores are broken down by percentile for each country and then ranked.

The U.S. ranks 5th out of the 8 countries for median score, substantiating the feeling that we're falling behind and that reform is needed. But, you can see a large difference in performance for the top scorers vs. the bottom scorers in our country. Only two of the countries have top scorers that do better than the top scorers (90th and 95th percentiles) in the United States. Meanwhile, the bottom scorers (5th and 10th percentiles) in the U.S. only outperform two of the other seven countries.

So maybe the problem isn't really the entire system. Maybe our system does fine for some students while it fails others. You hear a lot in the news about impoverished schools in the inner-city or rural America, but you also hear a lot about the proliferation of AP tests, and the increasing competitiveness of admissions to the top colleges.

So maybe it's not our system that's broken but, rather, part of our system. We have both a lot of success stories and a lot of failures. Maybe we should be trying to fix the system that exists for the least fortunate students rather than berating the entire system.

A Solution to the Shortage of Science Students?

Ever since the Soviets launched Sputnik we've been worrying about how well our schools are doing at teaching science. More recently, people have worried that not enough college students are choosing science majors. In the long-run we worry that we won't have enough people to drive a high-tech economy. Well, last week's edition of The Economist has a solution: let in more foreigners with science degrees.

The Economist, of course, will almost always think something is a good idea when it involves fewer restrictions on the market -- but they include some interesting statistics in defense of their position.

-"Some 70 or so of the 300 Americans who have won Nobel prizes since 1901 were immigrants"

-"Some 40% of American PhDs in science and engineering go to immigrants"

They write that there are 85,000 H1B visas (for highly educated immigrants) allotted each year and that hundreds of thousands of people each year are unsuccessful in obtaining one when they enter their names in the visa lottery. They also claim that both Bill Gates and respected economists agree that "every foreigner who is given an H1B visa creates jobs for five regular Americans."

All of this adds up to a shortcut solution for fixing our shortage of science students -- let in more science grads from abroad. An imperfect solution to be sure, but it's probably better than simply letting our economy fall into oblivion over the next few decades. And, heck, it has to be cheaper to grant visas than provide two decades of education.

Could Judging Teachers by Test Scores do Harm?

I'm sick of the discussion over the NY tenure law but, for some reason, I can't resist chipping in another two cents worth of thoughts. I've already written twice (here and here) that I can't see much utility to using the test scores, in their current form, to decide on teacher tenure in NY -- so I won't harp on that again.

NYCEducator argues that the scores are not needed to tell which teachers should be let go, while Kevin Carey fails to see the harm in including the scores in tenure decisions.

If it turns out that the scores provide even a snippet of useful data (which is debatable) then I'd agree with Carey that it also wouldn't do much harm to include these scores in decision-making about teacher ratings (and, hence, firings). But lets turn to a different side of this that I haven't seen anybody address. I've already argued that not using them does no harm, and I think Carey's arguing that using them would also do no harm. We can argue all day over whether the test scores will provide any useful knowledge, but let's think about whether using them could do any harm. Since I'm in the middle of finals week, I'll choose the most intellectually flabby way possible to examine this question -- a game of what if.

Imagine that schools in NY started using the test data (which is questionable in accuracy and only covers 3rd-8th grade Math and English):

What if this were used as the primary way to evaluate teachers instead of a supplemental one?

What if Math and English teachers were scrutinized more closely than teachers in all other subjects?

What if upper-elementary and middle school teachers were scrutinized more closely than other teachers?

What if a good teacher had artificially low scores?

What if a bad teacher had artificially high scores?

What if a teacher had high test scores but abused students?

What if we let gym teachers do whatever they want while we closely monitor other teachers?

What if that gym teachers (not to pick on gym teachers), while we're focusing on other teachers, takes advantage of a student?

What if we forget to monitor serious breaches of ethics because we're so focused on test scores?

What if a teacher cheats on the tests but cleverly enough so as to avoid detection?

What would happen if the hadn't passed is really just speculation, but it's entirely possible that using these test scores to evaluate teachers could have unintended consequences -- and severely hurt students as result.

More on School Expenditures Over Time

This is a follow-up to my last post where I posted a graph that surprised me. As a percentage of GDP, per-pupil expenditures have remained flat over time -- other than during WWII a little under a dollar per billion dollars of GDP per student. The first comment (by the author of the post that prompted my post) asked why we should expect to spending on education to rise over time compared to GDP. I gave a terse response in my follow-up comment, but I'd like to take more time to flesh that out.

To oversimplify economic theory, many goods can be counted as either necessities or luxuries and people (and countries) are expected to increase their spending on luxury items as income increases while simultaneously decreasing their spending on necessities relative to their income. So, for example, somebody making a million dollars per year is expected to spend a smaller portion of their budget on food and shelter than is somebody making twenty thousand dollars per year. At the same time, the wealthier person is expected to spend a greater share of their budget on wine or boats or vacation homes. They're able to do this because they have more "discretionary income" -- money beyond what's needed to pay for the bare necessities of life.

Countries work in a similar fashion. There are certain things that even the poorest country must spend money on (e.g. a basic system of defense or a basic network of roads) and some things that wealthy countries can afford to devote a larger share of income towards as their discretionary income increases (e.g. science research or museums).

So, whether you believe that education spending should increase, decrease, or remain the same relative to GDP depends on how you view education spending. If you think that it's a bare necessity, then you would expect it to decrease over time as we gain more money. If you view it as more of a luxury good, then you'd expect spending to increase over time as discretionary income increases. If you view it as somewhere in between, then you might expect spending to stay relatively flat over time.

I said at the beginning of the post that I was surprised when I saw that it was flat. I was surprised because I was under the distinct impression that countries tended to spend a larger share of GDP on education as they become wealthier. I guess further investigation is needed to find out whether I was wrong or if the US is an outlier.

Update: Actually, when I corrected the graph, it appears as though spending per pupil compared to GDP per capita has been falling rather than flat

Has Education Spending Really Skyrocketed?

I just read a post at D-Ed Reckoning about the astronomical increases in per-pupil education spending over the past century. Though expenditures rose in terms of real dollars, I wondered how they compared to the GDP of the U.S. So I found data on the GDP of the U.S. since 1940 (here) and combined the two. The results are above. As a percentage of GDP, per-pupil expenditures (other than a drop during WWII) have stayed almost exactly flat over the past 65 years. The chart is a bit small, but if you click on it then you can see a larger version.

The question remains over what the most relevant comparison is over time. The fact that per-pupil spending relative to GDP has remained the same over time indicates that we're putting forth about the same financial effort relative to our capacity to spend, but if our capacity to spend has increased then we might still expect larger gains in achievement over that time. On the other hand, increased spending capacity should also increase the country's discretionary income -- apparently we've chosen to spend that on items other than education.

Update: More on this in a follow-up post

Later update: Corrected version of the graph (oops) here

How Should We Get Rid of Bad Teachers?

My my last post had far too much vitriol for my liking, and I feel the need to cleanse my system with a less argumentative and more thoughtful post.

One problem in schools is that there are bad teachers who continue to harm students year after year. What do we do about these teachers? Well, fire them of course -- that's the easy answer.

Unfortunately, the solution is far from simple. Furthermore, the extent of the problem is maybe just as complex.

Let's start with the scope of the problem. How many bad teachers are there? What qualifies a teacher as "bad?" Are there more bad teachers than there are bad lawyers or accountants? Are teachers bad because they have no talent, put forth no effort, or because they attempt to harm students? I'm not sure that we have answers to any of these questions.

Assuming that there are a lot of bad teachers, the next question is why. Why are there so many bad teachers? Are the unions protecting them? Are principals failing to evaluate thoroughly? Do teachers tend to burn out? Do bad people try to be put in a position of authority? Are there no good teachers to replace them? There might be merit to theories that accompany all of these questions.

So, given that we (or at least I) don't really know how bad the problem is, how should we deal with the problem? Create better evaluation systems? Outlaw unions? Put more pressure on principals to evaluate teachers? Put more pressure on bad teachers? Give bad teachers more training? Fire them all?

I'm of the distinct opinion that there's no easy solution. That, of course, doesn't mean that we shouldn't try or should indefinitely delay action (any teacher that abuses a student, for example, should spend another second in the classroom). But I'll bet we'd be closer to an answer if people tried to answer all of these questions rather than just pushing a pet solution. In other words, it's likely that the solution is no simpler than the problem.

More questions inspired by comments: Would "bad" teachers be more effective in a different environment? When should "bad" teachers be filtered out?

More on the NY Tenure Law

I wrote before on the tenure law that was passed in NY and had no intention of writing about it again, in part because I think the end result is essentially inconsequential. But the more I read about it the more annoyed I become with critics of the law. Not because it's not a bad law -- it is -- but because the effects of the law are constantly overstated.

The latest offender is Kevin Carey over at The Quick and the Ed. I hesitate to pick on him b/c I've read more ridiculous assertions than his, but his just happened to be the straw that broke the camel's back. Before I begin let me say that he's written a lot of other things that I thought were very good.

Though not the main focus of his post, Carey implies at the end that using standardized tests to make tenure decisions would eliminate a lot of bad teachers in NY. That theory, of course, makes sense on paper -- but it plays out differently in reality.

The tests are ill-suited for use as a measure of the amount that students learned in one year for three major reasons: 1.) They're given well before the end of the year, 2.) They're not designed to be value-added tests -- they're designed to determine proficiency in each separate grade, and 3.) The majority of teachers do not teach tested subjects.

Carey writes that academics are too cautious about endorsing programs, and he may have a point, but this one case where the caution is merited. It's not the case that the information gathered from these tests would be only a little helpful rather than a lot, it's the case that the information gathered from the tests would be so inaccurate as to be useless.

He writes that there are a number of bad teachers in NYC (and references this as proof). Of course there are a number of bad teachers -- anybody who disputes this is dead wrong -- but would the information gained from these tests really solve the problem?

First of all, a number of the bad teachers referenced in that blog post do not teach a tested subject (3rd-8th grade Math or English) and would, therefore, be completely unaffected by the use of the test scores. This means that a 4th grade English teacher, say, with slightly sub-par scores might face some heat while the Spanish teacher who takes his shirt off in class will not. Secondly, all of the offenses mentioned in the post are observable by peers and supervisors -- a test will not inform a principal that a teacher had an affair with a student. Thirdly, because the test results would be somewhat random it very likely that a good teacher could get bad results and a bad teacher good results -- completely defeating the purpose of using the tests in the first place. In short, using test results would likely hurt at least as many people as it hurt.

It is clear, then, that the test results should not be a deciding factor when evaluating teachers. The argument that NY should hold off for a couple years until they develop a better value-added testing system is correct.

As I previously wrote, it is a heavy-handed law and was passed in a devious manner -- which are both worthy of negative press attention -- but it is quite clear that the rule regarding the use of test results will not, in and of itself, do substantial harm over the next two years. I find any argument that it will to be disingenuous.

Let me end with two clarifications. 1.) I do not object to change. Not in the least. Bad teachers should be removed from schools. I will never, ever, argue this point. The tenure law, however, does not prevent bad teachers from being removed from schools. 2.) I bear no personal ill-will toward Kevin Carey. If I've been unfair to him then I assume either he or somebody else will let me know.

Update: I've been informed that I misspelled Kevin Carey's name 4 out of 5 times in the previous edition of this post. My apologies -- it's been corrected.

Random Thought on Generalizing

Before entering academia I was always told not to make generalizations about people. In academia, however, the main goal is to find results that you can generalize to the largest number of people. There is, of course, a reason for this -- but I still find it odd when I think about it.