Interesting Factoid of the Day

Skimming through the latest Bracey Report, I calculated the following statistic from the table on page 3:

If we look at achievement scores on the 2001 Progress in International Reading Literacy Study (PIRLS), Sweden ranks first with a median score of 561, while the U.S. is a little further back with an average score of 540.  If we look only at the at the students who attend schools where less than 50% of the students are in poverty (I assume measured by the percent eligible for free/reduced-price lunch, but I'm not sure), then we have a sample comprising 63.6% of the American population.  And their average score is 564. 

In other words, the average American student attending a school that doesn't rank among the poorest third in the country out-achieved the average student in every other country that took part in the assessment.

I don't know whether other, more recent, international assessments would yield similar results, but we do know that our top students out-achieved the top students in most other G-8 countries, while the opposite was true for our bottom students, on the PISA 2000 Literacy test.  Two pieces of data don't warrant a strong conclusion, but both indicate that our top students are doing pretty well while our bottom students lag far behind -- which would indicate that we should spend most of our time trying to pull up those at the bottom.

Are Confident Students a Bad Thing?

Mark Bauerlein has a provocative post over at the new EdNext blog. In it he refers to both a report from three years ago and a study that was just released to support his position that "higher confidence does not go with better math scores" and that "the same discorrelation between confidence and performance may hold in reading." Before you decide that berating your kid is a good idea, I should explain to you why this isn't really correct.

He mentions that in the first study the authors find that countries with more confident students also perform more poorly on the math part of the Trends in International Mathematics and Science Study (TIMSS). But he neglects to mention one tiny detail: that this correlation only holds up across countries, not within them. In other words, students in Japan report, on average, being less confident than students in the United States, but also score higher. On the other hand, students in the U.S. that report being confident in their mathematical abilities tend to score higher than students in the U.S. who report not being confident in their mathematical abilities. As the authors write, "In the TIMSS data, when one looks at the math scores of students within each country, those who express confidence in their own math abilities do indeed score higher than those lacking in confidence. That is true for 40 of the 46 countries with eighth grade test results" (p. 15).

This makes sense if you think about it. Let's say you live in a country where people perform very poorly in math. You know how to add double digit numbers, so you're convinced you're a genius. Meanwhile, in the next country over the kids are very good at math and your cousin thinks he's an idiot because he can only do derivatives but can't find the integral of a number. Then you both take the same test on multiplication and division; you get all the answers wrong and he gets them all right. But you were more confident than him. So confidence must be a bad thing. The problem with this argument is that you didn't base your confidence on your cousin's math ability, you based it on your performance relative to your peers. So it's really no surprise when the kid who's lacking in confidence that sits next to you in class also scores lower than you on the test. Comparing confidence levels across countries is interesting, but of somewhat limited utility because they're all relative to what's happening in their own country. If you want to determine whether or not confidence is correlated negatively or positively with achievement, you should really take a country-centered calculation (maybe even school or class-centered, one could argue), in other words, your confidence minus the average confidence level in your country.

Similarly, based on his summary of the second article one might think that the authors found that higher confidence led to lower scores. But, if one reads the article they would notice the following: "Reading [Self-Concept] was strongly correlated with reading performance in most countries, supporting the importance of this self-belief across cultures." (p. 381). In other words, more confident students scored higher. What the authors focused on is looking at the scores of students relative to their level of confidence in their ability and how under confidence and overconfidence affected scores in countries with different cultural characteristics (you probably won't be able to download the article if you're not on a campus with a subscription, but if you're interested you can e-mail me and I'll send you a copy). In the end, it seems that the underconfident tend to outperform the overconfident. It's unclear whether that's because one can only be classified as overconfident if they don't score super-high (and vice-versa), or because overconfidence is bad. But it's probably a mixture of both. I find the latter at least plausible -- if you think you're great at math, for example, you might be less likely to feel compelled to study for the test tomorrow,.

At any rate, Bauerlein's not wrong when he suggests that overconfidence may be a bad thing. But he's wrong to simply report that multiple studies have found that higher confidence yields worse results. In other words, while more confident students usually score higher on math, overconfidence can be dangerous . . . both in math class and in blogging.

How Much Achievement is Too Much?

This article on South Koreans studying abroad appears in the Sunday NY Times and I'd have to say it's a must-read. Not because it's perfect but b/c it touches on so many fascinating themes and ideas.

In short, a growing number of South Korean families are splitting up so that their children can attend school in an English-speaking country (apparently the U.S. once dominated this market, but now people with less money are moving to Canada, New Zealand, and Australia as well). Fathers stay behind in S. Korea while mothers live with the children -- usually starting in elementary school. The latest tally is that over 40,000 children are in such an arrangement.

The article claims that three main factors drive this trend: (1) Parents, especially mothers, are deeply concerned about their children's education; (2) Parents want their kids to learn English; (3) Parents want their kids out of South Korea's pressure-filled schools.

Here are a number of things that I find absolutely fascinating about this article:

1.) South Korea is a superstar in international testing -- if they're not number one on a particular test you can bet they're in the top 5. And yet, parents are so desperate to get their kids out of the educational system that they'll move thousands of miles.

2.) Parents are so concerned with their children's education that they're willing to live thousands of miles apart from their spouse for a decade or two -- with most spouses only seeing each other maybe a couple times per year.

3.) The response of the South Korean government to this phenomenon is not to try and reduce pressure on children but, rather, to hire 10,000 more English teachers (which I think either means that the govt. is out of touch or that the author of the article is mistaken about the motivation of these movers).

In my opinion, I think this is the most interesting paragraph of the article:

South Korean students routinely score at the top in international academic tests. But unhappiness over education’s financial and psychological costs is so widespread that it is often cited as a reason for the country’s low birthrate, which, at 1.26 in 2007, was one of the world’s lowest.

Now, before I get carried away, there's no way that this one article has done all of the legwork necessary to definitively define trends and establish causality. That said, even if we take this is a bit of information rather than gospel it still raises a lot of interesting questions.

The largest is probably the one raised in the above paragraph. South Korea supposedly has a model education system. And yet, people are going to great lengths to avoid it. Which begs the question: what is a perfect education system? Is it necessarily the one with the highest test scores? I think most of us agree that higher test scores are usually better, but there has to be a point at which the opportunity cost of higher test scores is too high. In other words, everybody must have a point at which the additional effort that will boost test scores isn't worth it. Maybe you'd rather your kid played soccer than attended cram classes. Maybe you'd rather your kid attended your neighborhood school rather than the better one across town. I think you get the idea.

I can't imagine living thousands of miles away from my spouse for a decade or more so that my child(ren) could attend a certain school, and I could see inferring from this article that some Korean parents are hyper-concerned about education -- that they've, in essence, gone crazy. But here's the thing: the parents who seem to most merit judgment that they've gone off the deep end are the ones who are apparently claiming that South Korea's system is too pressured.

If the article is correct in describing the level of desperation of parents to avoid the Korean education system (I'm not sure which is more extreme -- moving far away or simply not having kids), then we should all be careful what we wish for when demanding more rigorous schools and higher achievement in our country.

Is America's Entire Education System Failing?

Bashing our education system has been a popular sport since well before I was born. Indeed, if I didn't think there were serious problems I wouldn't be studying education policy. Nonetheless, I sometimes wonder if people go too far. After reading Bob Herbert's op-ed in the Times today I decided I should take a little time to wonder if our education system is really that bad.

Though standardized tests have their shortcomings, they're usually the best way to compare the performance of different countries' school systems. Pasted above is a chart I made last semester of the reading ("Literacy") scores from just the G-8 countries (basically the 8 wealthiest countries in the world) so that we're comparing apples to apples. The scores are broken down by percentile for each country and then ranked.

The U.S. ranks 5th out of the 8 countries for median score, substantiating the feeling that we're falling behind and that reform is needed. But, you can see a large difference in performance for the top scorers vs. the bottom scorers in our country. Only two of the countries have top scorers that do better than the top scorers (90th and 95th percentiles) in the United States. Meanwhile, the bottom scorers (5th and 10th percentiles) in the U.S. only outperform two of the other seven countries.

So maybe the problem isn't really the entire system. Maybe our system does fine for some students while it fails others. You hear a lot in the news about impoverished schools in the inner-city or rural America, but you also hear a lot about the proliferation of AP tests, and the increasing competitiveness of admissions to the top colleges.

So maybe it's not our system that's broken but, rather, part of our system. We have both a lot of success stories and a lot of failures. Maybe we should be trying to fix the system that exists for the least fortunate students rather than berating the entire system.

More on Inequality and National Achievement

Douglas Willms gave the keynote address last night here at CIES and spoke about increasing both equity and achievement ("raising and leveling the learning bar"). For every country that took part in two international assessments (PISA and PIRLS) he constructed a "learning bar" that showed the relationship between socio-economic status (SES) and achievement. SES was a significant predictor of achievement in every single country, but the slope of the line is different across nations.

For example, Canadians, on average, outscored Americans. But the wealthiest Americans outscored the wealthiest Canadians by a bit (meaning the poorest Canadians outscored the poorest Americans by a lot). This means that the American "learning bar" has a steeper slope -- that SES matters more in the U.S. than it does in Canada. The report of most of the results he discussed can be found here.

He makes a distinction between "raising" this learning bar -- meaning that everybody scores higher -- and "leveling" the learning bar -- meaning that low-SES students improve more than high-SES students, the line becomes flatter, and there is less of a gap between rich and poor. Here's what I found most interesting: his findings dovetail nicely with mine -- countries that scored higher overall also had lower gaps between rich and poor. Additionally, higher scoring countries have distributions that are less negatively skewed -- meaning that there are fewer very low scoring students. In other words, there doesn't seem to be a tradeoff between "raising" and "leveling" the "learning bar" -- or between aiming for high achievement and high levels of equality.

National Achievement and Inequality

I'm in New York City right now attending the meetings of the Comparative and International Education Society (CIES). I've been mostly hiding away in my room desperately trying to prepare my presentation for this afternoon (Tuesday). As such, I do not yet have much of interest to report other than about my paper.

My paper did not live up to my expectations for it, but I think my presentation made sense and I found one interesting thing. The basic motivation for the paper was to find out why some countries have a small spread in achievement and some countries have a large spread in achievement. In the end, I couldn't really find much that looked like compelling predictors for whether achievement in countries would be more or less spread out.

But I did find one thing that surprised me. The last thing that I checked on, mostly out of curiosity since it was really only tangentially related to my topic, was whether higher performing countries had more or less equality. I had a couple different measures of variation in achievement, and I compared them to the median score on the TIMSS (an international assessment involving about 50 countries the last time) and found a really strong correlation between equality and achievement (about .8, p<.001 for you stats nerds), meaning that, within the TIMSS countries, that higher achieving countries were distinctly more equal than lower achieving countries. When I compared performance on TIMSS to spread on PISA (another international assessment) the relationship still held and was moderately strong (about .4, p<.001).

The strength of the relationship was of a level that one just doesn't find while doing research, so I was sure I was doing something wrong, but nobody has given me reason to think that this isn't true and I can't think of any.

I don't know what this would look like for different assessments, different years, different subjects, and different ways of measuring variance (or spread, or inequality, or whatever you want to call it), but it's a result that could potentially be meaningful. It's at least as possible, of course, that the result is either meaningless or won't hold up with other data, but I think it's worth further investigation.

As of this moment, I'm seeing that more equality=higher achievement and wondering whether that means what one might assume it means.

Teachers in Martinique

On Friday I attended a conference for graduate students at Harvard. The conference was run quite well -- if any grad students are reading I'd encourage you to go and present next year -- and the amenities were quite remarkable given the unbeatable $0 registration fee. I saw a number of interesting presentations, but have been quite busy (I'm now in NYC for another conference) so I'm just going to mention one that I found even more interesting than others.

During the last session of the day Nick Gozik, a student from NYU, spoke about his research (I'm quite sure it's his dissertation) that he conducted in Martinique. Martinique is an island in the Caribbean (somewhat near Venezuela) that was a French colony and their education system is still under the control of France. He spent most of a year on the island and says he spoke with half of the high-school teachers in the system, so you can imagine how much information he collected.

Given that this will take him hundreds of pages to explain, I'm just going to briefly summarize two points I found interesting.

1.) He said that teachers repeatedly emphasized that they followed the curriculum because they were professionals (history teachers identified themselves as "historians," etc.) rather than due to any external forces. The current trend here right now is to essentially punish those who don't follow the curriculum by "holding them accountable." Teachers are different in the two different countries, but I found this contrast fascinating.

2.) Even more interesting is that principals were not responsible for evaluating teachers in their schools because they were not experts in most of the fields (e.g. when a math teacher becomes a principal they do not feel prepared to evaluate a history teacher). Instead, inspectors came to the schools occasionally and both trained and evaluated teachers. I do wonder if this model might actually be better than our current model in the U.S. where principals are essentially expected to both know how and actually do everything. We read an article in a class last week about educational administrators lacking deep knowledge of many subject areas and ways to address this. Our class was divided over whether attempts should be made to intensely train principals in all subjects or if instructional leadership should be ceded to an expert in each field (we were thinking perhaps a revision of the dept. head role, but an outside inspector is another interesting idea).