Thursday, December 1, 2011

Moving on from the "Search for Universals"

As I've delved deeper into the field of education research, I've grown increasingly frustrated by two large problems I see in that research.  While listening to Malcom Gladwell's TED talk on consumer choice, something he said helped crystallize the second issue (I'll explore the first in another post).

In the talk, Gladwell discusses the shift of food science (starting with Prego pasta sauce) from trying to create the "one best" product for all to creating multiple products that fit specific groups.  He compares that trend to cancer research, arguing that "the great revolution in science of the last 10-15 years" is "the movement from the search for universals to the understanding of variability".

If he's right, then it seems ed policy researchers missed a memo somewhere (to be fair, it's not just ed policy -- I see the same problem in lots of policy literature).

The vast majority of the policy research I've read, seen presented, or heard discussed focuses exclusively, or at least mainly, on the average effect of a particular variable, intervention, or policy.  Over the years, we've developed increasingly sophisticated methods to more accurately estimate these average effects.  But I rarely hear people discuss the differential effects of the variable, intervention, or policy of interest.

In other words, we keep trying to figure out if different policies "work," but we define "work," as making a statistically significant difference for the average student, teacher, principal, school, district, or state.  If we instead asked for whom a given policy works, we'd likely find that it works very well for some while harming others.

In the talk, Gladwell mentions a food scientist hired to create the "one best Pepsi" who conducts taste tests of the product with varying amounts of sugar.  To his surprise, no one, clear winner emerges.  Some people prefer only a little sugar, some like a medium amount, and some like a lot.  He then has an epiphany and realizes that there's no such thing as the "one best Pepsi" (which eventually leads to the creation of a wide variety of Prego sauces that target different audiences).

I'd argue that education policy is almost exactly the same.  For example, imagine a new math curriculum.  How do we decide if it works?  The gold standard of research would dictate that we randomly assign, say, 100 classrooms to use the new curriculum and 100 to use the old one.  We'd then compare the average scores at the beginning and end of the year of the treatment and control groups.  If kids in the treatment group score significantly higher, on average, than the control group then that curriculum earns a gold star.

We spend increasing amounts of time trying to figure out which math curricula will yield the largest achievement gains across the students who use them.  But it seems exceedingly unlikely that "one best math curriculum" truly exists.  Some states, districts, schools, teachers, and/or students will do best with curriculum A, some with curriculum B, and some with curriculum C.

Wouldn't our time be better spent figuring out for which students curriculum A would be best and for which students curriculum B would be best (and why)?  And we could say the same about all of the largest issues in education policy -- teacher training, teacher pay, charter schools, and so on.  In later posts, I'll explore how our research and policy would differ if we aimed to understand and account for variability rather than simply finding the one best policy.

I've long thought it odd that we spend most of our lives being told not to stereotype and make generalizations while the most educated people in the country strive to make the largest generalization possible in their research.  Most (though not all) researchers focus incessantly on "generalizability" (one of those words we use in the field but the spellchecker won't recognize): if we can generalize your findings to 10 million students across the country, you're likely to publish the paper in a better journal than if we can only generalize your results to students in one school.

As it turns out, I was right to think that odd.  Greater generalizability matters in many ways.  But, at some point, we start missing the point -- and for exactly the same reasons teachers and parents across the country are telling our kids not to generalize.  Everybody is different.

Every student is different.  Every teacher is different.  Every principal is different.  Every school is different.  Every district is different.  Every state is different.  The best policy, on average, will help one and hurt another.  Until we understand this variability, we're doomed to fail.


Anonymous said...

Your post makes a lot of sense of me and I really hope it gets the attention of anyone who matters. It is one of the reasons- what I call the "silver bullet" illusion - that teachers are going crazy in classrooms, students are not learning as well as they could, and parents are abandoning public schools. When policy folks propose the next best thing that will close the achievement gap and improve education, they fail to account for the effect of unintended consequences. I see it happening in countless ways in MNPS, and I imagine it is the same in every other school district in the country. It is as if we imagine there is a magic wand or silver bullet that will solve all education woes (and if we can also somehow make it cheaper to educate, then all the better), but every new fad that is rolled out because "research shows" that it works, only ends up causing more distress and leaves in it's wake much suffering and gnashing of teeth. The district and state folks gnash their teeth because standardized test scores do not improve enough, teachers suffer because they are set up to fail, and parents with ways and means yank their kids from the insanity . . . or are in distress because they can't spare their child from the "next best thing."

Anonymous said...

On the one hand, I completely agree with you; Certain policies work well for some students and not so well for others, and the constant search for generalizable solutions to big problems makes little sense if your goal is to solve the problems.

However you left one thing out: You can't sell variability. A politician can't get elected on it. You can't become a powerful member of your community by implementing a program that helps 10%, even if that is the 10% that needed it the most.

In practice most research findings have such small effect sizes that they are essentially meaningless anyway. If your goal is to make a change you need to be operating either at the base (I took a job working in administration at a huge university where I can influence how we treat our students) or in politics. Policymakers don't read the research that is out there and even when they do they won't act on it unless it can be explained to a 5 year old.

That sounds really cynical, I know. But I don't think you are the first to realize that there is no single solution to the problems in XXX. Where you could make your mark is figuring out how to get tenure while writing about variability, individual solutions and what works out to be mass customization.

Corey Bunje Bower said...

Anonymous #2: I understand a lot of the practical reasons why people search for the one best solution rather than attempting to understand variability, though some of these can be overcome (one could argue that charters, for example, can be seen as a way to match every kid with their best school instead of creating the one best school for every kid).

And you're right that most studies find incredibly small effect sizes. But I think that's another argument that we should instead try to understand variability. Maybe that math curriculum only raises achievement .1 standard deviations and everybody yawns. But maybe it raises achievement of kids with a particular learning disability or kids who aren't native English speakers by .5 SD. *Now* it's a big deal.

Will policymakers take the time to understand that? A lot of the time they won't. But it's not like they read the current research anyway.

Anonymous said...

Thank you!! The factory model for education has not worked for are not widgets; each brings his/her own set of needs to the school. This leads to the need for the schools to be built from the ground up to best serve the needs of the students in each school vs the one size fits all. I hope you keep pressing this issue...School systems today, especially government schools, are too top heavy in administrative personnel (they need to flatten out the organizations and put the savings into adding teachers); the systems are built for what works for the adults in the central office vs what is needed in the school and more importantly in the classroom for the kids and the teachers. We need to change this. Tennessee released it's report card yesterday. The numbers for Nashville are very, very sad and have been for a very, very long time. We have done a major disservice to decades of children. When will it stop? Thanks again!!


Roger Sweeny said...

I think this is a very profound point. Students are different and different things work for different students.

However ... how do you "operationalize" that? You have to be able to say, "this student is type X and will do best using system Xprime; that student is type Y and will do best using system Yprime." You need to be able to distinguish which student is which and you have to have multiple ways of teaching available. That's not easy.

Moreover, you are going to run into the tracking problem. Many people in the business are morally/politically opposed to treating different people differently. And if it turns out that group X and group Y are noticeably different ethnically, well ...

Corey Bunje Bower said...


It's not just students that are different. Different policies work better or worse for different parents/families, teachers, schools, districts, states, etc. That said, you're right that people will oppose differentiation for a lot of policies. Sometimes that can be overcome and sometimes it can't.

Let me give you a couple research findings with variable results that I think could be embraced:

Phonics is beloved by some and hated by others, but a number of studies indicate that it works really for most kids up until about 3rd or 4th grade, at which point other strategies work better. We could strive to find greater variation within those results, but that's a good place to start: if I were running an elementary school, I might include phonics in the curriculum only in grades K-3.

ESL is another controversial topic. One study found that taking ESL in high school seemed to help students who had just arrived in the country, but hurt students who'd been here for a while. We could certainly search for a lot more variation, but if I were running a high school I might strive to enroll new arrivals in ESL classes while trying to get longer-term residents more integrated with the rest of the student body.

It seems likely that these two variation-minded approaches would be better than simply offering or not offering phonics or ESL.

I think those examples also demonstrate the largest policy flaw with "the search for universals". While I was mainly speaking about researchers in the post, one could certainly accuse policymakers of the same flaw. If policymakers were to focus more on figuring out for whom which policy worked best than on finding the one, best policy they might be more likely to incorporate local autonomy into many policies and become less frustrated by their inability to get every local actor on board with their exact vision.

Neil Brown said...

There is already movement in public education towards individualized instruction and "authentic" or alternate assessments, which, if properly implemented, would address your concerns with variability in students and finding effective ways to "connect" with each. Your suggestion about finding the appropriate curriclum for individual kids is not feasible, wrought with its owns problems, and quite frankly a waste of time. A more appropriate goal would be to design curriculum that empowers teachers to design courses that reflect their own passions and knowledge,and then design classroom where students are empowered to pursue content that then reflects their own passions, questions, and curiosities. We have become caught up in broadening the content knowledge requirements of students when quite frankly their is only a small amount of core knowledge that all kids should learn prior to graduation. We should be less concerned with what kids learn than how they learn, making sure that true learning is actually taking place in class. True learning is a time consuming process that most teachers are unable to properly assess within the constraints of today's mandates.