Monday, September 15, 2008

Oops, Forgot a Variable

The Fordham blog alerted me to the fact that George Will again mentioned David Whitman's new book in his latest column (which, if you haven't noticed, intensely interests me).

Will's column focuses on one of the schools profiled in the book -- Cristo Rey Jesuit High School -- and its positive outcomes. As far as I can tell, Cristo Rey is doing an admirable job educating low-income students and deserves a lot of praise. So I'm glad Will took the time to highlight their educational model and achievements.

But near the end Will makes a statement that is somewhere between overly simplistic and absurd. I'll post the entire paragraph below:

CRJHS can have its work program, its entirely college preparatory courses ("the old, dead white man's curriculum," says an English teacher cheerfully), its zero tolerance of disorder (from gang symbols down to chewing gum), its enforcement of decorum (couples dancing suggestively are told to "leave some space there for the Holy Spirit") and its requirement that every family pay something, if only as little as $25 a month. It can have all this because it is not shackled by bureaucracy or unions, as public schools are.

From a research standpoint, here's the problem with that statement: his model is misspecified -- it suffers from omitted variable bias. And it's particularly egregious in this case b/c he leaves out a potentially confounding variable -- selection. If he published a paper with such findings, they would be dubbed spurious.

He attributes all of the successes of the school to the fact that they don't have to deal with as much red tape as other schools (particularly public schools). But he fails to mention that the larger reason the school can have such policies is because they can select their students -- both which ones are admitted and which ones are allowed to remain enrolled. And, perhaps even more important, the students who enroll are from families who chose to apply and chose to pay at least a small fee to enroll their children -- and then chose to remain enrolled once they knew what the school was like.

In Whitman's book, he writes that 43% of students who applied in 2006 were admitted (p. 142), and that the retention rate (percentage of students who remained enrolled) from freshman to senior year was 62% (p. 144). In other words, a significant number of kids (about 73% of those who apply) either choose not to or are not allowed to complete 4 years at the school.

Part of the reason why public schools can't have such strict discipline codes is due to bureacracy, but the larger reason is because they can't select their students -- meaning that they have to accomodate everybody, and they must have options for students who don't like the rules other than expulsion.

3 comments:

Roger Sweeny said...

From a research standpoint, here's the problem with that statement: his model is misspecified -- it suffers from omitted variable bias. And it's particularly egregious in this case b/c he leaves out a potentially confounding variable -- selection. If he published a paper with such findings, they would be dubbed spurious.

Oh my God, do you have a rose-colored view of academic publishing.

Take, for example, just about every article that says going to college will increase your income.

People who go to college and people who don't are very different. The ones who go to college are, on average, smarter, but perhaps more important, they are also more organized, more goal-oriented, more able to defer present gratification for future rewards, and a host of other things.

There is a terrible amount of omitted variable bias in such studies. The "go to college" group is selected just like the "go to CRJHS" group.

The go to college studies are just about useless in distinguishing between "going to college makes people successful" and "the people who go to college are the ones who were going to succeed anyway."

But most of us don't even notice the omission in the college studies. The result seems right, and it's something we want to believe.

When you start looking skeptically, you find that almost all non-experimental studies suffer from omitted variables.

RDT said...

This issue isn't just "leaving out a variable." It's a major bit of blindness in the school reform debate.

Many of the school pointed out as successful are effectively choosing their students, even if its only choosing students whose families are open to working with the school's "no-nonsense" philosophy.

It may be that there success is purely a matter of selection. But -- and this is where things get philosophically difficult -- it may also be that not having to deal with recalcitrant students raises the achievement of the rest of the students at the school.

But, despite the current focus on individual schools, the mandate of public education is to educate all students. If we're worried about the high school drop out rate, it doesn't really help the overall situation to create "successful schools" by pushing difficult students off into some educational never-never-land.

Now when CRHJS expels a student, that student likely goes back to a public school. What happens if you re-create CRHJS strictness in public schools, and the public school expels the student?

Is it okay for some number of students to be expelled from the public educational system? If not, what happens to these students?

Anonymous said...

Rachel: I agree with you that selection can have two positive effects for the school in question: (1) The students selected for the school are likely to be of a higher caliber than students who are not selected (or who do not apply); and also (2) The fact that other students in the class are motivated and not disruptive almost definitely benefits their classmates' ability to succeed and learn.

The question is: How can public schools enforce the same level of discipline and committment? And would society be satisfied with a public school system in which a certain number (say, one third) of the students are expelled for failing to live up to the requirements?

If so, what would happen to the expelled students? How early could the kids be expelled from the system? 6th grade? 8th grade? 10th grade? A lot of questions to be answered on this one...