Wednesday, September 26, 2012

Guest Post: Teacher Opposition to VAM

by Kerri Tobin, PhD

Teacher opposition to value-added modeling gets portrayed in the media as teachers refusing to take any kind of responsibility for student learning or academic growth. But most teachers do not balk at the idea that they should be accountable for advancing their students’ knowledge. What they oppose is value-added modeling, or VAM, the highly-imprecise tool that is being used to measure teachers' impact on students' learning.. When the New York Times won the right to publishteacher VAM scores this spring, only the tiny print noted the inaccuracy of the data, including some standard errors larger than the purported effect sizes. Researchers familiar with value-added modeling have repeatedly voiced their concern about its use in high-stakes decisions like teacher evaluations (for example: herehere, and here).

Often, opponents of current uses of standardized testing fall back on attempts to draw parallels between teaching and medicine: “It’s like grading doctors on how many of their patients die!” But these analogies are imprecise; they do not do the problem justice. John Ewing’s fertilizer analogy is fascinating but perhaps too long for the common person. What we need is an “elevator speech” – one that can be delivered in the time it takes to get from the ground to the 3rd floor – to explain why teachers oppose VAM. A better analogy is this: using VAM to evaluate teachers is akin to evaluating chemotherapists based on how much their patients’ tumors shrink every year. The expected rate of tumor shrinkage is calculated based on patients’ race and socioeconomic status. So if a patient’s tumor shrinks more, over that year, than expected, the doctor gets a positive score. If it shrinks less than it “should,” the doctor gets a negative score. The average of these patients’ scores becomes the doctor’s overall VAM score. Sound reasonable? Maybe, until you consider that a) different kinds of tumors respond differently to chemotherapy; b) the doctor has no control over what patients do outside the office (for example, lung cancer sufferers who continue to smoke); c) his patients saw a different chemotherapist last year and will see another one next year; and, perhaps most importantly, d) the doctor is not allowed to treat any co-morbid conditions (for example, a cancer patient with diabetes gets no treatment to manage his blood sugar) – even if he wanted to, and even in cases where the patient or his family might prefer that other conditions be treated instead of the cancer (e.g., when parents value social or self-confidence issues more than test scores), there simply aren’t enough hours in his day. Factors like the overall health of the patient, his lifestyle, eating habits, substance use, weight, and blood pressure might impact the effectiveness of the chemo. But the doctor cannot control these, in much the same way teachers cannot control where students live, if they have enough to eat and get regular medical care, whether anyone reads to them at home, how much TV they watch, or what time they go to bed at night. And in the same way our VAM-assessed doctor would be powerless to decide that a patient dying of AIDS needs antiretroviral therapy before he can tolerate chemo for a concurrent cancer, teachers have neither the time, resources, nor training to solve the problems in their students’ lives – emotional problems, health challenges, family issues, etc. – that impact their academic growth. This is how VAM works, and why teachers oppose it.


Kerri Tobin is Assistant Professor of Education at Marywood University in Scranton, PA, where she researches the educational needs of students living in poverty and prepares teachers and school leaders to meet those needs.


RDT said...

I'm coming increasingly to believe that one of the over-arching problems facing education today is the speed with which "interesting ideas" get turned into "solutions" by policy pundits.

Value-added is certainly a step forward from raw test scores as a way to evaluate schools and teachers. But it seems like only days elapsed between when VAM hit the literature and when it was adopted by reformers as the next silver bullet.

Roger Sweeny said...

Oh, puhlease.

Teachers are certainly not omnipotent, any more than doctors are. The question is, do they have any power?

If there were two doctors, working in the same practice and each seeing a hundred patients a year, and one doctor had an average shrinkage rate of 75% and the other doctor had an average shrinkage rate of 25%, I would have no problem in paying the first doctor more, and saying that there is a very good chance she is a better cancer doctor.

Each of the doctors will be powerless to do a lot of things that would improve their patients' health, but those things average out.

It is an empirical question just how much "better" one teacher is than another. There's a lot of randomness in life and teachers' value added scores are going to bounce around year to year. Only by doing the experiment and looking seriously at the results will we see if some teachers consistently outperform others--and whether the difference is worth extra money or perks.

RDT said...

That was part of my point -- that there needs to be time to do the experiments. But it seems like 3 or 4 years ago VAM was the new new thing, and now we're debating what "percentage" of a teachers evaluation they should be. If there's a body of literature that shows what the "noise" from year to year is, or whether there are interesting unintended consequences, I haven't seen it.

Unknown said...

Determine which cancer organization you wish to donate money to. There are a variety of cancer organizations and medical institutions that will accept donations towards cancer research, including Hospital, the National Breast Cancer Foundation and the Cancer Research Institute. Thanks for sharing.
hcg 1234