Many people noted the article on the turnaround of Brockton HS in yesterday's NY Times. But I didn't notice anybody critiquing the headline. The piece is titled "4,100 Students Prove 'Small is Better' Rule Wrong".
Yipes. Regardless of whether or not small schools really outperform larger ones* or whether or not Brockton has really turned around, the headline is absurd. Exceptions don't disprove rules in social science. Exceptions are, by definition, exceptional. Can we please take that into account when we discuss education?
Someone driving home safely while drunk doesn't prove that it's not dangerous to drink and drive. A sole survivor of a plane crash doesn't prove that plane crashes don't kill. A child from a troubled neighborhood graduating from Harvard doesn't prove that growing up in a troubled neighborhood doesn't impact one's educational performance. And one large school excelling doesn't mean that small schools aren't generally better.
update: GothamSchools outdoes the Times yet again by linking to the article with a far more responsible phrase -- "Brockton High School in Massachusetts dispels the myth that only small schools can improve."
*the evidence that small schools actually are better is, to my understanding, weak at best -- but that's not relevant to the point I'm making
Tuesday, September 28, 2010
Thursday, September 23, 2010
The Primary Purpose of Merit Pay
The most popular opinion of the last few days seems to be that the primary purpose of merit pay is to re-shape the teacher labor force by attracting and retaining better teachers. The notion that performance incentives would motivate teachers to perform better in the classroom has been implicitly or explicitly derided as silly and/or unimportant.
Did I miss something? Maybe I need to do some archival research, but I could've sworn that before the release of the results there weren't many merit pay proponents making this argument. But since learning of the lack of effect on standardized test scores in the Nashville experiment, it seems to be the only one I hear.
After learning of the results, Rick Hess wrote that
The second school of thought, and the one that interests serious people, is the proposition that rethinking teacher pay can help us reshape the profession to make it more attractive to talented candidates, more adept at using specialization, more rewarding for accomplished professionals, and a better fit for the twenty-first century labor force.
and the Washington Post quotes Eric Hanushek saying
The biggest role of incentives has to do with selection of who enters and who stays in teaching - i.e., how incentives change the teaching corps through entrance and exits . . . I have always thought that the effort effects were small relative to the potential for getting different teachers. Their study has nothing to say about this more important issue.
and Tom Kane writes:
the impact of the specific incentive they tested depends on what underlies the differences in teacher effectiveness–effort vs. talent and accumulated skill. I’ve never believed that lack of teacher effort–as opposed to talent and skills–was the primary issue underlying poor student achievement gains. Rather, the primary hope for merit pay is that it will encourage talented teachers to remain in the classroom or to enter teaching.
the Obama administration's official position seems to align with that too. Here's how the same Washington Post article described their repsonse:
While this is a good study, it only looked at the narrow question of whether more pay motivates teachers to try harder," said Peter Cunningham, assistant U.S. education secretary for communications and outreach. "What we are trying to do is change the culture of teaching by giving all educators the feedback they need to get better while rewarding and incentivizing the best to teach in high-need schools, hard to staff subjects. This study doesn't address that objective.
Maybe I'm wrong and there are more people that would've agreed with these four statements a few days ago than I think, but there were certainly more than a couple people arguing that performance incentives would increase teachers' motivation, improve their classroom performance, and subsequently increase the academic performance of their students. I've had conversations with people who've directly told me that lack of motivation is a huge problem in teaching and that providing proper incentives would fix this.
Without more research, I can't tell you whether people have conveniently changed their mind about the primary purpose of performance pay or whether those who believe it should be used primarily to alter the teacher labor force are now simply stepping to the forefront while those who believed in its motivational potential are shrinking into the background. But I'd guess that it's a little of both.
On the plus side, might everyone now agree that teacher pay should be re-fashioned with the primary goal being to encourage the recruitment and retention of excellent teachers? Do I hear a consenus emerging? I guess time will tell . . .
Did I miss something? Maybe I need to do some archival research, but I could've sworn that before the release of the results there weren't many merit pay proponents making this argument. But since learning of the lack of effect on standardized test scores in the Nashville experiment, it seems to be the only one I hear.
After learning of the results, Rick Hess wrote that
The second school of thought, and the one that interests serious people, is the proposition that rethinking teacher pay can help us reshape the profession to make it more attractive to talented candidates, more adept at using specialization, more rewarding for accomplished professionals, and a better fit for the twenty-first century labor force.
and the Washington Post quotes Eric Hanushek saying
The biggest role of incentives has to do with selection of who enters and who stays in teaching - i.e., how incentives change the teaching corps through entrance and exits . . . I have always thought that the effort effects were small relative to the potential for getting different teachers. Their study has nothing to say about this more important issue.
and Tom Kane writes:
the impact of the specific incentive they tested depends on what underlies the differences in teacher effectiveness–effort vs. talent and accumulated skill. I’ve never believed that lack of teacher effort–as opposed to talent and skills–was the primary issue underlying poor student achievement gains. Rather, the primary hope for merit pay is that it will encourage talented teachers to remain in the classroom or to enter teaching.
the Obama administration's official position seems to align with that too. Here's how the same Washington Post article described their repsonse:
While this is a good study, it only looked at the narrow question of whether more pay motivates teachers to try harder," said Peter Cunningham, assistant U.S. education secretary for communications and outreach. "What we are trying to do is change the culture of teaching by giving all educators the feedback they need to get better while rewarding and incentivizing the best to teach in high-need schools, hard to staff subjects. This study doesn't address that objective.
Maybe I'm wrong and there are more people that would've agreed with these four statements a few days ago than I think, but there were certainly more than a couple people arguing that performance incentives would increase teachers' motivation, improve their classroom performance, and subsequently increase the academic performance of their students. I've had conversations with people who've directly told me that lack of motivation is a huge problem in teaching and that providing proper incentives would fix this.
Without more research, I can't tell you whether people have conveniently changed their mind about the primary purpose of performance pay or whether those who believe it should be used primarily to alter the teacher labor force are now simply stepping to the forefront while those who believed in its motivational potential are shrinking into the background. But I'd guess that it's a little of both.
On the plus side, might everyone now agree that teacher pay should be re-fashioned with the primary goal being to encourage the recruitment and retention of excellent teachers? Do I hear a consenus emerging? I guess time will tell . . .
Wednesday, September 22, 2010
Nashville Incentive Pay Experiment: Results Wrap-Up
The National Center on Performance Incentives released the final report on the Nashville performance pay experiment (known as POINT) yesterday. The press release is available here, and the full report is available here.
The study involved 296 middle school math teachers in Nashville who were assigned to either a treatment group (eligible for bonuses of 5, 10, or $15,000) or a control group and then tracked for three years.
The main result was that students assigned to bonus-eligible teachers did not perform any better than students assigned to treatment group teachers. The lone exception were 5th grade students in years 2 and 3 of the study, but the gains did not persist through the end of 6th grade. The main portion of the executive summary reads as follows:
Prior to implementation, researchers calculated that 55% of teachers would be able to obtain bonuses if their students answered only an additional 2-3 questions correct on the state test. Across the three years of the study, 33.6% of bonus-eligible teachers earned a bonus in one or more years.
This means that most teachers did not dramatically improve, even for one year. It also means that the tendency of test scores to bounce around significantly did not result in different random groups of teachers receiving a bonus each year. About two-thirds of teachers in the treatment group never received a bonus. And about two-thirds of the teachers who received a bonus earned one in multiple years.
Of potentially more import are the results from the teacher interviews and surveys. These will continue to be analyzed in the coming months, but right now the main takeaway point is that teachers in the treatment group really didn't report changing much at all. About 80% of teachers reported that they were already working as hard as they could before the incentives were implemented and were therefore unable to work any harder afterward.
There are two potential bright spots, however, for merit pay proponents:
1.) The project ran smoothly (e.g. the right teachers received the right bonuses at the right time) and didn't suffer any major backlash. That this was truly a partnership between union, university, district, and other groups probably helped in this regard.
2.) It's unclear right now, but the bonuses may have had a small effect on the patterns of teacher attrition for those in the bonus group. 27 teachers in the control group left to teach in another district while only 15 teachers in the treatment group did. The numbers are small, so conclusions are hard to draw, but the most popular criticism of the study by performance pay advocates seems to be that it didn't shed any light on how performance pay might affect teacher recruitment and retention.
There was also no evidence that treatment group teachers were successful in gaming the system by having more kids (or lower performing kids) removed from their class, preventing new students from transferring into their classes, focusing more efforts on math instruction at the expense of other subjects, or helping their students cheat on the state tests.
What does it mean?
I wrote before that this was one of the most important studies this decade. That said, it's still just one study. And we have to be careful about how we extrapolate from the results of any one study. The main question the study was set up to answer was whether offering performance incentives to individual teachers would result in their students performing better on standardized tests. The study offered no evidence that this was the case, and little reason to believe that this would be the case for similarly designed incentive systems.
Why is this so? The main reason seems to be the lack of any major changes by teachers. But there are a couple of other possibilities. Just because teachers reported that they didn't change anything doesn't necessarily mean that they didn't change anything. So it's not out of the realm of possibility that at least some of the teachers made changes but that these changes didn't yield subsequent results. It's also possible that, despite the fact that teachers received the project fairly well, they wanted to prove the working hypothesis wrong -- that is, that teachers eligible for a bonus, at least on some level, resented the fact that somebody thought they'd teach better if they were offered a carrot, and decided to not work harder despite the carrot looking awfully delicious. At the same time, the control group teachers could've worked a little harder to prove that they were plenty motivated to teach solely because they wanted their students to succeed. While the answer is likely a little of all of the above, I tend to think the most likely scenario is that teachers simply weren't all that much more motivated by the prospect of a bonus fairly far down the road (teachers were paid the following November).
What it doesn't mean is that all merit pay schemes forevermore are doomed to abject failure. We don't know if different types of bonuses awarded in different ways (shorter time spans, group awards, non-monetary awards, etc.) might have a larger effect, and we know little about how performance pay affects the long-term make-up of the teacher labor force. At the same time, it does call into serious question the application of the overly simplistic homo economicus model used by economists. My gut feeling is that economists tend to view teacher motivation and the teaching profession in an overly simplistic manner guided too much by basic economic theories and not enough by the literature on the sociology of teaching or the psychology of motivation.
Where do we go from here?
NCPI has another study utilizing team-based incentives that should be out at some point over the next couple of years. In addition, other non-randomized studies of the myriad incentive systems that have sprouted all over the country the past couple years are certainly underway as well. I'm not sure why the NCPI researchers chose to claim that this was "the first scientific study of performance pay ever conducted in the United States," since the definition of "scientific" is hotly debated, but the literature base will certainly continue to grow in the coming years regardless.
In addition to the continuing analysis of the data from this study, there are plenty of opportunities for other researchers and funding agencies to examine questions surrounding the impact of merit pay on the teacher labor force, on school culture, on student outcomes other than test scores, and many other areas.
In the meantime, it's important that merit pay opponents not claim that this study proves once and for all that merit pay does not, and will not ever, work in schools. If nothing else, it appears likely that better teachers got paid more money than worse teachers -- which is arguably an improvement on the current system. And, at the same time, it's important that merit pay proponents not claim that this study is meaningless and that we should recklessly proceed with merit pay in schools at breakneck speed.
Merit pay has grown considerably more popular among teachers over the past decade, so it's eminently possible that districts and unions can work together to design and implement better, more nuanced merit pay systems that might have a better chance of success. I don't think anybody would argue that the status quo is the perfect system, so simply saying that merit pay won't work isn't a solution. But these results indicate that we should proceed with caution. The assumption that teachers will work harder for financial incentives is now a dangerous one to make and should be made with caution. As such, the performance pay systems that will undoubtedly continue to emerge should begin with a more nuanced and informed understanding of the practices and motivations of current and prospective teachers. We can only hope that an informed and open-minded approach from both sides will eventually result in compensation systems that attract and reward good teachers in ways that current teachers find meaningful and fair.
For more information, see my previous posts on the experiment:
Part 1: Background Info
Part 2: What to Look For
Part 3: Why it Matters
Part 4: What We Can Learn
Live-Blog of Results
The study involved 296 middle school math teachers in Nashville who were assigned to either a treatment group (eligible for bonuses of 5, 10, or $15,000) or a control group and then tracked for three years.
The main result was that students assigned to bonus-eligible teachers did not perform any better than students assigned to treatment group teachers. The lone exception were 5th grade students in years 2 and 3 of the study, but the gains did not persist through the end of 6th grade. The main portion of the executive summary reads as follows:
POINT was focused on the notion that a significant problem in American education is the absence of appropriate incentives, and that correcting the incentive structure would, in and of itself, constitute an effective intervention that improved student outcomes.
By and large, results did not confirm this hypothesis. While the general trend in middle school mathematics performance was upward over the period of the project, students of teachers randomly assigned to the treatment group (eligible for bonuses) did not outperform students whose teachers were assigned to the control group (not eligible for bonuses).
Prior to implementation, researchers calculated that 55% of teachers would be able to obtain bonuses if their students answered only an additional 2-3 questions correct on the state test. Across the three years of the study, 33.6% of bonus-eligible teachers earned a bonus in one or more years.
This means that most teachers did not dramatically improve, even for one year. It also means that the tendency of test scores to bounce around significantly did not result in different random groups of teachers receiving a bonus each year. About two-thirds of teachers in the treatment group never received a bonus. And about two-thirds of the teachers who received a bonus earned one in multiple years.
Of potentially more import are the results from the teacher interviews and surveys. These will continue to be analyzed in the coming months, but right now the main takeaway point is that teachers in the treatment group really didn't report changing much at all. About 80% of teachers reported that they were already working as hard as they could before the incentives were implemented and were therefore unable to work any harder afterward.
There are two potential bright spots, however, for merit pay proponents:
1.) The project ran smoothly (e.g. the right teachers received the right bonuses at the right time) and didn't suffer any major backlash. That this was truly a partnership between union, university, district, and other groups probably helped in this regard.
2.) It's unclear right now, but the bonuses may have had a small effect on the patterns of teacher attrition for those in the bonus group. 27 teachers in the control group left to teach in another district while only 15 teachers in the treatment group did. The numbers are small, so conclusions are hard to draw, but the most popular criticism of the study by performance pay advocates seems to be that it didn't shed any light on how performance pay might affect teacher recruitment and retention.
There was also no evidence that treatment group teachers were successful in gaming the system by having more kids (or lower performing kids) removed from their class, preventing new students from transferring into their classes, focusing more efforts on math instruction at the expense of other subjects, or helping their students cheat on the state tests.
What does it mean?
I wrote before that this was one of the most important studies this decade. That said, it's still just one study. And we have to be careful about how we extrapolate from the results of any one study. The main question the study was set up to answer was whether offering performance incentives to individual teachers would result in their students performing better on standardized tests. The study offered no evidence that this was the case, and little reason to believe that this would be the case for similarly designed incentive systems.
Why is this so? The main reason seems to be the lack of any major changes by teachers. But there are a couple of other possibilities. Just because teachers reported that they didn't change anything doesn't necessarily mean that they didn't change anything. So it's not out of the realm of possibility that at least some of the teachers made changes but that these changes didn't yield subsequent results. It's also possible that, despite the fact that teachers received the project fairly well, they wanted to prove the working hypothesis wrong -- that is, that teachers eligible for a bonus, at least on some level, resented the fact that somebody thought they'd teach better if they were offered a carrot, and decided to not work harder despite the carrot looking awfully delicious. At the same time, the control group teachers could've worked a little harder to prove that they were plenty motivated to teach solely because they wanted their students to succeed. While the answer is likely a little of all of the above, I tend to think the most likely scenario is that teachers simply weren't all that much more motivated by the prospect of a bonus fairly far down the road (teachers were paid the following November).
What it doesn't mean is that all merit pay schemes forevermore are doomed to abject failure. We don't know if different types of bonuses awarded in different ways (shorter time spans, group awards, non-monetary awards, etc.) might have a larger effect, and we know little about how performance pay affects the long-term make-up of the teacher labor force. At the same time, it does call into serious question the application of the overly simplistic homo economicus model used by economists. My gut feeling is that economists tend to view teacher motivation and the teaching profession in an overly simplistic manner guided too much by basic economic theories and not enough by the literature on the sociology of teaching or the psychology of motivation.
Where do we go from here?
NCPI has another study utilizing team-based incentives that should be out at some point over the next couple of years. In addition, other non-randomized studies of the myriad incentive systems that have sprouted all over the country the past couple years are certainly underway as well. I'm not sure why the NCPI researchers chose to claim that this was "the first scientific study of performance pay ever conducted in the United States," since the definition of "scientific" is hotly debated, but the literature base will certainly continue to grow in the coming years regardless.
In addition to the continuing analysis of the data from this study, there are plenty of opportunities for other researchers and funding agencies to examine questions surrounding the impact of merit pay on the teacher labor force, on school culture, on student outcomes other than test scores, and many other areas.
In the meantime, it's important that merit pay opponents not claim that this study proves once and for all that merit pay does not, and will not ever, work in schools. If nothing else, it appears likely that better teachers got paid more money than worse teachers -- which is arguably an improvement on the current system. And, at the same time, it's important that merit pay proponents not claim that this study is meaningless and that we should recklessly proceed with merit pay in schools at breakneck speed.
Merit pay has grown considerably more popular among teachers over the past decade, so it's eminently possible that districts and unions can work together to design and implement better, more nuanced merit pay systems that might have a better chance of success. I don't think anybody would argue that the status quo is the perfect system, so simply saying that merit pay won't work isn't a solution. But these results indicate that we should proceed with caution. The assumption that teachers will work harder for financial incentives is now a dangerous one to make and should be made with caution. As such, the performance pay systems that will undoubtedly continue to emerge should begin with a more nuanced and informed understanding of the practices and motivations of current and prospective teachers. We can only hope that an informed and open-minded approach from both sides will eventually result in compensation systems that attract and reward good teachers in ways that current teachers find meaningful and fair.
For more information, see my previous posts on the experiment:
Part 1: Background Info
Part 2: What to Look For
Part 3: Why it Matters
Part 4: What We Can Learn
Live-Blog of Results
Tuesday, September 21, 2010
Live-Blogging the Release of the Nashville Performance Pay Experiment Results
Today marks the release of the National Center on Performance Incentives' report on the Nashville performance pay experiment (known as POINT). The full report will be available on their website after the discussion. In the meantime, I'm going to provide you with some snapshots of what's being said and written. Please check out my previous posts on this study as well. A live, streaming, video of the press conference is online here. The press release (which is pretty good) is now available here. The full report is available here.
Previous posts:
Part 1: Background Info
Part 2: What to Look For
Part 3: Why it Matters
Part 4: What We Can Learn
12:50pm: The press conference has begun. I'm going to begin by posting the summary statement from the executive summary of the report:
POINT was focused on the notion that a significant problem in American education is the absence of appropriate incentives, and that correcting the incentive structure would, in and of itself, constitute an effective intervention that improved student outcomes.
By and large, results did not confirm this hypothesis. While the general trend in middle school mathematics performance was upward over the period of the project, students of teachers randomly assigned to the treatment group (eligible for bonuses) did not outperform students whose teachers were assigned to the control group (not eligible for bonuses).
Before you get excited, or disappointed, about this, bear in mind what I've written before -- the most important things we can learn from this study aren't what happen to test scores, they're insights into teacher behavior from the interviews and surveys. Keep checking back for more details on this, and other, topics.
Also, please keep in mind that this study does not definitively prove either that merit pay systems are a bad idea or a good idea.
1:00pm: The number of teachers who received bonuses remained steady throughout (40, 41, 44), but the number of eligible teachers declined significantly (143, 105, 84) -- meaning over half of the teachers received a bonus for being above the historical 80th percentile of teachers in the last year. This could mean that less successful teachers tended to leave the study -- whether by switching subjects, schools, or careers. Or it could mean that the tests became easier and more teachers were rewarded. I would say that it could mean that it took three years for the incentives to have an effect, but the treatment group did not outperform the control group -- even in the final year.
from the executive summary: "attrition of teachers from POINT was high. By the end of the project, half of the initial participants had left the experiment."
The report says that the differences in attrition -- and reasons for attrition -- between control and treatment groups was not statistically significant. But it reports that 27 control group teachers, and only 15 treatment group teachers, left the study because they'd switched districts during the experiment. I'd like to know more about this and whether this might be evidence that incentives made teachers slightly more likely to remain in the Metro Nashville school system.
about teacher attrition, from the report:
Teachers who left the study tended to differ from stayers on many of the baseline variables. Teachers who dropped out by the end of the second year of the experiment were more likely to be black, less likely to be white. They tended to be somewhat younger than teachers who remained in the study all three years. These dropouts were also hired more recently, on average. They had less experience (including less prior experience outside the district), and more of them were new teachers without tenure compared to teachers who remained in the study at the end of the second year. Dropouts were more likely to have alternative certification and less likely to have professional licensure. Their pre-POINT teaching performance (as measured by an estimate of 2005-06 value added) was lower than that of retained teachers, and they had more days absent. Dropouts completed significantly more mathematics professional development credits than the teachers who stayed.Dropouts also tended to teach classes with relatively more black students and fewer white students. They were more likely to be teaching special education students.
I'm going to need a little time to digest that, but the next table demonstrates that treatment and control group teachers, in all three years, did not differ in terms of effectiveness (as measured by tests) in the years prior to the experiment -- meaning that more effective teachers didn't seem more likely to stay if they could possibly earn bonuses. Treatment group teachers, however, were six percentage points less likely to leave the middle school in which they started during the three years.
1:03pm: There were, however, positive effects for 5th grade teachers in the 2nd and 3rd years, though the effects did not persist until the end of 6th grade. The 5th grade teachers had the same students all day for multiple subjects, so it's possible that they shifted focus to math instruction or that they simply knew their students better and were able to get better results. The center did analyze results on other subject tests and found no differences that would indicate teachers ignored these subjects and only focused on math.
33.6% of the treatment group received a bonus in at least one year (out of 152, 16 won once, 17 twice, and 18 thrice). Analysis done by the researchers prior to the experiment found that 55% of teachers were within a few more correct questions per student of attaining scores that would earn them a bonus.
1:08pm: Here are some other interesting tidbits from the executive summary, report, and press conference (which is now in Q&A):
-80% of teachers reported that they were already working as hard as they could and didn't change their effort due to the opportunity to earn an incentive.
-from the executive summary: "The introduction of performance incentives in MNPS middle schools did not set off significant negative reactions of the kind that have attended the introduction of merit pay elsewhere. But neither did it yield consistent and lasting gains in test scores. It simply did not do much of anything."
-from the report: "From an implementation standpoint, POINT was a success. This is not a trivial result, given the widespread perception that teachers are adamantly opposed to merit pay and will resist its implementation in any form."
-I didn't mention this before, but the placement of teachers in control/treatment groups -- and whether or not they received a bonus was, officially, confidential. The center didn't distribute this information to principals or teachers and participating teachers signed statements saying that they wouldn't tell anybody their status. It looks like this was at least moderately successful, as about 75% of teachers reported that they didn't know if anybody had won a bonus in their school.
-There were no differences in student attrition between groups -- meaning that there's no evidence that bonus-eligible teachers were more likely to get problem students removed from their class. There were also no differences in students enrolling late, and students who left treatment teachers' classes were no lower scoring.
1:12pm: Dale Ballou responds to a question by saying that test scores bounced around a lot before the experiment, making it "difficult to extrapolate" from the data prior to the start of the experiment and tell whether teachers were doing better post-bonus than pre-bonus.
1:16pm: From the report, here are the results from the teacher survey. In short, nothing huge or shocking (note TCAP is the TN state test):
There are few survey items on which we have found a significant difference between the responses of treatment teachers and control teachers. (We note all contrasts with p values less than 0.15.) Treatment teachers were more likely to respond that they aligned their mathematics instruction with MNPS standards (p = 0.11). They spent less time re-teaching topics or skills based on students’ performance on classroom tests (p = 0.04). They spent more time having students answer items similar to those on the TCAP (p = 0.09) and using other TCAP-specific preparation materials (p = 0.02). The only other significant differences were in collaborative activities, with treatment teachers replying that they collaborated more on virtually every measured dimension. Data from administrative records and from surveys administered to the district’s math mentors also show few differences between treatment and control groups. Although treatment teachers completed more hours of professional development in core academic subjects, the difference was small (0.14 credit hours when the sample mean was 28) and only marginally significant (p = 0.12). Moreover, there was no discernible difference in professional development completed in mathematics. Likewise, treatment teachers had no more overall contact with the district’s math mentors than teachers in the control group.
1:20pm: One question I really want answered (that I don't see in the report) is how the bonus winners and losers differed (experience, school type, etc.), if they differed at all. A questioner just asked something along those lines, and the response was that they've just begun to look at that, but that they're wary to draw too many conclusions about this because the sample sizes will get smaller the more focused the question is.
1:27pm: not sure this is quite verbatim, from the panel, but it's a good point nonetheless: "just because we didn't find an effect from doing incentives this way doesn't mean you wouldn't find results by doing it another way."
1:43pm: The questions aren't really helping to advance the discussion all that much -- in part b/c there are too many speeches and not enough questions. The line at the microphone is now too long for me to get a question in before the end of the session, but here's what I'd ask:
1.) There are two assumptions behind performance pay that this study was designed to test: one is that teachers will respond to the opportunity to earn incentives by changing their practice, and two is that these changes in practice will result in better student performance. The report says that there were very few differences in reported teacher behavior between treatment and control groups, but is there enough data to draw any conclusions about the performance of those who did report making changes?
I had a second question, but it now completely escapes me. I'll try to find an answer to at least that question, at least. That's going to do it for now, but expect a more concise summary of the report's findings in the next 24 hours or so.
Wait, just kidding: here are the other four questions that are rattling around in my brain right now. I ran into one of the researchers in the hallway and asked him the question above in addition to the first three below. The answer was basically that they're good questions, and ones they plan on looking into more -- the focus for today was completing the analysis of the trends in test scores.
2.) What, if any, differences were there between bonus winners and losers (e.g. were winners more experienced, teaching in better schools, teaching higher/lower performing students, teaching a different subject, etc.)?
3.) What, if any differences, were there between the reactions of bonus winners and losers (e.g. more likely to stay/leave, more/less critical of experiment, more/less enthusiastic, more/less likely to report subsequent changes in behaviors)?
4.) I think 15 treatment group and 27 control group teachers left the district during the experiment. Can we consider that statistic (and maybe any relevant survey/interview questions) evidence that teachers might be less likely to leave an urban district if they can earn a bonus there but not elsewhere?
5.) A much higher percentage of the treatment group teachers (over half, compared to less than one-third) won bonuses in the final year. Is that simply because the less experienced teachers were much more likely to leave their assignments, because the test changed, or something else?
Now that's (really) all for now.
Previous posts:
Part 1: Background Info
Part 2: What to Look For
Part 3: Why it Matters
Part 4: What We Can Learn
12:50pm: The press conference has begun. I'm going to begin by posting the summary statement from the executive summary of the report:
POINT was focused on the notion that a significant problem in American education is the absence of appropriate incentives, and that correcting the incentive structure would, in and of itself, constitute an effective intervention that improved student outcomes.
By and large, results did not confirm this hypothesis. While the general trend in middle school mathematics performance was upward over the period of the project, students of teachers randomly assigned to the treatment group (eligible for bonuses) did not outperform students whose teachers were assigned to the control group (not eligible for bonuses).
Before you get excited, or disappointed, about this, bear in mind what I've written before -- the most important things we can learn from this study aren't what happen to test scores, they're insights into teacher behavior from the interviews and surveys. Keep checking back for more details on this, and other, topics.
Also, please keep in mind that this study does not definitively prove either that merit pay systems are a bad idea or a good idea.
1:00pm: The number of teachers who received bonuses remained steady throughout (40, 41, 44), but the number of eligible teachers declined significantly (143, 105, 84) -- meaning over half of the teachers received a bonus for being above the historical 80th percentile of teachers in the last year. This could mean that less successful teachers tended to leave the study -- whether by switching subjects, schools, or careers. Or it could mean that the tests became easier and more teachers were rewarded. I would say that it could mean that it took three years for the incentives to have an effect, but the treatment group did not outperform the control group -- even in the final year.
from the executive summary: "attrition of teachers from POINT was high. By the end of the project, half of the initial participants had left the experiment."
The report says that the differences in attrition -- and reasons for attrition -- between control and treatment groups was not statistically significant. But it reports that 27 control group teachers, and only 15 treatment group teachers, left the study because they'd switched districts during the experiment. I'd like to know more about this and whether this might be evidence that incentives made teachers slightly more likely to remain in the Metro Nashville school system.
about teacher attrition, from the report:
Teachers who left the study tended to differ from stayers on many of the baseline variables. Teachers who dropped out by the end of the second year of the experiment were more likely to be black, less likely to be white. They tended to be somewhat younger than teachers who remained in the study all three years. These dropouts were also hired more recently, on average. They had less experience (including less prior experience outside the district), and more of them were new teachers without tenure compared to teachers who remained in the study at the end of the second year. Dropouts were more likely to have alternative certification and less likely to have professional licensure. Their pre-POINT teaching performance (as measured by an estimate of 2005-06 value added) was lower than that of retained teachers, and they had more days absent. Dropouts completed significantly more mathematics professional development credits than the teachers who stayed.Dropouts also tended to teach classes with relatively more black students and fewer white students. They were more likely to be teaching special education students.
I'm going to need a little time to digest that, but the next table demonstrates that treatment and control group teachers, in all three years, did not differ in terms of effectiveness (as measured by tests) in the years prior to the experiment -- meaning that more effective teachers didn't seem more likely to stay if they could possibly earn bonuses. Treatment group teachers, however, were six percentage points less likely to leave the middle school in which they started during the three years.
1:03pm: There were, however, positive effects for 5th grade teachers in the 2nd and 3rd years, though the effects did not persist until the end of 6th grade. The 5th grade teachers had the same students all day for multiple subjects, so it's possible that they shifted focus to math instruction or that they simply knew their students better and were able to get better results. The center did analyze results on other subject tests and found no differences that would indicate teachers ignored these subjects and only focused on math.
33.6% of the treatment group received a bonus in at least one year (out of 152, 16 won once, 17 twice, and 18 thrice). Analysis done by the researchers prior to the experiment found that 55% of teachers were within a few more correct questions per student of attaining scores that would earn them a bonus.
1:08pm: Here are some other interesting tidbits from the executive summary, report, and press conference (which is now in Q&A):
-80% of teachers reported that they were already working as hard as they could and didn't change their effort due to the opportunity to earn an incentive.
-from the executive summary: "The introduction of performance incentives in MNPS middle schools did not set off significant negative reactions of the kind that have attended the introduction of merit pay elsewhere. But neither did it yield consistent and lasting gains in test scores. It simply did not do much of anything."
-from the report: "From an implementation standpoint, POINT was a success. This is not a trivial result, given the widespread perception that teachers are adamantly opposed to merit pay and will resist its implementation in any form."
-I didn't mention this before, but the placement of teachers in control/treatment groups -- and whether or not they received a bonus was, officially, confidential. The center didn't distribute this information to principals or teachers and participating teachers signed statements saying that they wouldn't tell anybody their status. It looks like this was at least moderately successful, as about 75% of teachers reported that they didn't know if anybody had won a bonus in their school.
-There were no differences in student attrition between groups -- meaning that there's no evidence that bonus-eligible teachers were more likely to get problem students removed from their class. There were also no differences in students enrolling late, and students who left treatment teachers' classes were no lower scoring.
1:12pm: Dale Ballou responds to a question by saying that test scores bounced around a lot before the experiment, making it "difficult to extrapolate" from the data prior to the start of the experiment and tell whether teachers were doing better post-bonus than pre-bonus.
1:16pm: From the report, here are the results from the teacher survey. In short, nothing huge or shocking (note TCAP is the TN state test):
There are few survey items on which we have found a significant difference between the responses of treatment teachers and control teachers. (We note all contrasts with p values less than 0.15.) Treatment teachers were more likely to respond that they aligned their mathematics instruction with MNPS standards (p = 0.11). They spent less time re-teaching topics or skills based on students’ performance on classroom tests (p = 0.04). They spent more time having students answer items similar to those on the TCAP (p = 0.09) and using other TCAP-specific preparation materials (p = 0.02). The only other significant differences were in collaborative activities, with treatment teachers replying that they collaborated more on virtually every measured dimension. Data from administrative records and from surveys administered to the district’s math mentors also show few differences between treatment and control groups. Although treatment teachers completed more hours of professional development in core academic subjects, the difference was small (0.14 credit hours when the sample mean was 28) and only marginally significant (p = 0.12). Moreover, there was no discernible difference in professional development completed in mathematics. Likewise, treatment teachers had no more overall contact with the district’s math mentors than teachers in the control group.
1:20pm: One question I really want answered (that I don't see in the report) is how the bonus winners and losers differed (experience, school type, etc.), if they differed at all. A questioner just asked something along those lines, and the response was that they've just begun to look at that, but that they're wary to draw too many conclusions about this because the sample sizes will get smaller the more focused the question is.
1:27pm: not sure this is quite verbatim, from the panel, but it's a good point nonetheless: "just because we didn't find an effect from doing incentives this way doesn't mean you wouldn't find results by doing it another way."
1:43pm: The questions aren't really helping to advance the discussion all that much -- in part b/c there are too many speeches and not enough questions. The line at the microphone is now too long for me to get a question in before the end of the session, but here's what I'd ask:
1.) There are two assumptions behind performance pay that this study was designed to test: one is that teachers will respond to the opportunity to earn incentives by changing their practice, and two is that these changes in practice will result in better student performance. The report says that there were very few differences in reported teacher behavior between treatment and control groups, but is there enough data to draw any conclusions about the performance of those who did report making changes?
I had a second question, but it now completely escapes me. I'll try to find an answer to at least that question, at least. That's going to do it for now, but expect a more concise summary of the report's findings in the next 24 hours or so.
Wait, just kidding: here are the other four questions that are rattling around in my brain right now. I ran into one of the researchers in the hallway and asked him the question above in addition to the first three below. The answer was basically that they're good questions, and ones they plan on looking into more -- the focus for today was completing the analysis of the trends in test scores.
2.) What, if any, differences were there between bonus winners and losers (e.g. were winners more experienced, teaching in better schools, teaching higher/lower performing students, teaching a different subject, etc.)?
3.) What, if any differences, were there between the reactions of bonus winners and losers (e.g. more likely to stay/leave, more/less critical of experiment, more/less enthusiastic, more/less likely to report subsequent changes in behaviors)?
4.) I think 15 treatment group and 27 control group teachers left the district during the experiment. Can we consider that statistic (and maybe any relevant survey/interview questions) evidence that teachers might be less likely to leave an urban district if they can earn a bonus there but not elsewhere?
5.) A much higher percentage of the treatment group teachers (over half, compared to less than one-third) won bonuses in the final year. Is that simply because the less experienced teachers were much more likely to leave their assignments, because the test changed, or something else?
Now that's (really) all for now.
Shame on Rick Hess
Rick Hess' post on the Nashville performance pay experiment yesterday received a lot of attention. In the post, Hess, a merit pay proponent, argues that the results of this experiment will "tell us nothing of value".
I found the post somewhat surprising (as did many others, I think). And I'm somewhat ashamed to admit that, as I read it, my inner cynic said "I wonder if Hess knows that the results weren't spectacular and he's just trying to discredit them ahead of time?". But I quickly shook that off, since I tend to give people the benefit of the doubt. Hess raises a lot of good points, and he seems sincere when he warns readers about the dangers of either side making wild claims based on the results.
Well, I was wrong. Wrong to trust Rick Hess. Re-read his post again, but this time be aware that Hess already knew the results of the experiment -- and that the results weren't what he wanted.
That's right. Hess already knew the results. I was informed of this by a reliable source familiar with the project -- who also tells me that Hess has spent the last few years lauding the researchers and chomping at the bit to rub the positive results in the faces of unions across the country. And now that they didn't turn out the way Hess had hoped, he's decided to pretend that he didn't know the results and tell everybody that the results don't matter.
Regardless of one's point of view, that's simply dishonest and disgraceful. Shame on Rick Hess.
Now, that's not to say that Hess is the only one being dishonest. I'm sure that there are plenty of people on both sides of the debate that have been, and/or will be, dishonest to some extent. So I don't want this post to be construed as vindication that opponents of merit pay are always victims and proponents are always evil liars.
But, at the same time, I find this post particularly galling. Hess has made a name for himself lately by being somewhat independent-minded on a number of issues. Sometimes I agree with him, sometimes I disagree -- but he always has something interesting and insightful to say. In short, he gave me good reason to hold him to a higher standard than most policy wonks who too often seem incapable of seeing an issue from multiple sides. Indeed, a good portion of what he wrote yesterday is useful and insightful (though he takes it too far), and that's what made the post so remarkable. But now I don't know what to believe. A number of his points should still stand (for example, examining the ways in which merit pay affect recruitment and retention is important), but I don't know which ones are sincere and which ones are not.
I respect honesty. I respect people who seek to advance the discussion of education policy. Today, I do not respect Rick Hess.
I found the post somewhat surprising (as did many others, I think). And I'm somewhat ashamed to admit that, as I read it, my inner cynic said "I wonder if Hess knows that the results weren't spectacular and he's just trying to discredit them ahead of time?". But I quickly shook that off, since I tend to give people the benefit of the doubt. Hess raises a lot of good points, and he seems sincere when he warns readers about the dangers of either side making wild claims based on the results.
Well, I was wrong. Wrong to trust Rick Hess. Re-read his post again, but this time be aware that Hess already knew the results of the experiment -- and that the results weren't what he wanted.
That's right. Hess already knew the results. I was informed of this by a reliable source familiar with the project -- who also tells me that Hess has spent the last few years lauding the researchers and chomping at the bit to rub the positive results in the faces of unions across the country. And now that they didn't turn out the way Hess had hoped, he's decided to pretend that he didn't know the results and tell everybody that the results don't matter.
Regardless of one's point of view, that's simply dishonest and disgraceful. Shame on Rick Hess.
Now, that's not to say that Hess is the only one being dishonest. I'm sure that there are plenty of people on both sides of the debate that have been, and/or will be, dishonest to some extent. So I don't want this post to be construed as vindication that opponents of merit pay are always victims and proponents are always evil liars.
But, at the same time, I find this post particularly galling. Hess has made a name for himself lately by being somewhat independent-minded on a number of issues. Sometimes I agree with him, sometimes I disagree -- but he always has something interesting and insightful to say. In short, he gave me good reason to hold him to a higher standard than most policy wonks who too often seem incapable of seeing an issue from multiple sides. Indeed, a good portion of what he wrote yesterday is useful and insightful (though he takes it too far), and that's what made the post so remarkable. But now I don't know what to believe. A number of his points should still stand (for example, examining the ways in which merit pay affect recruitment and retention is important), but I don't know which ones are sincere and which ones are not.
I respect honesty. I respect people who seek to advance the discussion of education policy. Today, I do not respect Rick Hess.
A Primer on the Nashville Incentive Pay Experiment, Part 4
Part 4: What Can We Learn?
Previous posts:
Part 1: Background Info
Part 2: What to Look For
Part 3: Why it Matters
Despite what Rick Hess says, we won't learn "nothing" from the results of the study (but do read his post, as most of his points are good ones). So what, exactly, will we learn from the results of the study? It's hard to say exactly, but here are some things that we can and cannot learn from the study:
We can learn how individual teachers respond to a financial incentive offered for individual results.
We cannot learn how individual teachers, groups of teachers, or entire schools respond to financial or other types of incentives offered to groups of teachers or schools.
We can learn whether teachers will change their teaching in ways that will raise student test scores in response to an individual financial incentive.
We cannot learn whether teachers will change their teaching in ways that will increase student engagement, critical thinking, creativity or a myriad of other factors in response to an individual financial incentive.
We can learn whether middle school math teachers in Nashville were more likely to switch schools/districts or leave the profession if they were in the control or treatment group.
We cannot learn whether talented people across the country are more likely to become teachers, and subsequently remain in teaching, if there are performance bonuses in place.
We can learn whether, under this particular system, teachers who are "better" as measured by standardized tests across all three years tend to be rewarded on a year-to-year basis.
We cannot learn whether, under performance pay systems, better teachers (as measured in any number of ways) tend to be paid more.
In short, yes, there are rather severe limitations on what we can learn from this one study. But, at the same time, I'd argue that we can learn more from this study than from most others. Despite what Rick Hess, we will not learn "nothing of value," in part because different people value different things. Hess might think that the teacher recruitment/retention aspect of performance pay is the most important, but plenty of others think the incentivizing of effort is the most important.
What does this mean for the interested observer watching from afar? It means that your ears should perk upif when you hear strongly worded statements from both sides of the debate. This study is one piece in the puzzle -- and an important piece, at that. It's neither the end all and be all of research into performance pay nor an utterly useless waste of time that fails to inform the debate in the least.
Previous posts:
Part 1: Background Info
Part 2: What to Look For
Part 3: Why it Matters
Despite what Rick Hess says, we won't learn "nothing" from the results of the study (but do read his post, as most of his points are good ones). So what, exactly, will we learn from the results of the study? It's hard to say exactly, but here are some things that we can and cannot learn from the study:
We can learn how individual teachers respond to a financial incentive offered for individual results.
We cannot learn how individual teachers, groups of teachers, or entire schools respond to financial or other types of incentives offered to groups of teachers or schools.
We can learn whether teachers will change their teaching in ways that will raise student test scores in response to an individual financial incentive.
We cannot learn whether teachers will change their teaching in ways that will increase student engagement, critical thinking, creativity or a myriad of other factors in response to an individual financial incentive.
We can learn whether middle school math teachers in Nashville were more likely to switch schools/districts or leave the profession if they were in the control or treatment group.
We cannot learn whether talented people across the country are more likely to become teachers, and subsequently remain in teaching, if there are performance bonuses in place.
We can learn whether, under this particular system, teachers who are "better" as measured by standardized tests across all three years tend to be rewarded on a year-to-year basis.
We cannot learn whether, under performance pay systems, better teachers (as measured in any number of ways) tend to be paid more.
In short, yes, there are rather severe limitations on what we can learn from this one study. But, at the same time, I'd argue that we can learn more from this study than from most others. Despite what Rick Hess, we will not learn "nothing of value," in part because different people value different things. Hess might think that the teacher recruitment/retention aspect of performance pay is the most important, but plenty of others think the incentivizing of effort is the most important.
What does this mean for the interested observer watching from afar? It means that your ears should perk up
A Primer on the Nashville Incentive Pay Experiment, Part 3
Part 3: Why it Matters
Previous posts:
Part 1: Background Info
Part 2: What to Look For
You may have noticed that I'm devoting a fair amount of attention to the results of the Nashville incentive pay experiment that are being released today. Let me take a couple of minutes to explain why.
The first, and most obvious, point is that this is the first randomized field trial evaluating the effectiveness of a merit pay system. The debates to date on whether or not we should use some form of performance pay in school have largely relied on ideology and theory. This will give us the first concrete, empirical, and comprehensive evidence to inform our future policy decisions. Given the importance of merit pay in the national discussion right now, it makes this one of the most important education studies of the decade.
Now, that is not to insinuate that there still won't be a ton of unanswered questions about merit pay after the results of this are digested (no one study is ever enough to close the book on such a wide-ranging topic) but, rather, that we will know significantly more about how merit pay plays out at the ground level after the release of this study.
Or, at least, we certainly hope we will -- especially given that this represents countless hours of effort by dozens of people over the past five years or so . . . and millions of dollars. Things can always be done bigger and better, but there won't be anything bigger and better than this for quite some time (if ever), so expect the results to be bandied about by both sides of the debate for years to come. In other words, expect this study to be the definitive study on how individual teachers respond to financial incentives well into the future.
My next post will address what, exactly, we might learn from the results. It will likely be followed up at an attempt to live-blog the release of the results beginning around 12:30pm central time.
Previous posts:
Part 1: Background Info
Part 2: What to Look For
You may have noticed that I'm devoting a fair amount of attention to the results of the Nashville incentive pay experiment that are being released today. Let me take a couple of minutes to explain why.
The first, and most obvious, point is that this is the first randomized field trial evaluating the effectiveness of a merit pay system. The debates to date on whether or not we should use some form of performance pay in school have largely relied on ideology and theory. This will give us the first concrete, empirical, and comprehensive evidence to inform our future policy decisions. Given the importance of merit pay in the national discussion right now, it makes this one of the most important education studies of the decade.
Now, that is not to insinuate that there still won't be a ton of unanswered questions about merit pay after the results of this are digested (no one study is ever enough to close the book on such a wide-ranging topic) but, rather, that we will know significantly more about how merit pay plays out at the ground level after the release of this study.
Or, at least, we certainly hope we will -- especially given that this represents countless hours of effort by dozens of people over the past five years or so . . . and millions of dollars. Things can always be done bigger and better, but there won't be anything bigger and better than this for quite some time (if ever), so expect the results to be bandied about by both sides of the debate for years to come. In other words, expect this study to be the definitive study on how individual teachers respond to financial incentives well into the future.
My next post will address what, exactly, we might learn from the results. It will likely be followed up at an attempt to live-blog the release of the results beginning around 12:30pm central time.
Monday, September 20, 2010
A Primer on the Nashville Incentive Pay Experiment, Part 2
Part 2: What to Look For
The results from the Nashville incentive pay experiment are due to be released tomorrow (see last week's post for background info on the experiment) -- here are a few things to keep an eye out for in the final report:
Stability of scores
One of the issues with value-added scores has been their high variability from year to year. Researchers were worried about the effects of "statistical noise" and random variations in scores before the start of the experiment. In practical terms, you'll want to know how many teachers received a bonus each year versus how many earned a bonus one or two out of the three years -- and how well teachers who ever earned a bonus did across all three years (e.g. did they earn a bonus for being in the 85th percentile one year but then have a score in the 40th percentile the other two years). In other words, were bonuses going to the same teachers who tended to outperform other teachers, or were they just randomly assigned each year?
What, if anything, did teachers do differently?
To me, this is the most interesting question. Did treatment group teachers report putting more effort into teaching because of the incentive? Did they assign more homework? Did they crack down more on misbehaving students? Did they work harder to get certain students out of their class? Did they focus more on their math class than their other class (some of the teachers taught multiple subjects)? Did they attend more PD? Did they spend more time on test prep? Or did they not bother to change at all? The baseline numbers are going to get all the attention (i.e. did students of treatment group teachers score higher?) but, I think answering this question is far more important -- both because it informs us as to why, or why not, treatment teachers do better and because it gives us valuable insights into how teachers think and how their actions influence student achievement.
Teacher reaction to receipt/non-receipt of bonus
Did teachers who earned a bonus enthusiastically redouble their efforts while those who didn't simply gave up and maybe quit? Or did those who failed to earn a bonus decide to redouble their efforts to make sure they got one the next year while those who earned one became cocky and put their teaching on cruise control? In terms of the numbers, it will be interesting to see if there's any divergence between the scores of first-year winners versus first-year losers (in terms of bonus receipt) -- does one group go up in second years and other down, are both steady, or something else? Of course, if the scores are simply random each year, then we'd expect first-year losers to do better than first-year winners in the 2nd/3rd years because of regression to the mean.
Demographics of winning/losing teachers
Bonuses are not based on a value-added measure that attempts to control for everything and isolate the individual teacher's effect. The only thing the computed score takes into account is prior achievement of the students. So it will be interesting to see if winning teachers taught in better schools, were more experienced, had smaller classes, used a different curriculum, had lower student transience, or other factors that might influence students' gains on the state math test.
Performance in different types of math classes
Were teachers of advanced algebra classes more or less likely to earn bonuses than teachers of remedial math classes? Since the formula for determining bonuses is somewhat simplified, it may be the case that it's a lot easier to get low-performing student to advance x points than high-performing students, especially if there's a ceiling effect. Or it may be the case that some subjects are better aligned with the contents of that year's state test.
Improvement of scores
For merit pay systems to transform our schools, we need teachers to improve -- and continue to improve -- while they're eligible for these bonuses. Why? Let's say that the treatment teachers, as a group, have a score that ranks them in the 50th percentile historically each year. Now let's say that the treatment group teachers average a score in the 60th percentile each year. That would appear to be strong evidence that incentives make teachers better (at least as measured by gains on Tennessee's state tests, anyway), but it would only offer a limited amount of hope for the future because it would indicate only a one-time boost to scores. Let's say another experiment on a professional development system of some sort yields growth instead of simply a step up -- teachers are in the 50th percentile the first year, the 55th the second, the 60th the third, and the 65th the fourth . . . that system would offer more promise for future growth in school success.
Keep your eyes open for some more pre- and post-release analysis of the experiment . . .
The results from the Nashville incentive pay experiment are due to be released tomorrow (see last week's post for background info on the experiment) -- here are a few things to keep an eye out for in the final report:
Stability of scores
One of the issues with value-added scores has been their high variability from year to year. Researchers were worried about the effects of "statistical noise" and random variations in scores before the start of the experiment. In practical terms, you'll want to know how many teachers received a bonus each year versus how many earned a bonus one or two out of the three years -- and how well teachers who ever earned a bonus did across all three years (e.g. did they earn a bonus for being in the 85th percentile one year but then have a score in the 40th percentile the other two years). In other words, were bonuses going to the same teachers who tended to outperform other teachers, or were they just randomly assigned each year?
What, if anything, did teachers do differently?
To me, this is the most interesting question. Did treatment group teachers report putting more effort into teaching because of the incentive? Did they assign more homework? Did they crack down more on misbehaving students? Did they work harder to get certain students out of their class? Did they focus more on their math class than their other class (some of the teachers taught multiple subjects)? Did they attend more PD? Did they spend more time on test prep? Or did they not bother to change at all? The baseline numbers are going to get all the attention (i.e. did students of treatment group teachers score higher?) but, I think answering this question is far more important -- both because it informs us as to why, or why not, treatment teachers do better and because it gives us valuable insights into how teachers think and how their actions influence student achievement.
Teacher reaction to receipt/non-receipt of bonus
Did teachers who earned a bonus enthusiastically redouble their efforts while those who didn't simply gave up and maybe quit? Or did those who failed to earn a bonus decide to redouble their efforts to make sure they got one the next year while those who earned one became cocky and put their teaching on cruise control? In terms of the numbers, it will be interesting to see if there's any divergence between the scores of first-year winners versus first-year losers (in terms of bonus receipt) -- does one group go up in second years and other down, are both steady, or something else? Of course, if the scores are simply random each year, then we'd expect first-year losers to do better than first-year winners in the 2nd/3rd years because of regression to the mean.
Demographics of winning/losing teachers
Bonuses are not based on a value-added measure that attempts to control for everything and isolate the individual teacher's effect. The only thing the computed score takes into account is prior achievement of the students. So it will be interesting to see if winning teachers taught in better schools, were more experienced, had smaller classes, used a different curriculum, had lower student transience, or other factors that might influence students' gains on the state math test.
Performance in different types of math classes
Were teachers of advanced algebra classes more or less likely to earn bonuses than teachers of remedial math classes? Since the formula for determining bonuses is somewhat simplified, it may be the case that it's a lot easier to get low-performing student to advance x points than high-performing students, especially if there's a ceiling effect. Or it may be the case that some subjects are better aligned with the contents of that year's state test.
Improvement of scores
For merit pay systems to transform our schools, we need teachers to improve -- and continue to improve -- while they're eligible for these bonuses. Why? Let's say that the treatment teachers, as a group, have a score that ranks them in the 50th percentile historically each year. Now let's say that the treatment group teachers average a score in the 60th percentile each year. That would appear to be strong evidence that incentives make teachers better (at least as measured by gains on Tennessee's state tests, anyway), but it would only offer a limited amount of hope for the future because it would indicate only a one-time boost to scores. Let's say another experiment on a professional development system of some sort yields growth instead of simply a step up -- teachers are in the 50th percentile the first year, the 55th the second, the 60th the third, and the 65th the fourth . . . that system would offer more promise for future growth in school success.
Keep your eyes open for some more pre- and post-release analysis of the experiment . . .
Friday, September 17, 2010
Today's Random Thoughts
-The results from the Nashville incentive pay experiment are being released on Tuesday, so keep your eyes open . . . and check back here for some more pre- and post-release thoughts.
-Hopefully everybody read this post entitled "Can Exercise Make Kids Smarter?". The short answer: yes, it can. There's a fast-growing body of literature linking academic performance to all sorts of social conditions and environmental factors that I think we should start paying more attention to. In the end, though, the bigger question will be what we do with this knowledge (e.g. figuring out how to get kids to exercise more is probably harder than figuring out how exercise affects brain development, body function, and academic performance).
-There seems to be an awful lot of gnashing of the teeth over Fenty's loss and how it proves that mayors that tackle education reform can't get re-elected. Isn't it instead possible that the objections of the public were more over style than substance? Maybe people like Rhee's reforms but don't like the way she implemented them (or maybe they would just rather she implemented different reforms). I think we'd need a heck of a lot more analysis before we could conclude that Fenty lost simply because he tried to change schools -- and that any other mayor who tries will share a similar fate.
-Hopefully everybody read this post entitled "Can Exercise Make Kids Smarter?". The short answer: yes, it can. There's a fast-growing body of literature linking academic performance to all sorts of social conditions and environmental factors that I think we should start paying more attention to. In the end, though, the bigger question will be what we do with this knowledge (e.g. figuring out how to get kids to exercise more is probably harder than figuring out how exercise affects brain development, body function, and academic performance).
-There seems to be an awful lot of gnashing of the teeth over Fenty's loss and how it proves that mayors that tackle education reform can't get re-elected. Isn't it instead possible that the objections of the public were more over style than substance? Maybe people like Rhee's reforms but don't like the way she implemented them (or maybe they would just rather she implemented different reforms). I think we'd need a heck of a lot more analysis before we could conclude that Fenty lost simply because he tried to change schools -- and that any other mayor who tries will share a similar fate.
Wednesday, September 15, 2010
The Most Surprising Result of the TIME Poll
I'm somewhat skeptical of national polling on most educational issues, since the public doesn't truly understand much of what they're being asked about (heck, half the wonks don't really understand NCLB). So I read a few posts about the TIME poll, bookmarked it, and forgot about it. Finally got around to reading it today . . . there are mixture of results that anybody can use to support their pet reform or ideology, but I think they're all at least fairly close to what we'd expect. Only one question really surprised me:
Talk about flying in the face of recent rhetoric . . . if we took the last 100 editorials and op-eds on education policy from the nation's major newspapers, how many would call for better teachers and how many would call for more involved parents? I can only remember one recent one that sort of called for the latter (Samuelson's op-ed about motviation), so I might have to guess 99 to 1.
Talk about flying in the face of recent rhetoric . . . if we took the last 100 editorials and op-eds on education policy from the nation's major newspapers, how many would call for better teachers and how many would call for more involved parents? I can only remember one recent one that sort of called for the latter (Samuelson's op-ed about motviation), so I might have to guess 99 to 1.
Subscribe to:
Posts (Atom)