I've encountered a number of surprises after switching from teaching to researching, but perhaps none have been more notable than the role of complicated statistical methods in education research. It seems that, historically, education research has been denigrated as non-rigorous but that, currently, the trend is toward more quantitative methods -- much of which is done by economists. Pick up virtually any journal and you'll find an article with statistical tables and strings of Greek letters that 99% of the population won't understand. In some cases this is a good thing, but in some cases I'm not so sure that it's not hurting the field. And the question still remains: why have increasingly complicated statistical models become so commonplace in educational research?
Let me start with an anecdote (a decidedly non-rigorous research method):
When I presented my paper on teacher retention at a conference earlier this month, there were apparently a number of audience members who had not been trained in statistics (I used some fairly basic ones for my analysis). I didn't really receive any criticism about the presentation, and the feedback seemed to indicate that most people didn't really understand the tables I'd presented. Afterwards, one guy walked up to me and said "you lost me on the statistics, so I'll take your word for it."
At the time, I chuckled and continued with the conversation. But, in retrospect, this troubles me. Why should he take my word for it? Just because I used some statistics that he didn't understand? And it got me thinking. Does this happen on a larger scale as well? Are people scared to argue with the methods in these papers because they don't understand them? Do some people take economists and others at their word when they do complicated statistical analyses b/c they simply assume that, since they don't understand them, they must be thorough and correct?
Probably the other most notable thing I've learned is that there is no such thing as a perfect research study -- every single one has significant flaws (at least in the social sciences, I claim ignorance on physicists observing quarks). Part of the motivation for this post was the conversation in which I found myself embroiled on another website (here) . The blog post and comments seem (at least to me) to assume that the results of the study provide a definitive answer -- in part b/c it uses incredibly complicated statistics. The paper is better than most, so I hesitate to use it as an example, but, nonetheless, it has flaws and limitations -- just like any other. And I wonder if the methods were more accessible and understandable if people would be so willing to accept the findings without further discussion.
Sunday, March 30, 2008
Thursday, March 27, 2008
How Does Teacher Retention Affect Schools?
I've been on the road for two weeks now and all the conference presentations are starting to muddle together in my head. As such, I'm just going to talk briefly about the paper I presented yesterday. It's one of the few things that I can keep straight in my head right now.
Here's the premise of the paper:
Lots of people have investigated why teachers leave schools, but I haven't seen much on how schools are affected by teachers leaving. The simplest version of this question would be "does retaining more teachers improve student performance?"
As I've discussed before, the nature of teacher retention is very different in different schools. Since retention rates are notably low in high-poverty urban schools, I chose to focus solely on them. I had some data and was able to compile a dataset of 43 NYC middle schools that had large percentages of poor students (not including magnet, K-8, and some other schools with unreliable statistics). Across these schools, about 40% of teachers were in their first or second year of teaching at their current schools and less than half had a total of 5 or more years of teaching experience.
There was also a moderately strong (r=.44, p<.05) relationship between the average student score on the 8th grade state math test and the percent of teachers who had been teaching for at least two years at that school. In other words, schools with higher rates of teacher retention also had higher student achievement.
Using regression analysis and controlling for race and attendance rate, teacher retention was still significantly related to student achievement. Teacher retention had a fairly sizable effect (effect size of .20) and the model did a pretty good job of explaining the variance in test scores (R-squared of .70).
So, in short, among high-poverty middle schools in NYC, those with higher rates of teacher retention also had higher test scores, even when controlling for other things that influence student test scores. The question that I can't answer with the data is whether better schools make teachers want to stay there more or if more teachers staying in a school improves the school and boosts student achievement (or a little of both). I hope to come closer to answering this question in the future.
Here's the premise of the paper:
Lots of people have investigated why teachers leave schools, but I haven't seen much on how schools are affected by teachers leaving. The simplest version of this question would be "does retaining more teachers improve student performance?"
As I've discussed before, the nature of teacher retention is very different in different schools. Since retention rates are notably low in high-poverty urban schools, I chose to focus solely on them. I had some data and was able to compile a dataset of 43 NYC middle schools that had large percentages of poor students (not including magnet, K-8, and some other schools with unreliable statistics). Across these schools, about 40% of teachers were in their first or second year of teaching at their current schools and less than half had a total of 5 or more years of teaching experience.
There was also a moderately strong (r=.44, p<.05) relationship between the average student score on the 8th grade state math test and the percent of teachers who had been teaching for at least two years at that school. In other words, schools with higher rates of teacher retention also had higher student achievement.
Using regression analysis and controlling for race and attendance rate, teacher retention was still significantly related to student achievement. Teacher retention had a fairly sizable effect (effect size of .20) and the model did a pretty good job of explaining the variance in test scores (R-squared of .70).
So, in short, among high-poverty middle schools in NYC, those with higher rates of teacher retention also had higher test scores, even when controlling for other things that influence student test scores. The question that I can't answer with the data is whether better schools make teachers want to stay there more or if more teachers staying in a school improves the school and boosts student achievement (or a little of both). I hope to come closer to answering this question in the future.
Wednesday, March 26, 2008
Discipline: The "Dirty Little Secret?"
I often say that I spent two years trying to teach in the Bronx. Why trying? Because discipline problems in my school were so severe that it was sometimes hard to believe I was teaching much at all. I have no evidence on exactly how typical my school was, but it was abundantly clear that discipline issues were, far and away, the biggest problem.
I spent most of the day in a training session on how to use international datasets rather than attending sessions, so I was planning to go to sleep and not bore anybody with the details, but I just happened to notice Diane Ravitch's latest post right before turning out the lights.
Ravitch basically writes that discipline problems are a major problem in the United States b/c kids don't respect adults and that other countries don't have the same problem (she, of course, says this more eloquently). This immediately stirred a number of thoughts:
1. Based on my experiences, I couldn't agree more. Discipline is a major hurdle for a number of schools. It is much more important than standards or curriculum in these schools, but seems to be researched far less frequently. Is discipline not viewed as a serious problem by academics b/c it's limited to only some schools, b/c it's seen as more of an issue for the popular press, or something else?
2. Based on my research, I cannot uphold her claims. I've found zero evidence that the U.S. is an outlier in terms of discipline problems. There are, however, at least two major problems with my research so far: 1.) The international data on discipline is far from conclusive and 2.) I just found out today that the TIMSS international assessment does not use a nationally representative sample of teachers and, therefore, saying x% of teachers say that behavior is a major problem is not a valid statistic (I have to look at how many students have teachers who said this instead).
3. I wonder how widespread this problem is and how different it is from previous generations. The first question is somewhat easily answered while I'm not sure that it's possible to obtain an answer to the second. The U.S. did stand out in one way from other countries in my research: there was a stronger relationship between problems a principal reported in a school and the SES of the students (schools with poorer students reported both more frequent and more severe problems) than in any other country. I know discipline was the major issue in my school. I know discipline is the major issue in other schools where friends have taught. But I'm unsure how much of an issue discipline is across the country. Do adults always think that kids are less respectful than they should be, or do we have a real problem across the country?
p.s. I stole "dirty little secret" from a commenter on the Ravitch blog entry. I've used similar terminology in the past, but am too tired to think of exactly what it was. I may have more to say on this topic tomorrow.
I spent most of the day in a training session on how to use international datasets rather than attending sessions, so I was planning to go to sleep and not bore anybody with the details, but I just happened to notice Diane Ravitch's latest post right before turning out the lights.
Ravitch basically writes that discipline problems are a major problem in the United States b/c kids don't respect adults and that other countries don't have the same problem (she, of course, says this more eloquently). This immediately stirred a number of thoughts:
1. Based on my experiences, I couldn't agree more. Discipline is a major hurdle for a number of schools. It is much more important than standards or curriculum in these schools, but seems to be researched far less frequently. Is discipline not viewed as a serious problem by academics b/c it's limited to only some schools, b/c it's seen as more of an issue for the popular press, or something else?
2. Based on my research, I cannot uphold her claims. I've found zero evidence that the U.S. is an outlier in terms of discipline problems. There are, however, at least two major problems with my research so far: 1.) The international data on discipline is far from conclusive and 2.) I just found out today that the TIMSS international assessment does not use a nationally representative sample of teachers and, therefore, saying x% of teachers say that behavior is a major problem is not a valid statistic (I have to look at how many students have teachers who said this instead).
3. I wonder how widespread this problem is and how different it is from previous generations. The first question is somewhat easily answered while I'm not sure that it's possible to obtain an answer to the second. The U.S. did stand out in one way from other countries in my research: there was a stronger relationship between problems a principal reported in a school and the SES of the students (schools with poorer students reported both more frequent and more severe problems) than in any other country. I know discipline was the major issue in my school. I know discipline is the major issue in other schools where friends have taught. But I'm unsure how much of an issue discipline is across the country. Do adults always think that kids are less respectful than they should be, or do we have a real problem across the country?
p.s. I stole "dirty little secret" from a commenter on the Ravitch blog entry. I've used similar terminology in the past, but am too tired to think of exactly what it was. I may have more to say on this topic tomorrow.
Tuesday, March 25, 2008
What Does Alternative Certification Say About the Value of Traditional Certification?
In my last post I mentioned debate over alt cert programs. Allow me to elaborate on one strand of the debate.
It is my impression that alt cert programs were originally started as a way to fill vacancies in hard to staff schools. To generalize, the problem was that not enough certified teachers were willing to teach in certain places, but prospective (uncertified) teachers were -- but, at the same time, were not willing to go through the whole certification process. "Alternative Certification," then, allowed competent but uncertified individuals a different, more acceptable, route to certification and also enabled districts and schools to fill vacancies. It's a rational compromise.
Both academia and the press frequently discuss the pluses and minuses of these programs. One of the most frequent arguments against (at least some of) them is that, though they are better than the status quo, they are not the ideal solution -- with some saying they're really more of a band-aid. On the opposite side, many argue that alternatively teachers are just as effective as regularly certified teachers (research has usually found either small or no benefits to certification depending on how it's defined) and sometimes take the opportunity to impugn certification programs and schools of education.
A number of students in my program (including myself) taught through alt cert programs, so they often come up in conversation. One issue that's raised is the effect that alt cert programs have on regular certification programs and public perception of them.
I am wholly unsure to what degree the people who run alt cert programs (and I'm sure there is a lot of difference on this) are out to prove that regular certification is useless. I get the feeling that most programs do not explicitly make this case. It seems to me that the programs are more focused on attracting qualified individuals than making political statements. But whether or not they explicitly criticize schools of education, the mere presence of these programs ultimately implies that certification is not important. In other words, they might not be trying to offend teachers who went through regular certification, but it's understandable if these teachers are offended.
For example, the NYC Teaching Fellows (my former program) regularly blankets subway cars with idealistic ads in order to attract applicants. The ads imply (whether or not they are intended to) that anybody who is smart and works hard can be a good teacher no matter their background and, therefore, that work ethic and intelligence matter more than training.
I'm not saying that there isn't some merit to the statement, but imagine if there was an alternative med school program and anybody who was smart and willing to work hard could be in the operating room after one summer of training. Regardless of whether the program fails or succeeds, its mere presence implies that it's not hard to gain the same expertise as doctors who went to regular medical school.
I'm not saying that this is necessarily good or bad (like most things, it's probably some of each), but I've noticed that this part often escapes people and I find it interesting.
It is my impression that alt cert programs were originally started as a way to fill vacancies in hard to staff schools. To generalize, the problem was that not enough certified teachers were willing to teach in certain places, but prospective (uncertified) teachers were -- but, at the same time, were not willing to go through the whole certification process. "Alternative Certification," then, allowed competent but uncertified individuals a different, more acceptable, route to certification and also enabled districts and schools to fill vacancies. It's a rational compromise.
Both academia and the press frequently discuss the pluses and minuses of these programs. One of the most frequent arguments against (at least some of) them is that, though they are better than the status quo, they are not the ideal solution -- with some saying they're really more of a band-aid. On the opposite side, many argue that alternatively teachers are just as effective as regularly certified teachers (research has usually found either small or no benefits to certification depending on how it's defined) and sometimes take the opportunity to impugn certification programs and schools of education.
A number of students in my program (including myself) taught through alt cert programs, so they often come up in conversation. One issue that's raised is the effect that alt cert programs have on regular certification programs and public perception of them.
I am wholly unsure to what degree the people who run alt cert programs (and I'm sure there is a lot of difference on this) are out to prove that regular certification is useless. I get the feeling that most programs do not explicitly make this case. It seems to me that the programs are more focused on attracting qualified individuals than making political statements. But whether or not they explicitly criticize schools of education, the mere presence of these programs ultimately implies that certification is not important. In other words, they might not be trying to offend teachers who went through regular certification, but it's understandable if these teachers are offended.
For example, the NYC Teaching Fellows (my former program) regularly blankets subway cars with idealistic ads in order to attract applicants. The ads imply (whether or not they are intended to) that anybody who is smart and works hard can be a good teacher no matter their background and, therefore, that work ethic and intelligence matter more than training.
I'm not saying that there isn't some merit to the statement, but imagine if there was an alternative med school program and anybody who was smart and willing to work hard could be in the operating room after one summer of training. Regardless of whether the program fails or succeeds, its mere presence implies that it's not hard to gain the same expertise as doctors who went to regular medical school.
I'm not saying that this is necessarily good or bad (like most things, it's probably some of each), but I've noticed that this part often escapes people and I find it interesting.
An Urban Teaching Corps?
I was all set to go to bed when I noticed this post on the eduwonkette blog. Given that I was less than satisfied with my last two posts (I was too busy to do much more than regurgitate what happened in those sessions) I felt pressured to write something a little better for anybody clicking here for the first time. So I clicked on the other link provided in that posting and found an interesting idea that Jennifer Steinberger Pease wrote about in EdWeek.
She writes about an acquaintance who wants to apply to Teach For America (TFA), but will already be certified and have a master's degree and, therefore, is not really TFA material (I checked TFA's website to see if they would accept such a candidate, but their description didn't answer the question in either direction). She feels that this teacher-to-be would benefit enormously from, and be ideal for, TFA, but sees no other comparable option available for certified teachers. She, therefore, proposes an urban teaching corps similar to TFA -- but for certified teachers.
I've thought (and debated) a lot about the pluses and minuses of TFA and other alternative certification programs but, honestly, such a scenario had never occurred to me (nor had such a solution). Jennifer does seem to have found quite the gap in the current system. I've had many discussions about whether TFA and other programs imply that teacher certification is meaningless, but never really thought about the TFA-quality people who are already certified. I'm not sure that there are quite as many people who want to teach in high-poverty schools but are unable to as she seems to believe, but I don't doubt that more would teach in these schools if they could enter a TFA-like program.
I can see only two major holes in her plan:
1. TFA may, in fact, take certified teachers -- I simply don't know. And if not them, many other similar programs might (The New Teacher Project runs a number of sister programs). I taught with the NYC Teaching Fellows (one of those sister programs) and I know that other people in the program had previous teaching experience (though I'm unsure if any were certified).
2. High-Poverty schools are not only in urban areas. Accordingly, TFA is not only in urban areas (I know two former TFA members that taught in rural areas in the Mississippi Delta and along the Texas border). So I would propose either changing "urban" to "high-poverty" or adding a "rural" teaching corps as well.
Normally new ideas for education have to be taken with a large grain of salt -- it seems that there's always a strong reason that they'll never happen and/or a large downside. I see neither of these with this idea. The easiest solution (but not necessarily the best), of course, would be for alt cert programs to simply start accepting certified teachers. Either way, this idea makes sense to me.
She writes about an acquaintance who wants to apply to Teach For America (TFA), but will already be certified and have a master's degree and, therefore, is not really TFA material (I checked TFA's website to see if they would accept such a candidate, but their description didn't answer the question in either direction). She feels that this teacher-to-be would benefit enormously from, and be ideal for, TFA, but sees no other comparable option available for certified teachers. She, therefore, proposes an urban teaching corps similar to TFA -- but for certified teachers.
I've thought (and debated) a lot about the pluses and minuses of TFA and other alternative certification programs but, honestly, such a scenario had never occurred to me (nor had such a solution). Jennifer does seem to have found quite the gap in the current system. I've had many discussions about whether TFA and other programs imply that teacher certification is meaningless, but never really thought about the TFA-quality people who are already certified. I'm not sure that there are quite as many people who want to teach in high-poverty schools but are unable to as she seems to believe, but I don't doubt that more would teach in these schools if they could enter a TFA-like program.
I can see only two major holes in her plan:
1. TFA may, in fact, take certified teachers -- I simply don't know. And if not them, many other similar programs might (The New Teacher Project runs a number of sister programs). I taught with the NYC Teaching Fellows (one of those sister programs) and I know that other people in the program had previous teaching experience (though I'm unsure if any were certified).
2. High-Poverty schools are not only in urban areas. Accordingly, TFA is not only in urban areas (I know two former TFA members that taught in rural areas in the Mississippi Delta and along the Texas border). So I would propose either changing "urban" to "high-poverty" or adding a "rural" teaching corps as well.
Normally new ideas for education have to be taken with a large grain of salt -- it seems that there's always a strong reason that they'll never happen and/or a large downside. I see neither of these with this idea. The easiest solution (but not necessarily the best), of course, would be for alt cert programs to simply start accepting certified teachers. Either way, this idea makes sense to me.
Monday, March 24, 2008
More on Alternative Schools
The final session that I attended was a symposium of researchers from CA that investigated alternative schools (previous post), which Milbrey McLaughlin said were essentially "invisible."
Alternative schools in CA started around 1917 when the Smith-Hughes Act established federal funding for continuation high schools which, at that time, were designed to provide vocational training to people with other jobs. They eventually morphed into places for "over-aged, under-credited" students to go in a last ditch attempt to avoid dropping out. Today, the 520 continuation high schools in CA vary widely in every way; from pedagogy to the students they enroll.
Among the largest challenges these schools face (other than that most people are unaware of them) are that about half of students enroll for less than 90 days, that they often serve as dumping grounds for both unwanted students and teachers, and that they are often last in line for funding. Indeed, one superintendent told the researchers that it was acceptable for 10% of the students to fail and end up at the local alternative school b/c nobody would notice.
It is my perception that alternative schools are usually started to remove "problem" kids from classrooms in order to better facilitate learning for the rest and/or to "fix" these problem kids. According to the panel, the most successful schools were ones who tried to fix the school environment to suit the kids instead of fixing the kids so that they could fit in a typical school environment.
McLaughlin said that the system was basically "Balkanized" -- there was little communication with other county services, everybody seemed to envision different missions for the schools, etc. What they did find, however, was that the most successful schools were led by "supermen/women" who treated their job as a calling and worked tirelessly to make their school the best place possible.
The work of some of those individuals impresses me, but I do wonder if that is really replicable or scalable -- in other words, if there are enough people willing to sacrifice their lives to these schools in order to run all alternative schools in the country. It's the same question confronting KIPP and other successful schools. Many of them are built on the backs of supermen/women who work tirelessly for their schools and essentially sacrifice years of their life to make their school successful. I am in awe of what these people do, but is it really a solution for the nation? Can we find enough qualified people to treat schools as missionary work to make all schools uber-successful?
Alternative schools in CA started around 1917 when the Smith-Hughes Act established federal funding for continuation high schools which, at that time, were designed to provide vocational training to people with other jobs. They eventually morphed into places for "over-aged, under-credited" students to go in a last ditch attempt to avoid dropping out. Today, the 520 continuation high schools in CA vary widely in every way; from pedagogy to the students they enroll.
Among the largest challenges these schools face (other than that most people are unaware of them) are that about half of students enroll for less than 90 days, that they often serve as dumping grounds for both unwanted students and teachers, and that they are often last in line for funding. Indeed, one superintendent told the researchers that it was acceptable for 10% of the students to fail and end up at the local alternative school b/c nobody would notice.
It is my perception that alternative schools are usually started to remove "problem" kids from classrooms in order to better facilitate learning for the rest and/or to "fix" these problem kids. According to the panel, the most successful schools were ones who tried to fix the school environment to suit the kids instead of fixing the kids so that they could fit in a typical school environment.
McLaughlin said that the system was basically "Balkanized" -- there was little communication with other county services, everybody seemed to envision different missions for the schools, etc. What they did find, however, was that the most successful schools were led by "supermen/women" who treated their job as a calling and worked tirelessly to make their school the best place possible.
The work of some of those individuals impresses me, but I do wonder if that is really replicable or scalable -- in other words, if there are enough people willing to sacrifice their lives to these schools in order to run all alternative schools in the country. It's the same question confronting KIPP and other successful schools. Many of them are built on the backs of supermen/women who work tirelessly for their schools and essentially sacrifice years of their life to make their school successful. I am in awe of what these people do, but is it really a solution for the nation? Can we find enough qualified people to treat schools as missionary work to make all schools uber-successful?
The Costs and Benefits of Schooling
The first session I attended at AERA was an overview of a recent book, The Price We Pay. The book was published by a team of noted economists as an attempt to quantify the benefits of schooling to society and conduct a cost-benefit analysis of some education reforms with proven track records.
They concluded that a person who drops out of high school pays about $100,000 less in taxes (in today's dollars) over the course of their lifetime than does a person who completes high school (only high school, not college) and that, overall, each high school dropout costs the country about $209,000 in lost taxes, welfare, court costs, etc.
Given this figure, they look at how much various interventions cost per additional student that graduates from high school and find that most cost about 1/3 of the amount that society benefits from their graduation. I didn't read the book -- I only heard a 90 minute summary of it -- so I'm not sure exactly what they take into account, but two things that would bias these estimates jumped into my head.
1.) They only look at the additional benefits from the students who graduate from high school that otherwise would not have. What about the students who attended college or graduated from college that otherwise would not have? Taking this into account might mean that their estimates were biased downward -- that there are more benefits than they thought to the interventions.
2.) A shift of a couple percentage points probably wouldn't affect the benefits of a high school education much, but a large increase likely would. If, say, half as many students dropped out each year then it's likely that the benefits of graduating from high school would be smaller. This would mean that their estimates were biased upward -- that the benefits of the interventions were not as large as they thought.
Henry Levin assured a questioner who raised a similar issue that they had taken a lot of things into account and that their estimates were very complex, so it's quite likely that these concerns are addressed in the book. Especially without having read the book I hesitate to believe that these estimates can possibly be very precise, but I think there is some value to them nonetheless. I think a financial analysis of the costs and benefits of school interventions is valuable information to have before making a decision. That said, Henry Levin made sure to emphasize at the very beginning that the primary reason for helping the poorest children is a moral one rather than a financial one.
Also: perhaps the most notable tidbit I gathered from the session was the projection that in 2020 the workforce will be less educated than the workforce in 2000. I had always assumed that the number of people attending college was and will continue to steadily increase, but I had noticed in previous research that the % of 25-29 year olds with a bachelor's degree in 2006 is actually lower than it was in 2000. Given the changing demographics of the country, Marta Tienda concludes that this trend will continue into the future.
They concluded that a person who drops out of high school pays about $100,000 less in taxes (in today's dollars) over the course of their lifetime than does a person who completes high school (only high school, not college) and that, overall, each high school dropout costs the country about $209,000 in lost taxes, welfare, court costs, etc.
Given this figure, they look at how much various interventions cost per additional student that graduates from high school and find that most cost about 1/3 of the amount that society benefits from their graduation. I didn't read the book -- I only heard a 90 minute summary of it -- so I'm not sure exactly what they take into account, but two things that would bias these estimates jumped into my head.
1.) They only look at the additional benefits from the students who graduate from high school that otherwise would not have. What about the students who attended college or graduated from college that otherwise would not have? Taking this into account might mean that their estimates were biased downward -- that there are more benefits than they thought to the interventions.
2.) A shift of a couple percentage points probably wouldn't affect the benefits of a high school education much, but a large increase likely would. If, say, half as many students dropped out each year then it's likely that the benefits of graduating from high school would be smaller. This would mean that their estimates were biased upward -- that the benefits of the interventions were not as large as they thought.
Henry Levin assured a questioner who raised a similar issue that they had taken a lot of things into account and that their estimates were very complex, so it's quite likely that these concerns are addressed in the book. Especially without having read the book I hesitate to believe that these estimates can possibly be very precise, but I think there is some value to them nonetheless. I think a financial analysis of the costs and benefits of school interventions is valuable information to have before making a decision. That said, Henry Levin made sure to emphasize at the very beginning that the primary reason for helping the poorest children is a moral one rather than a financial one.
Also: perhaps the most notable tidbit I gathered from the session was the projection that in 2020 the workforce will be less educated than the workforce in 2000. I had always assumed that the number of people attending college was and will continue to steadily increase, but I had noticed in previous research that the % of 25-29 year olds with a bachelor's degree in 2006 is actually lower than it was in 2000. Given the changing demographics of the country, Marta Tienda concludes that this trend will continue into the future.
The Calm Before the Storm
Noon today marks the first session of the 89th annual conference of the American Educational Research Association (AERA). Close to 20,000 educational researchers (and people interested in education research) will descend on New York City for this year's conference. Through Friday there will be around 120 concurrent sessions every 90 minutes or so. That means that most days there is time to attend 4 or 5 sessions out of about 500 or so.
As with any conference this size there will be sessions that are phenomenal and sessions that are horrible. The trick is to pick the right ones. Given the large selection, there are usually at least a half-dozen sessions centered on topics that interest me and involve somebody who I've read, so I'm hoping I don't pick too many duds this year (I had pretty good luck last year).
I'll be posting anything that I find extraordinarily important/interesting, but I don't know when or how since the conference is hosted by a number of hotels -- all of which likely feel the need to charge extra for internet access despite charging exorbitantly just to stay there.
As with any conference this size there will be sessions that are phenomenal and sessions that are horrible. The trick is to pick the right ones. Given the large selection, there are usually at least a half-dozen sessions centered on topics that interest me and involve somebody who I've read, so I'm hoping I don't pick too many duds this year (I had pretty good luck last year).
I'll be posting anything that I find extraordinarily important/interesting, but I don't know when or how since the conference is hosted by a number of hotels -- all of which likely feel the need to charge extra for internet access despite charging exorbitantly just to stay there.
Thursday, March 20, 2008
More on Inequality and National Achievement
Douglas Willms gave the keynote address last night here at CIES and spoke about increasing both equity and achievement ("raising and leveling the learning bar"). For every country that took part in two international assessments (PISA and PIRLS) he constructed a "learning bar" that showed the relationship between socio-economic status (SES) and achievement. SES was a significant predictor of achievement in every single country, but the slope of the line is different across nations.
For example, Canadians, on average, outscored Americans. But the wealthiest Americans outscored the wealthiest Canadians by a bit (meaning the poorest Canadians outscored the poorest Americans by a lot). This means that the American "learning bar" has a steeper slope -- that SES matters more in the U.S. than it does in Canada. The report of most of the results he discussed can be found here.
He makes a distinction between "raising" this learning bar -- meaning that everybody scores higher -- and "leveling" the learning bar -- meaning that low-SES students improve more than high-SES students, the line becomes flatter, and there is less of a gap between rich and poor. Here's what I found most interesting: his findings dovetail nicely with mine -- countries that scored higher overall also had lower gaps between rich and poor. Additionally, higher scoring countries have distributions that are less negatively skewed -- meaning that there are fewer very low scoring students. In other words, there doesn't seem to be a tradeoff between "raising" and "leveling" the "learning bar" -- or between aiming for high achievement and high levels of equality.
For example, Canadians, on average, outscored Americans. But the wealthiest Americans outscored the wealthiest Canadians by a bit (meaning the poorest Canadians outscored the poorest Americans by a lot). This means that the American "learning bar" has a steeper slope -- that SES matters more in the U.S. than it does in Canada. The report of most of the results he discussed can be found here.
He makes a distinction between "raising" this learning bar -- meaning that everybody scores higher -- and "leveling" the learning bar -- meaning that low-SES students improve more than high-SES students, the line becomes flatter, and there is less of a gap between rich and poor. Here's what I found most interesting: his findings dovetail nicely with mine -- countries that scored higher overall also had lower gaps between rich and poor. Additionally, higher scoring countries have distributions that are less negatively skewed -- meaning that there are fewer very low scoring students. In other words, there doesn't seem to be a tradeoff between "raising" and "leveling" the "learning bar" -- or between aiming for high achievement and high levels of equality.
Wednesday, March 19, 2008
National Achievement and Inequality
I'm in New York City right now attending the meetings of the Comparative and International Education Society (CIES). I've been mostly hiding away in my room desperately trying to prepare my presentation for this afternoon (Tuesday). As such, I do not yet have much of interest to report other than about my paper.
My paper did not live up to my expectations for it, but I think my presentation made sense and I found one interesting thing. The basic motivation for the paper was to find out why some countries have a small spread in achievement and some countries have a large spread in achievement. In the end, I couldn't really find much that looked like compelling predictors for whether achievement in countries would be more or less spread out.
But I did find one thing that surprised me. The last thing that I checked on, mostly out of curiosity since it was really only tangentially related to my topic, was whether higher performing countries had more or less equality. I had a couple different measures of variation in achievement, and I compared them to the median score on the TIMSS (an international assessment involving about 50 countries the last time) and found a really strong correlation between equality and achievement (about .8, p<.001 for you stats nerds), meaning that, within the TIMSS countries, that higher achieving countries were distinctly more equal than lower achieving countries. When I compared performance on TIMSS to spread on PISA (another international assessment) the relationship still held and was moderately strong (about .4, p<.001).
The strength of the relationship was of a level that one just doesn't find while doing research, so I was sure I was doing something wrong, but nobody has given me reason to think that this isn't true and I can't think of any.
I don't know what this would look like for different assessments, different years, different subjects, and different ways of measuring variance (or spread, or inequality, or whatever you want to call it), but it's a result that could potentially be meaningful. It's at least as possible, of course, that the result is either meaningless or won't hold up with other data, but I think it's worth further investigation.
As of this moment, I'm seeing that more equality=higher achievement and wondering whether that means what one might assume it means.
My paper did not live up to my expectations for it, but I think my presentation made sense and I found one interesting thing. The basic motivation for the paper was to find out why some countries have a small spread in achievement and some countries have a large spread in achievement. In the end, I couldn't really find much that looked like compelling predictors for whether achievement in countries would be more or less spread out.
But I did find one thing that surprised me. The last thing that I checked on, mostly out of curiosity since it was really only tangentially related to my topic, was whether higher performing countries had more or less equality. I had a couple different measures of variation in achievement, and I compared them to the median score on the TIMSS (an international assessment involving about 50 countries the last time) and found a really strong correlation between equality and achievement (about .8, p<.001 for you stats nerds), meaning that, within the TIMSS countries, that higher achieving countries were distinctly more equal than lower achieving countries. When I compared performance on TIMSS to spread on PISA (another international assessment) the relationship still held and was moderately strong (about .4, p<.001).
The strength of the relationship was of a level that one just doesn't find while doing research, so I was sure I was doing something wrong, but nobody has given me reason to think that this isn't true and I can't think of any.
I don't know what this would look like for different assessments, different years, different subjects, and different ways of measuring variance (or spread, or inequality, or whatever you want to call it), but it's a result that could potentially be meaningful. It's at least as possible, of course, that the result is either meaningless or won't hold up with other data, but I think it's worth further investigation.
As of this moment, I'm seeing that more equality=higher achievement and wondering whether that means what one might assume it means.
Subscribe to:
Posts (Atom)