Incentives and educational outputs

I've blogged previously about Rolan Fryer's interesting work on the use of financial incentives to improve students' test scores. The short version is that tying rewards directly to grades doesn't seem to work, but tying rewards to some 'inputs' that might lead to better grades (good attendance record, number of books read) can have a positive impact. I personally think his previous work fits ok with a motivation crowding/over-justification view of reward and motivation, since if you're, for example, paying kids to turn up to school then you're probably dealing with low motivation in the first place.

Last night I came across a paper published last where Fryer and others look at the use of financial incentives for teachers with an important behavioural tweak. Here's the abstract:

Domestic attempts to use financial incentives for teachers to increase student achievement have been ineffective. In this paper, we demonstrate that exploiting the power of loss aversion—teachers are paid in advance and asked to give back the money if their students do not improve sufficiently—increases math test scores between 0.201 (0.076) and 0.398 (0.129) standard deviations. This is equivalent to increasing teacher quality by more than one standard deviation. A second treatment arm, identical to the loss aversion treatment but implemented in the standard fashion, yields smaller and statistically insignificant results. This suggests it is loss aversion, rather than other features of the design or population sampled, that leads to the stark differences between our findings and past research.

The first thing to note about this is that there's a clear win here for a more behaviourally-informed approach to the use of incentives, as the conclusion is that loss aversion has a very strong effect. So pyschology matters after all... In fact early in the report there is another section worth flagging up:

To increase teacher productivity, there is growing enthusiasm among policy makers for initiatives that tie teacher incentives to the achievement of their students. At least ten states and many more school districts have implemented teacher incentive programs in an effort to increase student achievement (Fryer forthcoming). Yet, the empirical evidence on the effectiveness of such programs is ambivalent. In developing countries where the degree of teacher professionalism is extremely low and absenteeism is rampant, field experiments that link pay to teacher performance are associated with substantial improvements in student test scores (Duflo et al. (forthcoming); Glewwe et al. 2010; Muralidharan and Sundararaman (forthcoming)). On the other hand, the only two field experiments conducted in the United States have shown small, if not negative, treatment effects (Springer et al. 2010, Fryer (forthcoming)).

This, to me, is not really surprising. If motivation is low, incentives (financial or other) may well be able to push performance up. But where motivation is already there (eg teachers earning a decent salary and who believe in what they are doing) I can't see much of a role for performance-related pay.

However, Fryer et al's paper put an important new spin on this. Apparently you can make performance-related reward work for teachers, if you make it sting. Because let's be clear what we are talking about here - the theory of loss aversion tells us that we 'feel' losses about twice as much as we feel gains. The unpleasant feeling generated by a loss is twice as strong as the pleasant feeling generated by a gain, no doubt for some important evolutionary reason. So this research seems to show that we can get a slightly better result out of teachers by, basically, making them feel bad when their students don't do well. Or, in reality, we make them feel worse - I assume quite a few teachers don't feel great when their students don't do as well as they could.

Is this a reasonable way to pay teachers, or indeed anyone? It's certainly unusual. The only comparable thing I can think of is clawback. But, as far as I am aware, this usually applies to performance-related rewards that have already been made which turn out to have been based on performance that was either illusory or deliveered through unsustainable or unethical behaviour. I'm not aware of people in the City being awarded an upfront bonus and then losing a certain element of it if they don't hit their targets.

I have no doubt that a loss aversion based incentive system would have a strong behavioural pull, and for exactly that reason I would be nervous about it being applied in the City, where gaming the rules and a willingness to cut ethical corners to hit targets is... ahem... not unknown. Could it have a similar effect in the educational sector?

But more generally, this approach to paying teachers (or indeed anyone, really) simply feels really wrong to me. Holding out a reward and then taking it away again if you don't get the performance you want seems to me better suited to dog training than education. It is quite clearly an exercise in behavioural control, and I expect it would immediately be seen as such, and therefore be very controversial if ever actually put into practice. Clearly everyone wants teachers to teach well, but just because we might get better results by using financial incentives in this way doesn't mean we should (it's notable that the one example of a loss aversion based bonus scheme cited in the paper is in a Chinese factory). I think it has the potential to do a great deal of damage.

One final thought, the PwC research into executive incentive pay has an interesting element to it. PwC suggest that most execs (like the rest of us) don't really like variable pay, and the implication is that, therefore, actually existing executive pay is fudged to take account of this. I suspect if loss aversion based incentives ever become 'a thing' something similar would happen. After causing an initial massive stink they would be gradually reformed till they softened the behavioural bits that a) made people upset and b) were the sole reason for introducing the schemes in the first place.