Note: In keeping with our commitments made in 2016, we have completed a reanalysis of the data and the conclusions originally reported on this page. The full report on our reanalysis, Which Leaflet is More Effective: A Reanalysis, covers both quantitative and inferential claims made on this page. The code used in our reanalysis is included in the report and may also be downloaded from the Open Science Framework account of Humane League Labs.
Our reanalysis found that, assuming no methodological concerns and based on the data alone, one may reasonably conclude that Something Better is likely more effective as a leaflet than Compassionate Choices. However, given methodological concerns which include the lack of a control group, the presence of social desirability bias and the reliance on self-reported dietary data, a substantial but unquantifiable uncertainty is warranted around this conclusion. We do not recommend that activism strategies related to leafleting be guided by the results of this study alone.
— Humane League Labs [August 11, 2017]
Posted on July 19, 2013
In the fall of 2012, The Humane League carried out a study to measure the impact of passing out vegan leaflets on college campuses. The goal of the study was to see how much diet change is brought about — and how many animals are spared — through college leafleting. Approximately 450 students at two major east coast state universities filled out a survey two to three months after receiving a leaflet. The methodology of the study and the results were published here (basic results) and here (additional findings).
The results suggest that college leafleting is an extremely effective vegan advocacy strategy. About one out of every 50 students who received a leaflet indicated they became vegetarian or pescatarian as a result. Just as importantly, 7% of students said they now ate “a lot less” chicken, a lot fewer eggs, and a lot less dairy as a result of getting the leaflet. 6% ate a lot less fish, and 12% ate a lot less red meat. And one out of every five students said they shared the leaflet with someone else who then began to eat less meat.
After attempting to account for social desirability bias (people saying they are eating less meat when in fact they are not), the data suggested that one farm animal was spared for every two leaflets distributed.
What those reports did not mention, however, was that the study also included a split-test to determine which leaflet was more effective at inspiring diet change among college students: the Something Better leaflet or the Compassionate Choices leaflet.
Something Better appeared to be more significantly effective at inspiring diet change and sparing farm animals.
Among students who had never received either booklet before this study, here are the percentage who received each leaflet and then began eating “a lot less” of each animal product or stopped eating that product entirely.
When the overall results were translated to number of animals spared, Something Better appeared to spare 35% more animals among this group of students. But once social desirability bias is accounted for, the difference between the two booklets grows a lot larger. For example, imagine that one out of every 20 college students will claim they’ve cut out or seriously cut back on chicken, even when they have not, because they think that’s what researchers want to hear. If that were the case, then Something Better would have actually only inspired 10% of students to seriously cut back on or remove chicken (15% saying they did so, minus the 5% who are lying, equals 10% who actually changed). Compassionate Choices would have actually only inspired 5% to cut back or cut out chicken. In this world, Something Better would actually be 100% more effective than Compassionate Choices (10% changed vs. 5% changed) in getting people to cut out or seriously cut back on chicken.
Unfortunately we don’t know the exact level of social desirability bias. In other words, we don’t know exactly what percentage of students will claim to have cut out or seriously cut back on each product even when they did not. So, what do we know? We know that the data suggests Something Better spares at least 35% more animals. In reality, the increase in the number of animals spared is something larger than 35%, perhaps in the 50-100% range.
Next, let’s look at students who had already received a Compassionate Choices leaflet in a previous semester. Among those students, here are the percentage who received each leaflet and then began eating “a lot less” of each animal product or stopped eating it entirely.
Of the 7% who received the Something Better leaflet and began eating “a lot less” chicken, all 7% stopped eating chicken entirely. Of the 6% who received the Compassionate Choices leaflet and began eating “a lot less” chicken, 1% stopped eating chicken entirely.
When the overall results were translated to number of animals spared, Something Better appeared to spare 49% more animals. Once you factor in social desirability bias, the data indicates that among this group of students Something Better spared at least 49% more animals and probably a significantly higher percentage than that.
Obviously, this is an unfair comparison in that students who are receiving the same booklet for the second time are probably less likely to make a change than students receiving an entirely new booklet. So in one sense the comparison here is less meaningful than the comparison shown earlier, the comparison between those who have never received either booklet before.
On the other hand, the comparison here is quite meaningful for college leafleters. Most colleges have been saturated by Compassionate Choices booklets over the past few years. As a result, any animal advocate leafleting on a college campus in the next few years will be leafleting an audience among whom a large percentage has received a Compassionate Choices booklet in the past. The comparison here indicates that for this very large group of students, Something Better is significantly more effective. This greater effectiveness seems to be in part because the booklet is more effective in general, and in part because it is simply a novel booklet.
In carrying out the study, we noticed that many more students claimed to have received a Compassionate Choices that semester than claimed to have received a Something Better leaflet that semester. The numbers should have been nearly identical, since equal numbers of each leaflet were distributed. As a result, probably many students who claimed to have received a Compassionate Choices that semester had actually received one in a prior semester, but were misremembering that fact.
What does this mean for the data in general? If they received a leaflet in a previous semester, the students would have had more time to move towards vegetarian eating, but also more time to backslide away from it. By looking to students who definitely received a leaflet for the first time that semester, we can see whether the results above are accurate or whether they have been skewed.
Freshmen who said they were receiving a Compassionate Choices booklet for the first time almost surely received it that semester, since very few such booklets are distributed to high school students. And students who said they received a Something Better leaflet, and who had not received a Compassionate Choices leaflet before, definitely received the Something Better leaflet that semester, since the booklet did not exist prior to that semester.
The good news — for the reliability of this study — is that when we look to these groups, the results are right in line with the results of our previous two comparisons. In fact, Something Better fared even better than usual among this group. Here are the percentages of students who ate “a lot less” of each product, or stopped eating the product entirely, after getting each leaflet that semester:
When looked at in terms of number of animals spared, students in this group who received a Something Better leaflet spared about 93% more animals than those who received a Compassionate Choices leaflet. This is before accounting for social desirability response bias, which would drive the percentage difference even higher.
Lastly, while the number of vegetarians and pescatarians created by each booklet is far less important than the number of animals spared by all diet changes, it’s worth noting that Something Better was also much more likely to inspire students to go vegetarian or pescatarian. Among all students who took part in the overall survey:
|Said they went vegetarian||1%||0%|
|Said they went pescatarian||3.3%||0.8%|
In summary, the data suggests that among college students Something Better booklets spare at least 35% (and probably closer to 50-100%) more animals than Compassionate Choices booklets. They also seem to inspire many more students to go vegetarian or pescatarian.
It’s important to note that the results in this report aren’t certain, because based on the number of respondents many (though not all) of the differences in diet change were within the margin of error. However the uniformity of the results suggests the data is painting an accurate picture.
In 16 out of the 17 product/diet comparisons above, Something Better performed significantly better, often 50-100% better. And in each key between-group comparison, Something Better inspired students to spare at least 35% (and up to 93%) more animals, even before accounting for social desirability response bias — which would drive those numbers up even higher. Among respondents overall, Something Better spared 94% more animals, even before accounting for social desirability response bias.
Implications For Vegan Advocates
Unless future research suggests otherwise, vegan advocates should favor using the Something Better booklet over the Compassionate Choices booklet when it is possible to do so. Doing so should spare many more farm animals.
|Document (with link)||Description|
|Diet Change Data||Multi-tab Excel file showing the diet change impact of each leaflet among each group|
|Survey questionnaire||pdf document|
|Raw data 1||Corresponding to recipients of Compassionate Choices|
|Raw data 2||Corresponding to recipients of Something Better|
List of Corrections
[July 20, 2017] Table 4 in the original post had incorrectly swapped the location of the quantities 0% and 3.3%. The table now correctly reflects the data: 0% of those who received Compassionate Choices said they went vegetarian and 3.3% of those who received Something Better said they went pescatarian.