Caption: Meme saying “one does not simply claim that the Pro-Truth Pledge is effective without evidence”(Made by blog author)
So you want to know whether the Pro-Truth Pledge (PTP) is effective? Fair enough!
Let’s talk about what effectiveness means in this case. For the PTP, effectiveness involves making a substantial impact on shifting our society toward truthful behavior. For that to be the case, three things need to take place: getting the pledge in front of people for them to consider signing it, then seeing whether people take the pledge when they learn about it, and then finally seeing whether it changes behavior.
The first two are easy to evaluate. The PTP was launched in December 2016, and by the end of the year had just under 5,000 pledge-takers, including over 500 public figures and organizations. A number of prominent public figures have taken the pledge, such as globally-known public intellectuals Peter Singer, Steven Pinker, and Jonathan Haidt, prominent liberal US news personality Ethan Bearman and conservative news personality John B. Wells, owners of media venues like the CEO of Women’s Radio Network Pat Lynch and History News Network founder and chief editor Rick Shenkman, business leaders such as CEO of First City Bank Doug Simson and Managing Director of Essex Lake Group Michael Tyler, values-based leaders such as Episcopal Bishop Pierre Whalon and American Atheist magazine editor Pamela Whissell, and many dozens of politicians on both the left and the right, such as the Republican Christopher Peters and the Democrat Kathie Allen. The PTP has already had positive media coverage in prominent print venues, whether US newspapers such as The Columbus Dispatch, or UK ones such as The Guardian, magazines such as Scientific American, as well as on TV, on radio, and in podcasts and videocasts.
So we know that, when people are exposed to it, a certain proportion take it, including globally-known public figures. Moreover, the media coverage shows that there’s enough interest in the pledge for it to spread. The more difficult question to evaluate is whether taking the pledge changes behavior.
Behavior Change and the Pro-Truth Pledge: Case Studies
We have performed some follow-up conversations with pledge-takers to determine whether the pledge impacted their behaviors. A US Army veteran and member Special Operations community, John Kirbow, took the pledge. He then wrote a blog post about how it impacted him. He notes that “I’ve verbally or digitally passed on bad information numerous times, I am fairly sure, as a result of honest mistakes or lack of vigorous fact checking.” He describes how after taking the pledge, he felt “an open commitment to a certain attitude” to “think hard when I want to play an article or statistic which I’m not completely sold on.” Having taken the Pro-Truth Pledge, he found it “really does seem to change one’s habits,” helping push him both to correct his own mistakes with an “attitude of humility and skepticism, and of honesty and moral sincerity,” and also to encourage “friends and peers to do so as well.”
A Christian pastor and community leader, Lorenzo Neal, took the Pro-Truth Pledge. He related how he “took the Pro-Truth Pledge because I expect our political leaders at every level of government to speak truth and not deliberately spread misinformation to the people they have been elected to serve. Having taken the pledge myself, I put forth the effort to continually gather information validating stories and headlines before sharing them on my social media outlets.”
A former US intelligence officer, who retired from service after 4 decades, took the Pro-Truth Pledge (he prefers to remain anonymous due to his career). He later described how soon after taking the pledge, a piece of news “that played right to my particular political biases hit cable TV and then the Internet and of course my first inclination was to share it as quickly and widely as possible. But then I remembered the pledge I’d signed and put the brakes on. I decided to wait a bit to see how it played out (and boy-howdy am I glad I did.)… As it turned out the story was a complete dud, ‘fake news’ as they say. That experience has led me to be much more vigilant in assessing, and sharing, stories that appeal to my political sensibilities. I now make a much bigger effort to fact-check before I post or share.”
We also engaged in some observations of people engaging in behaviors associated with the pledge. A candidate for Congress, Michael Smith, took the Pro-Truth Pledge. He later posted on his Facebook wall a screenshot of a tweet by Donald Trump criticizing minority and disabled children. As he is a congressional candidate, it got a number of shares. However, after being called out on it, he went and searched Trump’s feed. He could not find the original tweet, and while Trump may have deleted that tweet, the candidate edited his own Facebook post to say that “Due to a Truth Pledge I have taken I have to say I have not been able to verify this post.” He indicated that he would be more careful with future postings.
Caption: Screenshot of Michael Smith’s Facebook post with retraction.
A candidate running for the state house in Arizona, Johny Martin, also took the pledge. He made a misstatement during a public rally, and later posted on Facebook about the misstatement, retracting it and citing the pledge.
In another case, Mark Kauffman, a photographer from New York, shared an article from OccupyDemocrats.com, a site shown by credible fact-checkers used by the PTP to be systematically unreliable. Other pledge-takers, following the behavior of asking people to stop using unreliable sources regardless of the credibility of the article, asked him to withdraw it, and he did so.
Pro-Truth Pledge Impact: Empirical Evaluation
Such case studies of interviews and observations, while illuminating, would be stronger if supported by more systematic quantitative data. Thus, we have conducted a survey evaluation of of pledge-takers to see whether their sharing of information on social media was impacted by the pledge. We decided to target Facebook, as the most popular social media platform: 44 percent of US adults got news via Facebook in 2016.
Our hypothesis was that taking the PTP will impact sharing on Facebook, both when people share news-relevant content themselves on their own Facebook profile, and when they engage in other venues on Facebook, such as Facebook groups or other people’s profiles. Examples of engaging in other venues would include behaviors like asking people to retract incorrect statements, as was the case with pledge-takers asking Michael Smith and Mark Kauffman to retract their statements. To test this hypothesis, we have conducted a study of private citizens who took the PTP and engage actively in sharing news-relevant content on Facebook. Since these people already care about the truth – otherwise, they would presumably not take the pledge – any difference between sharing behaviors before and after taking the pledge can be attributed to finding out about the pledge and taking it.
Method
We had participants fill out Likert scale (1-5) surveys self-reporting their Facebook engagement with news-relevant content on their own profiles and also with other people’s posts and in groups before and after they took the pledge. The specific questions asked were in the form of the following:
- Before you heard about the Pro-Truth Pledge, to what extent did your behavior on your personal Facebook profile align with the Pro-Truth Pledge’s 12 behaviors? Please give an estimate of 1 to 5, with 1 at lowest level of alignment to 5 being full alignment. Lowest alignment means sharing misinformation. Highest alignment means actively fighting lies and promoting truth. Remember that in this question, you are evaluating only your behavior on your personal profile, not your behavior in groups or in response to other people’s posts.
All other questions followed a similar format. We asked a separate question about whether study participants wanted to clarify any aspects of the questions, and no one reported being confused by the questions.
To avoid the Hawthorne effect of study participants being impacted by observation, the study did not evaluate current behavior, but past behavior. We only recruited participants who took the pledge 4 or more weeks ago to fill out the survey, and asked them about their behavior after taking the pledge. Giving them this period also gave people an opportunity to have the immediate impact of taking the pledge fade from their mind, thus enabling an evaluation of the medium-term impact of the PTP on sharing news-relevant content. We recruited participants via Facebook posts and emails to people who took the pledge and were interested in receiving pledge updates soliciting participation in a study about the pledge, and participants were not given any incentives to participate. With these limitations, we were able to secure 24 participants.
This study method was informed by the approaches used by studies of whether honor codes address cheating, which is the most comparable form of intervention to the PTP. Such studies similarly rely on self-reporting by students on whether they have cheated or not cheated. Similarly, studies of whether virginity pledges delay sexual onset similarly reply on self-reporting. Thus, our method of evaluating the PTP faces the same problems faced by those studies: self-reporting and self-selection. Regarding the former, we cannot be certain whether, despite the clarity of the questions to the participants and the clarity of the behaviors outlined in the pledge, the study subjects gave accurate evaluations of themselves. Regarding the latter, there is a possibility that some people who failed to uphold the virginity pledge or the honor code might have chosen to avoid participating in studies of the impact of these interventions. However, given that these studies of honor codes and virginity pledges have been acknowledged as appropriate and influential in the literature despite the potential problems of self-reporting and self-selection, we have chosen to use a similar methodology to test a similar intervention.
Results
Our results show that taking the pledge results in a statistically significant increase in alignment with the behaviors of the pledge, both on one’s own profile on Facebook and when interacting with other people’s posts and in groups. Specifically, on one’s own Facebook profile, the median alignment with the PTP score before taking the PTP is 4 (SD=1.14), and the median alignment score after taking the PTP is 4.5 (SD=0.51). We conducted an Asymptotic Wilcoxon-Pratt Signed-Rank Test to compare PTP alignment on one’s own profile before and after taking the PTP. The null hypothesis H0 for the test states that there is no significant score difference before and after taking the pledge and the alternative hypothesis H1 proposes a significant difference. The results reveal a significant increase of PTP alignment after taking the pledge with a large effect size; z = 6.12, p < 0.000, r = 0.88. Based on the p-values, the null hypotheses H0 can be rejected and the alternative hypotheses H1 is accepted. These results suggest that taking the pledge really does influence alignment on one’s own profile. Figures 1 and 2 represent the results visually.
Figure 1, Individual responses on the survey for PTP alignment on one’s own profile, with 1 at lowest alignment and 5 at highest alignment.
Figure 2, Average of responses on the survey for PTP alignment on one’s own profile, with 1 at lowest alignment and 5 at highest alignment.
For engaging with newsworthy content on other people’s profiles, the median PTP alignment score before taking the Truth Pledge is 3.5 (SD=1.06). The median PTP alignment score after taking the Truth Pledge is 4.5 (SD=0.65). As before, we conducted an Asymptotic Wilcoxon-Pratt Signed-Rank Test to compare alignment in groups and other people’s profiles before and after taking the Truth Pledge. The results show a significant increase of Truth Pledge Alignment after taking the pledge with a large effect size; z = 6.11, p < 0.000, r = 0.88. Again, H0 can be rejected and the alternative hypotheses H1 is accepted. These results suggest that taking the Truth Pledge really does influence Truth Pledge Alignment in groups and on other people’s posts. Figures 3 and 4 represent the results visually.
Figure 3, Individual responses on the survey for PTP alignment in groups and on other people’s profiles, with 1 at lowest alignment and 5 at highest alignment.
Figure 4, Average of responses on the survey for PTP alignment in groups and on other people’s profiles, with 1 at lowest alignment and 5 at highest alignment.
For sharing content on their own profile, 70.83% of participants (17 of 24 respondents) reported an increase of their PTP alignment after taking the PTP, eleven participants increased by one point on the alignment scale, five by two points, one by three points, while the rest maintained the same score. For sharing content in groups and on other people’s walls, again, 70.83% of participants reported an increase of their PTP alignment after taking the PTP, twelve participants increased by one point on the alignment scale, five by two, while the rest maintained their initial score.
These results indicate that taking the PTP indeed significantly improves social media sharing, and thus the PTP is an effective intervention for addressing the scourge of fake news. The results also contradict the hypothesis that all those who take the PTP are already honest and that taking the pledge is simply a way to signal their honesty publicly. If that was the case, there would be no statistically significant increase in alignment with the truth-oriented behaviors in the PTP before and after taking the pledge.
Moreover, if we look at Figures 1 and 3, we see that about half of the survey participants were actually at 3 or below for the quality of their Facebook engagement before taking the pledge, and thus can hardly be called truth-oriented. It is only after taking the pledge that they improved their behavior. In fact, all study participants who scored at 3 or below reported some improvements in their behavior to align more with the pledge after taking it.
This study does not tell us whether the difference in sharing behaviors can be attributed simply to finding out about the pledge, or specifically to taking the pledge. Thus, the actual intervention we are measuring is the combination of “finding out about and taking the pledge” as opposed to the more fine-grained intervention of taking the pledge after finding out about it, or finding out about the pledge without taking it. While it may be an interesting question of whether just finding out about the pledge makes a difference in Facebook sharing behavior, or whether it’s a matter both of finding out about the pledge and taking it, we had no reasonable way of reaching out and recruiting the many people who heard about the pledge without taking it. Moreover, from the perspective of social impact, it makes little difference: if finding out about the pledge and taking it is an effective intervention to decrease sharing misinformation, we can still ethically recommend encouraging people to learn about and take the pledge regardless of our lack of certainty about whether the actual mechanism involves just one or both components.
Overall, the survey results provide solid evidence that people improved the honesty of their behavior on social media because they have heard about and signed the pledge, and there is no reason to believe they would have improved if they did not hear about and sign the pledge. Thus, we can say with relative confidence that voluntarily taking the PTP decreases the spread of misinformation online. By extension, the same finding implies that pledge-takers practice more truthful behavior in other areas of their civic engagement. Further research is needed to determine whether that is indeed the case. We also do not know whether presenting the PTP in a semi-voluntary context, such as when students are presented with an honor code with an implicit expectation that they sign it in order to attend the college of their choice, will maintain the impact of the PTP: further research is needed as well. Likewise, we would also like to conduct research on what would be the impact of just hearing about the pledge without taking it to tease out that aspect of how the pledge works, although it would not substantially affect the policy implications of the pledge itself.
Conclusion
So with this evidence, we can clearly say that the Pro-Truth Pledge is effective in changing our society to be more truthful. The PTP successfully addresses the three components necessary to achieve this goal: it can garner media coverage, a certain percentage of people exposed to it take it, and those people actually do behave in a more truthful manner after taking it. We can confidently say that the remaining challenge is scaling up this project, meaning getting the pledge in front of enough people to get them to sign up and change our society to be more truthful.