Experimentally Testing the Effectiveness of Human Rights Treaties

I. Introduction

International human rights law is a field concerned with causality. While scholars in other fields argue about how laws can be changed to maximize their effectiveness, scholars of international human rights law still regularly debate whether major international agreements have had any effect on state behavior. Some scholars have examined empirical records and concluded that under the right conditions, the ratification of human rights treaties is associated with improved human rights practices,1 but others have found that the ratification of human rights treaties does not have any influence on the likelihood that states will violate human rights.2

One thing that scholars agree on, however, is that if international human rights treaties were to change state behavior, it would not be because of the usual mechanism that drives compliance with international law: reciprocity.3 This is because one state’s failure to respect the rights of its citizens cannot be corrected by another state reciprocally violating the rights of its own citizens. As a result, scholars have examined other mechanisms—including domestic politics, empathetic enforcement by powerful states, pressure from international treaty bodies, and the influence of non-governmental organizations—that may lead ratification of human rights agreements to result in changes to state behavior. After over a decade of empirical research on the effectiveness of human rights treaties, however, there is disagreement on whether each of these alternative mechanisms leads to improved rights practices.

This lack of consensus is in part due to the substantial obstacles that stand in the way of using observational data to study the causal effect of human rights treaties on state practices.4 Perhaps the most difficult of these obstacles to overcome is the fact that states are not randomly assigned commitments to human rights treaties, but instead select agreements based on their expected behavior. There is near universal ratification of the major human rights agreements, and most states have also subjected themselves to a series of overlapping international and domestic legal obligations that all seek to protect human rights. Furthermore, convincing evidence has recently emerged showing that the data used by researchers to measure human rights practices is systematically biased because of changes in standards used for reporting human rights abuses over time.5 It has proven difficult enough to design an observational study that accounts for one of these problems, but designing a study that accounts for all of them simultaneously has so far proven to be close to impossible.

Given these obstacles to using observational data, and the importance of the topic, scholars have begun to use experimental methods to study the effects of commitments to human rights agreements. This is because experimental studies can be designed to test the plausibility of the mechanisms theorized as ways that the ratification of human rights treaties may change state behavior. Although experimental methods are unlikely to definitively establish whether international treaties improve human rights outcomes, the handful of experimental studies conducted to date have found that information on commitments to international agreements does have a modest impact on public opinion.6 By doing so, these studies have provided some qualified support for the theorized mechanisms through which treaties may improve human rights.

There are, however, several reasons why these experiments have extremely limited generalizability. This is not only because of the general problem that subjects often behave differently in experimental settings than they do in the real world, but also because the human rights experiments that have been conducted to date have largely relied on surveys administered to samples of Americans. As a result, although the evidence produced by these studies is informative, it does not offer any conclusive answers on whether human rights treaties change state behavior.

That said, these early studies have demonstrated that using experimental methods to study human rights may help bring new evidence to this important debate. Future work should build upon this foundation by expanding: the countries in which experiments are conducted; the topics that are studied; and the type of experiments being conducted. By building on existing studies in this way, experimental research on human rights may be able to provide crucial evidence that supplements what has been produced using observational research methods.

This Essay proceeds as follows. Section II discusses mechanisms that may lead states to improve their human rights practices as a result of signing human rights agreements. Section III summarizes the evidence from observational studies on whether those mechanisms have led states to improve their human rights practices. Section IV outlines why observational studies face difficulties trying to empirically test the effects of human rights treaties. Section V presents experimental studies that have tried to examine the effect of human rights agreements. Section VI discusses future directions that experimental research on human rights should take.

II. Theories of Compliance

A central question in the study of international law is whether states change their behavior because of commitments to international agreements.7 This question has taken on unique importance in the human rights context because the mechanism that drives compliance in many areas of international law—reciprocity—is not present.8 In most areas of international law, the logical response to State A breaking a commitment is for State B to apply pressure on State A to change its behavior by reciprocating.9 For example, if State A were to violate an investment agreement by blocking firms from State B from acquiring their domestic companies, State B can respond by doing the same thing to firms from State A.10 In the human rights context, however, the same logic does not hold.11 Unlike investment agreements, if State A were to torture its citizens in violation of an international human rights treaty, State B will not try to convince State A to change course by torturing its own citizens. The implication is that although it is possible that international human rights treaties have helped to improve human rights practices, it is not because of the possibility of reciprocity. Instead, for commitments to human rights treaties to produce changes in state behavior, it would have to be through mechanisms other than reciprocity.

Figure 1 illustrates some of the mechanisms through which commitments to international human rights treaties could result in changes to state behavior. The solid lines represent the commitments that states make to human rights treaties.12 The dashed lines represent four mechanisms that might create pressure for states to change their human rights practices after having signed a human rights treaty: (1) ratification changes the domestic politics of countries that are at least partially democratic; (2) other countries coercing states to comply with their commitments; (3) treaty bodies pressuring the state to improve its human rights practices; and (4) non-governmental organizations (NGOs) advocating for compliance.

Figure 1: Mechanisms Through Which Human Rights Treaties May Change State Behavior

[[{"fid":"511","view_mode":"full","fields":{"format":"full","field_file_image_alt_text[und][0][value]":false,"field_file_image_title_text[und][0][value]":false,"field_caption[und][0][value]":"","field_photo_credit[und][0][value]":""},"type":"media","field_deltas":{"1":{"format":"full","field_file_image_alt_text[und][0][value]":false,"field_file_image_title_text[und][0][value]":false,"field_caption[und][0][value]":"","field_photo_credit[und][0][value]":""}},"attributes":{"class":"media-element file-full","data-delta":"1"}}]]

A. Domestic Politics

The primary mechanism that has been theorized as driving compliance with human rights treaties—illustrated by the dashed loop below State A—is that ratification of human rights treaties may change the domestic political landscape of partially democratic countries in ways that make it easier for activists and opposition political parties to lobby for improved rights practices.13 For all but the most powerful states, new human rights treaties create exogenous shocks to countries’ domestic agendas that force governments to decide whether to sign the treaty and make a public commitment to human rights. This shock may create an opening for domestic groups to lobby for improved rights practices, and if the agreement is signed, it may improve the ability of domestic groups to pressure the government to live up to the commitments it made. This pressure may come in many forms. For example, it may be human rights activists bringing litigation to try and have the courts pressure the government to bring human rights practices into compliance with the treaty, or it may be opposition political parties attacking the government for failure to live up to prior international commitments.

B. Empathetic Enforcement

A second mechanism that may drive compliance with human rights treaties—illustrated by the dashed lines between countries—is “empathetic enforcement” by powerful states.14 Although states are not obligated to apply strong pressure on repressive states to comply with human rights agreements, they may choose to do so. This pressure could come in many forms. For example, developed countries might condition foreign aid upon recipient states improving human rights practices, or be more likely to militarily intervene in countries that violate human rights commitments they have made. If powerful states were more likely to apply these forms of pressure against repressive states that had previously committed to human rights treaties, this would be one way that treaty commitment could lead to improved rights practices.

C. Treaty Body Pressure

A third mechanism that may drive compliance with human rights agreements—illustrated by the dashed line from the treaty to the states—is pressure applied by the international treaty bodies themselves.15 By voluntarily signing onto international human rights agreements, states subject themselves to monitoring by treaty bodies. The treaty bodies then have the power to produce reports that recommend ways for the states to improve human rights, or to issue statements chastising states for failure to live up to their commitments. Although the international treaty bodies cannot back up these statements with explicit sanctions, the reports may indirectly influence human rights practices.

D. NGO Advocacy

A fourth mechanism that may lead states to improve human rights practices after ratifying international agreements—illustrated by the dashed line from the NGOs to State A—is the influence of non-governmental organizations.16 There are a large number of NGOs dedicated to trying to improve human rights practices around the world. This includes organizations that cover a broad range of issues like Amnesty International and Human Rights Watch, as well as organizations focused on specific issue areas. These organizations use a range of tactics to pressure governments to improve their rights practices. These tactics include filing reports with treaty bodies, lobbying powerful governments to pressure repressive states, and naming and shaming states that violate human rights.

III. Evidence From Observational Studies

Despite years of scholarship researching the four mechanisms discussed in the previous Section, it is still disputed whether any of the four are viable channels through which treaty commitments may result in improved human rights practices. In this Section, I will briefly discuss some of the empirical research that has used observational data to test whether these four mechanisms have caused states to change their behavior.17

First, empirical research has examined whether domestic political mechanisms may lead states to improve their human rights practices after ratifying international treaties. In the most prominent research testing this theory, Beth Simmons tested the effectiveness of thirteen human rights treaties.18 Simmons found evidence that ratification was associated with improved human rights practices for seven of the thirteen agreements. Simmons’ research did not, however, suggest that the effectiveness of ratification was consistent for all countries. Instead, Simmons found that ratification of human rights treaties is not associated with improved human rights in stable democracies or stable autocracies, but is associated with better rights practices in transitioning democracies. According to Simmons, the reason that transitioning democracies change their behavior after ratifying human rights treaties is that the existence of the agreement helps domestic groups pressure their governments to be more respectful of rights. In addition to Simmons’ research, two recent empirical studies by political scientist Yonathan Lupu have used sophisticated empirical techniques to test whether domestic mechanisms increase the effectiveness of human rights treaties. Lupu’s studies have found that compliance with the International Covenant on Civil and Political Rights (ICCPR) has increased when there are more veto points in the state’s legislature,19 or when there is an independent judiciary that can evaluate claims that certain rights are being violated.20 Although Simmons’ and Lupu’s research has produced perhaps the most promising evidence that committing to human rights treaties may drive states to change their behavior, it is worth noting that it has been suggested that these studies at best show that ratification has made a marginal improvement in rights practices for a small number of countries on a handful of issues.21

Second, empirical research has tested whether empathetic enforcement might cause states to improve their human rights behavior. One body of research testing this theory has examined whether aid donors have punished states that violate human rights by withholding aid.22 Although that line of scholarship has produced mixed results, in the most recent research on the topic, political scientist Richard Nielsen found that states only selectively sanction human rights violators. That is, although aid is sometimes withheld from states that violate human rights, “[d]onors seem willing to abandon human rights norms when their friends and allies violate human rights.”23 Based on this finding and other similar research, scholars have largely been skeptical that states would actually improve their human rights practices because of rewards or punishments from more powerful countries.24

Third, there has been limited research trying to test whether pressure from human rights treaty bodies leads states to improve their human rights practices. Since human rights bodies do not have any strong enforcement powers and countries do not uniformly take their obligations to report to human rights bodies seriously,25 it is perhaps unsurprising that few scholars have tried to link activities of human rights bodies to improved rights practices. In one study on the topic, James Lebovic and Erik Voeten tested whether resolutions condemning a state’s human rights practices passed by the U.N. Commissions on Human Rights (UNCHR) resulted in that state receiving lower levels of foreign aid.26 Their study found that resolutions condemning human rights practices are associated with states receiving lower amounts of aid from multilateral organizations like the World Bank, but that resolutions condemning human rights practices are not associated with aggregate bilateral aid. Even if UNCHR resolutions result in lower foreign aid, however, it does not necessarily follow that the resolutions cause improvements in human rights. As a result, despite this finding, Beth Simmons has argued that “[n]o one has shown any significant external enforcement behind the provisions of international human rights treaties of a kind that might plausibly account for the patterns of compliance observed across a number of rights areas.”27

Fourth, a handful of empirical studies have tried to test whether pressure from NGOs has led to improved human rights practices. This research has focused on whether efforts to “name and shame” countries for violation of human rights has led to better practices. In what is perhaps the most widely cited study on the subject, Emilie Hafner-Burton found that efforts to spotlight human rights violations by NGOs are associated with increased repression.28 Hafner-Burton explains this finding by arguing that oppressive governments have incentive to crack down on opposition groups after they are empowered by the attention created by international organizations. In contrast to Hafner-Burton’s finding, Anna Schrimpf found that there is likely no relationship (but perhaps a positive relationship) between NGO attention and human rights practices.29 Finally, in related research, Judith Kelley and Beth Simmons found that including states on “watch lists” is associated with those states being likely to then criminalize human trafficking.30

IV. Barriers to Causal Inference With Observational Data

As the preceding discussion hopefully made clear, a consensus among empirical researchers has not emerged on the question of whether ratification of international human rights agreements improves state behavior. Although this is likely in part because empirical scholarship on international law is relatively new, it is also because there are a number of factors that make it incredibly difficult to study the causal effects of human rights treaties using observational data.31

First, observational research on human rights treaties has difficulty accounting for selection bias. Selection bias exists when the assignment of a treatment variable is systematically related to the dependent variable of interest. For example, if the subjects given Drug X in a clinical trial were picked to receive the drug because they were the healthiest subjects in the group, any positive effect of the drug may be attributable to the fact that the receipt of the Drug X (the treatment) was related to likely health outcomes (the dependent variable). The same problem exists in studying human rights treaties. Countries do not enter into treaties randomly. Instead, countries voluntarily choose to enter into human rights agreements based on their preferences and planned policies. To be clear, it might be the case that countries decide to enter into agreements when they do not plan on complying, or that they enter into the agreements when they do. Either is problematic for empirical research. Although empirical researchers have developed strategies to try and account for this selection bias,32 ultimately there is only so much that can be done given the fact that countries voluntarily opt into human rights regimes and thus any later human rights practices are endogenous to the decision to ratify the treaties.

Figure 2: Number of Countries Party to Major Human Rights Treaties

[[{"fid":"516","view_mode":"full","fields":{"format":"full","field_file_image_alt_text[und][0][value]":false,"field_file_image_title_text[und][0][value]":false,"field_caption[und][0][value]":"","field_photo_credit[und][0][value]":""},"type":"media","field_deltas":{"2":{"format":"full","field_file_image_alt_text[und][0][value]":false,"field_file_image_title_text[und][0][value]":false,"field_caption[und][0][value]":"","field_photo_credit[und][0][value]":""}},"attributes":{"class":"media-element file-full","data-delta":"2"}}]]

Second, even if it were possible to account for selection bias, a basic requirement of causal inference is variance. Simply put, there must be variance in the explanatory variable of interest to be able to assess the impact that it has on a given dependent variable. For example, if everyone that contracted a new disease was given Drug X, it would be impossible to know the effectiveness of Drug X (which is exactly why clinical trials randomly give some subjects placebos). This basic requirement of causal inference is a major hurdle for studying human rights treaties because there is very little variance in applicability of the major international human rights agreements (and no variance in the applicability of the parts of human rights law that are considered part of customary international law). As Figure 2 shows, nearly every country has signed onto the major human rights treaties. To further illustrate this point, the red line is the number of countries that are currently members of the U.N. Since there are very few countries that are not party to the major human rights agreements, and countries that are not party to the agreement are likely not representative of the countries where human rights agreements are likely to have an effect, it is thus very difficult to understand the effectiveness of these agreements on human rights practices.

Figure 3: Ratification History of Major Human Rights Agreements

[[{"fid":"521","view_mode":"full","fields":{"format":"full","field_file_image_alt_text[und][0][value]":false,"field_file_image_title_text[und][0][value]":false,"field_caption[und][0][value]":"","field_photo_credit[und][0][value]":""},"type":"media","field_deltas":{"3":{"format":"full","field_file_image_alt_text[und][0][value]":false,"field_file_image_title_text[und][0][value]":false,"field_caption[und][0][value]":"","field_photo_credit[und][0][value]":""}},"attributes":{"class":"media-element file-full","data-delta":"3"}}]]

Third, the current lack of variance in ratification of major human rights treaties would not be as much of a problem if historically there had been variance in which countries were party to the treaties. If that were the case, it would be possible to study the period between when the treaty had opened for ratification and when nearly all countries joined the agreements. For the first major human rights treaties, this was the case. As Figure 3 shows, countries continued to ratify the Convention on Elimination of Racial Discrimination (CERD), the International Covenant on Civil and Political Rights (ICCPR), and the International Covenant on Economic, Social, and Cultural Rights (ICESCR) for the entire latter half of the twentieth century.33 As Figure 3 also shows, however, more recent human rights treaties have been ratified incredibly quickly. For example, the most recent major human rights treaty—the Convention on the Rights of the Child (CRC)—became widely adopted in about five years.34 Although these increasingly short ratification windows may be a positive development for the human rights movement, they are problematic for human rights researchers who need longer windows before widespread ratification to study.

Figure 4: Countries that Have Made Commitments to Eliminate Torture

[[{"fid":"526","view_mode":"full","fields":{"format":"full","field_file_image_alt_text[und][0][value]":false,"field_file_image_title_text[und][0][value]":false,"field_caption[und][0][value]":"","field_photo_credit[und][0][value]":""},"type":"media","field_deltas":{"4":{"format":"full","field_file_image_alt_text[und][0][value]":false,"field_file_image_title_text[und][0][value]":false,"field_caption[und][0][value]":"","field_photo_credit[und][0][value]":""}},"attributes":{"class":"media-element file-full","data-delta":"4"}}]]

Fourth, another difficulty with conducting empirical research on the effectiveness of human rights treaties is that countries frequently have overlapping legal obligations. The problem is that it is only possible to understand the effect of a given variable if it is possible to isolate it from other variables that may be driving the outcome of interest. For example, if everyone that contracted a new disease were given both Drug X and Drug Y, it would be impossible to isolate the effects of Drug X and the effects of Drug Y (not to mention the effect of their interaction). The same problem often exists with international legal commitments. Countries often have signed multiple agreements that regulate human rights practices, have joined multiple organizations that monitor human rights compliance, and also have domestic laws that prohibit the same conduct as the international agreement. To illustrate this point, Figure 4 shows the number of countries that have signed the Convention Against Torture (CAT) and the number of countries that have provisions in their constitutions that prohibit torture.35 As the figure shows, there is nearly the same number of countries with each restriction as of 2015. These kinds of overlaps can make it difficult to tell whether changes (for better or worse) in human rights practices are attributable to international treaties or other sources of law.

Fifth, beyond the previous four concerns related to the ratification of human rights treaties, it is also the case that there are frequently only flawed measures of human rights practices available to use as dependent variables. Human rights treaties seek to protect a large number of rights. In fact, as Figure 5 shows, the number of rights protected by individual treaties ranges from eleven to fifty-eight. Currently, however, there is only data available on a handful of rights for a large number of countries. The result is that it is difficult to know whether a given treaty has improved rights because there are not good measures for all of the rights that any given treaty tries to protect.

                         Figure 5: Number of Rights Protected in Major Human Rights Agreements

[[{"fid":"531","view_mode":"full","fields":{"format":"full","field_file_image_alt_text[und][0][value]":false,"field_file_image_title_text[und][0][value]":false,"field_caption[und][0][value]":"","field_photo_credit[und][0][value]":""},"type":"media","field_deltas":{"5":{"format":"full","field_file_image_alt_text[und][0][value]":false,"field_file_image_title_text[und][0][value]":false,"field_caption[und][0][value]":"","field_photo_credit[und][0][value]":""}},"attributes":{"class":"media-element file-full","data-delta":"5"}}]]

Moreover, even if there was unbiased data for one of these rights—say the right of freedom of speech—it might be the case that the treaty improves protection for that right while making things worse for another dozen rights. That said, it is likely the case that all of the current data sources are biased. Recent research has shown that the most widely used dataset for human rights—the CIRI dataset—is systematically biased because reporting standards have changed over time.36 This not only calls into question the results of studies that have relied on the CIRI data,37 but also suggests that any other research that uses data on rights practices based on reports may be biased as well. Since the data on human rights practices has been shown to be biased and only covers a few rights, taken together these facts should make it clear that it is difficult to use conventional methods of empirical research to study the effectiveness of human rights agreements.

V. Experimental Evidence

One advantage of experimental research is that experiments can be designed to help test the mechanisms discussed in Section II that have been theorized as driving compliance with human rights agreements while accounting for the barriers to inference discussed in Section IV. Of course, my claim is not that experiments can address all of these problems at once.38 Instead, they can bring new evidence to evaluate the effectiveness of human rights agreements. To date, experimental studies have been designed to test the plausibility of all of the four mechanisms that were previously outlined: (1) domestic politics; (2) empathetic enforcement; (3) treaty body pressure; and (4) NGO advocacy.

A. Domestic Politics

There have been at least two experiments conducted that tested whether prior commitments to international agreements increased support for policy reforms that would make a state comply with its human rights obligations. In the first experiment designed to test this mechanism, Wallace conducted a survey on American adults to determine whether information on prior commitments to international law decreased support for torture.39 The survey told respondents that the U.S. often captures combatants from the opposing side during conflicts, and that those combatants may have information on future attacks. The survey further told respondents that some officials believe that using interrogation methods that include torture on the combatants may be a useful way to obtain information on future attacks. The survey then randomly told half of the sample that those interrogation methods would violate international law and treaties that the U.S. had previously committed to.40 After reading this information, respondents were asked whether they supported the use of interrogation techniques that include torture. Wallace found that respondents given information on international law treatment were six percent less supportive of using torture than the respondents who did not receive the information. This survey suggests that even on an issue where people have strong views, information on international law can have a statistically significant effect on the views of Americans.

The second study designed to test the Domestic Politics mechanism is Chilton.41 Chilton also conducted a survey on a sample of American adults to test the effect of information about international law, but made two changes to the design used by Wallace. First, the survey focused on a purely domestic topic to reduce the possibility that respondents changed their view in light of international law because they thought that retaliations may result from non-compliance. Second, the survey included a placebo treatment to see whether the legal commitments, instead of the general idea of human rights, is what lowered support for violations of human rights treaties. The survey specifically focused on the use of solitary confinement in American prisons. Respondents were told that solitary confinement is often used in American prisons, and that supporters of the practice argue that it is necessary to maintain prison discipline. There were then three randomly assigned treatments: the control group was not given any information on human rights; the placebo group was told that critics argue that the use of solitary confinement “violates the human rights of the prisoners held in solitary confinement;” and the international law group was told that critics argue that the use of solitary confinement “violates international human rights treaties that the United States has signed.” The idea is that the placebo group invoked the idea of human rights generally without relying on a specific legal commitment, thus making it possible to determine whether the existence of specific treaties—as opposed to general norms—is what influences opinion. The results of the survey suggested that there was no difference between the control and placebo groups in their level of support for using solitary confinement, but that there was a small but statistically significant difference with the international law treatment. Respondents that received international law treatment were roughly four percent more likely to support reforming the use of solitary confinement.

B. Empathetic Enforcement

Several experiments have tested whether information on international law makes respondents more likely to support efforts to change the policies of states that violate human rights. For example, Putnam and Shapiro (2013) tested respondents’ willingness to support economic sanctions against a foreign country because of its human rights violations.42 In a survey of American adults, Putnam and Shapiro told respondents about the use of forced labor in Myanmar. After being given varying amounts of information on the severity of the human rights violations in Myanmar, respondents were also given different information about the status of international law. The experiment suggested that information on international law increased support for government sanctions, but did not increase respondents’ willingness to say that they would personally boycott goods from Myanmar. Although the complete findings of the experiment were mixed, the authors did find qualified support for the idea that respondents learned that human rights violations were against international law.43

In another experiment that tests support for empathetic enforcement, Tomz and Weeks examined factors that influence support for military intervention.44 In the survey, respondents were told about a hypothetical country developing nuclear weapons. Respondents were randomly told about the country’s regime type—that is, whether or not it was democratic—as well as about the country’s human rights record. Half of the respondents were told that “[t]he country does not violate human rights; it does not imprison or torture its citizens because of their beliefs,” while the other half were told that “[t]he country violates human rights; it imprisons or tortures some of its citizens because of their beliefs.” The respondents were then asked about their willingness to support military strikes against the country. The results suggest that respondents were dramatically more likely to support military strikes when the country violated human rights. Specifically, the experiment revealed a twenty-three percent increase in support for military strikes against democracies that violate human rights, and a sixteen percent increase in support military strikes against autocracies that violate human rights.

C. Treaty Body Pressure

At least one experiment has tested whether treaty body pressure increases support for reforms aimed at improving human rights. Anjum, Chilton, and Usman conducted an experiment in Pakistan that tested whether learning that policy reforms aimed at improving women’s rights were proposed by the U.N. increases support for the reforms.45 In the experiment, roughly six hundred subjects were recruited in Pakistan to take an in-person survey on women’s rights. The respondents were asked whether they supported four reforms that the U.N. Committee on the Elimination of Discrimination Against Women recommended that Pakistan should implement. Half of the subjects were told that the reforms were proposed by the U.N., while half of the subjects were not told who proposed the reforms. The results of the experiment revealed that learning that the U.N. supported the reforms increased support for them. The results of the experiment thus provide some evidence that when the U.N. supports a policy reform, it might increase its domestic support.

D. NGO Advocacy

A recent experiment has also tested how international human rights law may help with NGO advocacy. Hafner-Burton, LeVeck, and Victor conducted an experiment with experienced NGO professionals that had experience working on environmental issues and human rights.46 For the human rights treatments, the experiment asked the sample of NGO professions how much more difficult they believed it would be to accomplish their advocacy goals if either the ICCPR (which is a binding agreement) or the Universal Deceleration of Human Rights (which is a non-binding agreement) did not exist. The results suggested that they believed that their job would be more difficult if the laws did not exist, and that it would be even more difficult if the binding ICCPR did not exist. Moreover, the respondents indicated that the reason that their jobs would be particularly difficult is that removal of the law would make litigation and accountability efforts more difficult. These results thus provide evidence that NGO advocates at least believe that the presence of international human rights law helps them to accomplish their goals.47

VI. Future Research and Concluding Thoughts

Although the experimental research on human rights has suggested that information on human rights agreements may be able to change public opinion, there are serious limitations to the studies that have been conducted to date. Most notably, the fact that nearly all of the experiments on human rights have been conducted using vignette studies with respondents in the U.S. seriously limits the generalizability of the findings.48 This method of research is still in its infancy, however, and could learn a great deal from the more developed experimental literatures in development economics, American politics, and international relations. I will briefly outline three ways that future experimental studies on human rights could improve on prior research.

First, experiments should be conducted in transitioning democracies. There is little evidence to suggest that human rights treaties have improved the rights practices of stable democracies or stable autocracies. Instead, the best available evidence suggests that human rights agreements are most likely to change the behavior of developing democracies.49 That is, countries like those in Eastern Europe or Central America that are not well-established, stable democracies, but that have moved towards democracy in the last few decades. To date, however, nearly all of the experiments on the effectiveness of human rights have been conducted in the U.S.50 This is a major shortcoming of the small experimental literature on human rights because the U.S. is not only a stable democracy, but also because Americans may have idiosyncratic views towards international human rights agreements. There have been a large number of experiments conducted by political scientists in developing countries in the last several years,51 and there is a great deal that scholars interested in international human rights law could learn from these efforts.

Second, experimental research should explore a wider range of topics. For better or worse,52 there are dozens of rights that are protected by international human rights agreements.53 One of the advantages of experimental research is that a study can be designed to test whether international commitments can increase support for providing any right that is included within a treaty. Since observational studies can only test the effectiveness of rights where cross-country data is available, this is one comparative advantage of experimental research. This advantage should be leveraged, and studies should be conducted to test whether the impact that information has on international law is conditional on the nature of the right at issue.

Third, scholars conducting experiments on human rights should start to utilize field experiments. Although the survey experiments that have been conducted provide new data on the influence of human rights agreements on public opinion, it is not clear whether those differences in opinion translate into meaningful changes in the respondents’ behavior, and in turn, into changes in public policies. Researchers in a number of fields related to international law—including international economic regulation, international relations, and development economics—have begun to conduct field experiments to determine whether experimental interventions result in changes to real world behavior.54 These experiments are thus able to reduce the concern that using experimental methods will produce internally valid results, but that they will not have any real-world applicability.

Of course, even if future experimental research on the effectiveness of human rights law moves in the direction that I have outlined, it does not mean that any one study will be able to definitely demonstrate whether human rights treaties have improved human rights around the world. Instead, experimental methods can produce evidence that complements empirical research using observational data. This would make it possible to triangulate on the answer to whether human rights treaties improve respect for rights.

• 1For research that concludes that human rights treaties improve human rights practices, see Yonatan Lupu, Best Evidence: The Role of Information in Domestic Judicial Enforcement of International Human Rights Agreements, 67 Int’l Org. 469 (2013); Yonatan Lupu, Legislative Veto Players and the Effects of International Human Rights Agreements, 59 Am. J. Pol. Sci. 578 (2015); Daniel W. Hill, Estimating the Effects of Human Rights Treaties on State Behavior, 72 J. Pol. 1161 (2010); Beth A. Simmons, Mobilizing for Human Rights: International Law in Domestic Politics (2009).
• 2For research that concludes that human rights treaties either have no (or at least a very small) effect on human rights practices, see, for example, Daniel W. Hill, Jr. & Zachary M. Jones, An Empirical Evaluation of Explanations for State Repression, 108 Am. Pol. Sci. Rev. 661 (2014); Eric A. Posner, The Twilight of International Human Rights Law (2014); Emilie M. Hafner-Burton & Kiyoteru Tsutsui, Human Rights in a Globalizing World: The Paradox of Empty Promises, 110 Am. J. Soc. 1373 (2005); Eric. Neumayer, Do International Human Rights Treaties Improve Respect for Human Rights?, 49 J. Conflict Resol. 925 (2005); Oona Hathaway, Do Human Rights Treaties Make a Difference?, 111 Yale L. J. 1935 (2002); Linda Camp Keith, The United Nations International Covenant on Civil and Political Rights: Does It Make A Difference in Human Rights Behavior?, 36 J. Peace Res. 95 (1999).
• 3See Simmons, supra note 1, at 123 (“Human rights regimes do not involve reciprocal compliance. . . . No government is likely to alter its own rights practice to reciprocate for abuses elsewhere.”); Posner, supra note 2, at 79 (“In the case of ordinary treaties, the main reason that states comply with their obligations is that they fear that if they do not, other treaty parties will violate their own obligations. . . . [T]his logic does not easily carry over to human rights treaties.”).
• 4The difficulties that arise when trying to empirically test compliance with human rights treaties have been well documented. For a general overview, see Adam S. Chilton & Dustin Tingley, Why the Study of International Law Needs Experiments, 52 Colum. J. Transnat’l L. 173 (2013). For a discussion focused on the problems posed by selection bias, see Yonatan Lupu, The Informative Power of Treaty Commitment: Using the Spatial Model to Address Selection Effects, 57 Am. J. Pol. Sci. 912 (2013). For a discussion of the bias in efforts to measure human rights violations, see Christopher J. Fariss, Respect for Human Rights Has Improved Over Time: Modeling the Changing Standard of Accountability, 108 Am. Pol. Sci. Rev. 297 (2014). For critical comments of approaches that have been used in the literature, see Eric A. Posner, Some Skeptical Comments on Beth Simmons’s “Mobilizing for Human Rights,” 44 NYU J. Int’l L. & Pol. 819 (2012).
• 5See Fariss, supra note 4.
• 6Experiments studying international human rights law include: Adam S. Chilton, The Influence of International Human Rights Agreements on Public Opinion: An Experimental Study, 15 Chi. J. Int’l L. 110 (2014); Geoffrey P.R. Wallace, Martial Law? Military Experience, International Law, and Support for Torture, 58 Int’l Stud. Q. 501 (2014); Geoffrey P.R. Wallace, International Law and Public Attitudes Toward Torture: An Experimental Study, 67 Int’l Org. 105 (2013); Michael Tomz & Jessica L. P. Weeks, Human Rights, Democracy, and Public Support for War (Working Paper, Mar. 2015), (https://perma.cc/RLA5-78SU); Tonya L. Putnam & Jacob N. Shapiro, International Law and Voter Preferences: The Case of Foreign Human Rights Violations, Hum. Rts. Rev. Online (Apr. 7, 2017), (http://perma.cc/27L9-KTPD); Michael Tomz, Reputation and the Effect of International Law on Preferences and Beliefs (Working Paper, Feb. 2008), (https://perma.cc/Q6WF-48LW).
• 7For an example of the debate, compare Abram Chayes & Antonia Handler Chayes, On Compliance, 47 Int’l Org. 175 (1993) with George W. Downs, David M. Rocke, & Peter N. Barsoom, Is the Good News about Compliance Good News about Cooperation?, 50 Int’l Org. 379 (1996).
• 8For longer discussions of the role of reciprocity in compliance with international law, see Jack L. Goldsmith & Eric A. Posner. The Limits of International Law (2005); Eric A. Posner & Alan O. Sykes, Economic Foundations of International Law (2013). In an example of an empirical work that explores the effects of reciprocity, James D. Morrow, Order Within Anarchy: The Laws of War as an International Institution (2014) shows that reciprocity drives compliance with the laws of war.
• 9See Posner, supra note 2, at 79.
• 10See, for example, Dustin Tingley, Christopher Xu, Adam S. Chilton, & Helen Milner, The Political Economy of Inward FDI: Opposition to Chinese Mergers and Acquisitions, 8 Chinese J. Int’l Pol. 27 (2015) (finding that the U.S. is more likely to block attempts by Chinese companies to acquire American firms when China has previously blocked American companies in the same industries).
• 11See Posner, supra note 2, at 79–82; Simmons, supra note 1, at 122.
• 12The exact reason that states choose to commit to human rights treaties is a subject of debate. For example, it may be the case that states make commitments because of domestic considerations, or it may be the case that states sign agreements because they are persuaded to do so by other states with either carrots or sticks. For discussions of the topic, see Posner, supra note 2, at 59–68; Posner & Sykes, supra note 8, at 202–06; Simmons, supra note 1, at 57–111. For a recent empirical consideration of this topic, see Richard A. Nielsen & Beth A. Simmons, Rewards for Ratification: Payoffs for Participating in the International Human Rights Regime, 59 Int’l Stud. Q. 179 (2015).
• 13For a discussion of the ways that ratifying human rights agreements may influence domestic politics and thus lead to changes in state behavior, see Simmons, supra note 1, at 125–48.
• 14I borrow the term “empathetic enforcement” from Beth Simmons. See Simmons, supra note 1, at 116. This mechanism is also often referred to simply as “coercion” by powerful states. For a discussion of the mechanism and examples of where it has been used, see Goldsmith & Posner, supra note 8, at 115–17.
• 15It is worth noting that scholars have largely been skeptical that this mechanism could actually lead states to change their behavior. See Posner, supra note 2, at 95–104; Emilie M. Hafner-Burton, Making Human Rights a Reality 44–66 (2013); Simmons, supra note 1, at 121–25.
• 16For a discussion of the role that NGOs can play in promoting human rights, see Hafner-Burton, supra note 15, at 151–63. For skepticism about this possibility, see Posner, supra note 2, at 82.
• 17A complete survey of the empirical human rights literature is beyond the scope of this Essay. For a short introduction to the empirical literature on human rights, see Adam S. Chilton & Eric A. Posner, The Influence of History on States’ Compliance with Human Rights Obligations, 56 VA. J. Int’l L. 211 (2016). For more thorough reviews, see Emilie M. Hafner-Burton, A Social Science of Human Rights, 51 J. Peace Res. 273 (2014); Emilie M. Hafner-Burton, International Regimes for Human Rights, 15 Ann. Rev. Pol. Sci. 265 (2012); Gregory Shaffer & Tom Ginsburg, The Empirical Turn in International Legal Scholarship, 106 Am. J. Int’l L. 1, 19–25 (2012); Beth Simmons, Treaty Compliance and Violation, 13 Ann. Rev. Pol. Sci. 273 (2010).
• 18Simmons, supra note 1.
• 19Lupu, Legislative Veto Players, supra note 1.
• 20Lupu, Best Evidence, supra note 1. Lupu’s claim is that the presence of an independent judiciary increases respect for the ICCPR for civil rights, but not for physical integrity rights. The theory Lupu puts forward to explain this finding is that an independent judiciary can police rights violations for issue areas where the standards, proof, or evidence production costs are low. For another empirical paper that considers the effectiveness of domestic legal institutions on compliance with human rights agreements, see Emilia Justyna Powell & Jeffery K. Stanton, Domestic Judicial Institutions and Human Rights Treaty Violations, 53 Int’l Stud. Q. 149 (2009).
• 21Posner, supra note 2, at 76–78.
• 22Richard A Nielsen, Rewarding Human Rights? Selective Aid Sanctions Against Repressive States. 57 Int’l Stud. Q. 791, 791–92 (2013).
• 23Id. at 791.
• 24See Simmons, supra note 1, at 116 (“If we are looking for empathetic enforcement [of human rights treaties] from other countries, we will be looking in vain for a long time.”); Posner, supra note 2, at 80–81 (“Even powerful countries often cannot exert sufficient pressure on a human rights violator to cause it to improve its behavior, because the target of sanctions can often retaliate by improving its ties with the sanctioning countries’ rivals.”).
• 25See Cosette Creamer & Beth A. Simmons, Ratification, Reporting and Rights: Quality of Participation in the Convention Against Torture, 37 Hum. Rts. Q. 579 (2015).
• 26James H. Lebovic & Erik Voeten, The Cost of Shame: International Organizations and Foreign Aid in the Punishing of Human Rights Violations, 46 J. Peace Res. 79 (2009).
• 27Simmons, supra note 17, at 291.
• 28Emilie Hafner-Burton, Sticks and Stones: Naming and Shaming the Human Enforcement Problem, 62 Int’l Org. 689 (2008).
• 29Anna Schrimpf, Does INGO Activism Improve Human Rights Practices? Evidence from a Multi-Issue Analysis (Working Paper 2014).
• 30Judith G. Kelley & Beth A. Simmons, Politics by Number: Indicators as Social Pressure in International Relations, 59 Am. J. Pol. Sci. 55 (2015).
• 31The following discussion is based on Chilton & Tingely, supra note 4. The discussion in Chilton & Tingley uses human rights law and international humanitarian law to make a broader point about empirical research in international law and also develops the arguments discussed here more fully. Additionally, the data in this essay was originally presented in Chilton & Tingley, supra note 4.
• 32For an excellent discussion of the ways that researchers have tried to account for selection bias while studying international law, see Lupu, supra note 4.
• 33It is important to note that the number of countries in the world also dramatically increased during this period. As a result, Figure 3 may suggest there was more variance in early years than there actually was. For example, in the early 1980s, when roughly 120 countries were party to the CERD, approximately 150 countries were members of the U.N.
• 34In another example, 159 countries have signed the Convention on the Rights of Persons with Disabilities (CRPD) since it opened for signature in 2007. Convention on the Rights of Persons with Disabilities, May 30, 2007, 2515 U.N.T.S., available at (https://perma.cc/Q5LE-7HA9).
• 35For information on the data on constitutional torture prohibitions used in this figure, see Adam S. Chilton & Mila Versteeg, The Failure of Constitutional Torture Prohibitions, 44 J. L. Studies 417 (2015). For another discussion of this issue, see Adam S. Chilton & Mila Versteeg, International Law, Constitutional Law, and Public Support for Torture, 3 Res. & Pol. 1 (2016).
• 36See Fariss, supra note 4. The CIRI dataset uses State Department and Amnesty International Reports to code the human rights practices of countries on a 3-point scale. Fariss, however, uses a sophisticated empirical model to show that these widely-used scores are biased because the reporting standards have changed over time. Although Fariss also has created an unbiased single measure of “repression” that can be used as a dependent variable, it is only one variable that does not cover all of the rights that treaties try to protect.
• 37This is not a criticism of scholars that have used the CIRI data in their research (which I readily admit to being one of); using the CIRI data is a completely reasonable decision, given that it is the data source with the broadest coverage available. After Fariss, supra note 4, however, scholars should be aware of the limitations with the CIRI data and any other dataset that uses reports on human rights to code countries.
• 38Of course, experimental research has problems of its own—the most notable of which being external validity. The limitations of experimental research are briefly discussed in Section VI, infra.
• 39Wallace, International Law and Public Attitudes Toward Torture, supra note 6. A companion paper also analyzes the results of this experiment, but focuses on whether there is a differential treatment effect for respondents with military experience. See Wallace, Martial Law, supra note 6.
• 40This summary only discusses on the primary manipulation in Wallace’s paper, but the paper also includes two other treatments and a related second experiment. See Wallace, International Law and Public Attitudes Toward Torture, supra note 6.
• 41Chilton, supra note 6.
• 42Putnam & Shapiro, supra note 6.
• 43In a related earlier study, Tomz, supra note 6, also tested whether respondents were willing to support sanctions in response to human rights violations in Myanmar. The difference, however, is that in the international law treatment, respondents were told that the sanctions would violate international law. Tomz found that information on international law decreased support for the use of sanctions by seventeen percent.
• 44Tomz & Weeks, supra note 6.
• 45Gulnaz Anjum, Adam S. Chilton, & Zahid Usman, United Nations Endorsement & Support for Human Rights: An Experiment on Women’s Rights in Pakistan, (U. of Chi. Coase-Sandor Inst. for L. & Econ. Research Working Paper No. 563, Apr. 2016).
• 46Emilie M. Hafner-Burton, Brad L. LeVeck, & David G. Victor, How Activists Perceive the Utility of International Law, 78 J. Pol. 167 (2016).
• 47For another experiment examining how International Law can be effectively used by advocates, see Kayla Jo McEntire, Michele Leiby, & Matthew Krain, Human Rights Organizations as Agents of Change: An Experimental Examination of Framing and Micromobilization, 109 Am. Pol. Sci. Rev. 407 (2015).
• 48But see Anjum, Chilton, & Usman, supra note 45.
• 49Simmons, supra note 1, at 152–53.
• 50See Chilton, supra note 6; Wallace, supra note 6; Putnam & Shapiro supra note 6; Tomz & Weeks, supra note 6.
• 51See, for example, Sarah Sunn Bush & Amaney A. Jamal, Anti-Americanism, Authoritarian Politics, and Attitudes about Women’s Representation: Evidence from a Survey Experiment in Jordan, 59 Int’l Stud. Q. 34 (2015); Graeme Blair, Kosuke Imai & Jason Lyall, Explaining Support for Combatants during Wartime: A Survey Experiment in Afghanistan, 107 Am. Pol. Sci. Rev. 679 (2013); Graeme Blair et al., Poverty and Support for Militant Politics: Evidence from Pakistan, 107 Am. Pol. Sci. Rev. 679 (2013).
• 52Posner, supra note 2, argues that including too many rights in international agreements may have adverse consequences on rights protections.
• 53See Chilton & Tingley, supra note 4, at 214.
• 54See, for example, Michael G. Findley, Daniel L. Nielsen, & J.C. Sharman, Using Field Experiments in International Relations: A Randomized Experiment of Anonymous Incorporation, 67 Int’l Org. 657 (2013); James D. Fearon, Macartan Humphreys, & Jeremy M. Weinstein, Can Development Aid Contribute to Social Cohesion after Civil War? Evidence from a Field Experiment in Post-Conflict Liberia, 99 Am. Econ. Rev. 287 (2009); James Habyarimana et al., Why Does Ethnic Diversity Undermine Public Goods Provision?, 101 Am. Pol. Sci. Rev. 709 (2007).