'Politics is the art of the possible,' proclaimed Bismarck, but our leaders are not alone in this great balancing act. Voters often face difficult tradeoffs when forming their preferences. Understanding this process, and evaluating policymakers’ success in responding to these preferences, is central to political science. In a new study, IAST researchers Daniel Chen and Karine Van der Straeten teamed up with Charlotte Cavaillé (Ford School, Michigan, former IAST fellow) to assess two methods for measuring preference intensity. They argue that it is not enough to know what voters want, we must also establish how much they want it.

WHY DO WE NEED BETTER TOOLS FOR MEASURING HOW STRONGLY VOTERS FEEL?
Many questions in political science require understanding the intensity of voters’ preferences but this is rarely explicitly measured in surveys. Respondents are often asked about policies using a Likert scale ranging from ‘strongly agree’ to ‘strongly disagree’. But the Likert method suffers from the ‘abundance problem’, giving respondents only limited incentives to consider tradeoffs across issues, such as immigration control at the expense of access to the European single market. As a result, their answers convey little information about their priorities and the issues they are willing to compromise on.
Another concern, particularly relevant in the US context, is the ‘bunching problem’. Partisan polarization may lead to respondents who care intensely about a politicized issue being lumped together with respondents who feel pressure to pay lip service to party norms. This bunching can make results more sensitive to underlying modeling assumptions, as well as favoring predictors tied to partisan identity.
WHICH METHODS DOES YOUR PAPER INVESTIGATE?
We examine, both theoretically and empirically, two easy-to-implement methods. The Likert+ method combines a Likert item with one that asks respondents whether an issue is personally important to them. The Likert+ and Likert methods should theoretically face some of the same limits, as both suffer from the abundance problem and do not penalize partisan motives.
We also study Quadratic Voting for Survey Research (QVSR), a new method in which respondents have only a limited budget to ‘pay’ for votes on a bundle of issues. They may express intense preferences by voting repeatedly for the same issue, but each additional vote is increasingly costly. This compels respondents to arbitrate between the issues, mimicking real-world tradeoffs. Unlike the Likert method, in which respondents face no tradeoffs and can pick end-of-scale responses (e.g. strongly favor/oppose) at no cost, QVSR respondents are expected to de-bunch and prioritize issues they care about the most.
HOW EFFECTIVE ARE THESE TOOLS?
To compare these methods, we randomly assign individuals to take the same survey varying only the measurement method. We then give respondents the option to perform tasks – such as donating to a gun-control advocacy group, or writing a letter to a senator – that involve a tradeoff between two policy issues. Higher values on this behavioral outcome imply more resources allocated to this issue, whether in dollars (e.g. amount donated) or effort (e.g. length of the letter). We then compare each tool’s ability to discriminate between respondents according to the preference intensity suggested by their behavior.
Our results indicate that asking about issue importance with Likert+ does not convey much more information. In contrast, QVSR more consistently discriminates between intense and weak preferences. Assuming researchers can afford the time to explain it to respondents, QSVR appears to provide a significant improvement over Likert.
We also find that, while Likert items convey little information regarding the likelihood of personally benefiting from a policy, preferences measured using QVSR do. This suggests that people directly affected by a policy, such as a minimum wage increase, have more intense preferences towards it than unaffected individuals.
HOW CAN POLITICAL SCIENCE BENEFIT FROM THIS RESEARCH?
Revisiting debates on the determinants of policy preferences, or the congruence between mass opinions and policy outcomes, we show that conclusions reached using Likert items can change once differences in preference intensity are better accounted for. In particular, we show that, using Likert, large majorities of Republicans and Democrats express support for fiscal discipline. Using QVSR, however, we find Republicans care more about fiscal discipline than Democrats. This suggests that demand-side factors may have played an important role in failed negotiations over the Build Back Better Bill.
Our model lays the foundations for evaluating how the abundance and bunching problems affect the data generation process and, ultimately, hypothesis testing. Our results show the benefits of grounding survey data in a theory of choice. But many methodological issues remain uninvestigated. To facilitate follow-up studies, we have made available a web application enabling researchers to vary key features of the survey method. We hope this will help generate additional evidence on QVSR and other innovative survey methods and spur new research on the measurement strategies that underpin reliable empirical findings.

Article published in IAST Magazine #19, Spring 2022

Photo by Arnaud Jaegers on Unsplash