Coral Reef Incentive Compatibilty Memo

Memorandum
To:

Norman Meade, Assessment and Restoration Division, NOAA

From:

David J. Chapman and Richard Bishop, Stratus Consulting Inc.

Date:

10/21/2009

Subject:

Incentive Compatibility of Coral Reef Conjoint Survey

This memorandum addresses the question of incentive compatibility (sometimes also referred to
as demand revelation) of the Coral Reef conjoint survey instrument. For purposes of this
memorandum, we use the term “incentive compatibility” to mean truthful revelation of one’s
preferred choice of the conjoint questions presented in the survey.1
We first present a brief summary of incentive compatibility criteria as presented in Carson and
Groves (2007) and then discuss applications to conjoint studies. We conclude with a specific
evaluation of the incentive compatibility properties of the Coral Reef survey.
Summary of Incentive Compatibility Criteria
Carson and Groves (2007) used the mechanism design literature to identify three main conditions
for incentive compatibility, willingness-to-pay (WTP) elicitation formats: (1) whether
respondents care about how the outcome might be influenced by the answers they provided
(consequentiality), (2) whether the aspects of the scenario described are plausible (plausibility),
and (3) how the survey results are likely to be used. For a question to be consequential, a
respondent must believe that his/her response may influence some action and he/she must care
about possible outcomes. As stated in Carson and Groves (2007) “As long as the economic
agents (hereafter, agents) being surveyed believe that their responses might influence the actions
taken by businesses or governments (hereafter, agency), the standard economic model suggests
that agents should respond to the survey in such a way as to maximize their expected welfare.”
Incentive compatibility of conjoint type survey formats mainly focus on choice, rather than
ranking, question formats, but the results are generally transferable. For example, Bateman et al.
(2004), Ding et al. (2005), Ding (2007), and Collins and Vossler (2009) find that if researchers
use provision mechanisms that require participants to “live with” then the design is incentive
compatible. Carson and Groves (2007) also find that choices should be consequential and that
respondents should believe payments would be enforceable in order to be incentive compatible.
In other words, if respondents feel their answers are consequential, then they should truthfully
reveal their preferences.

1. Truthful demand revelation and incentive compatibility are used interchangeably in the literature.

SC11883

Stratus Consulting

Memorandum (10/21/2009)

An additional issue is whether or not respondents fully understand the task presented to them.
Bateman et al. (2004) looked at the issue of respondent’s comprehension of the possible choice
set in evaluating performance. They found that “When a stepwise disclosure procedure is
adopted, the observed scope sensitivity is substantially and significantly affected by the order in
which goods are presented but such procedural variance is not observed within advance
disclosure designs.” Thus, respondents understanding of the full choice set to be evaluated
improve reliability. Finally, Collins and Vossler (2009) looked at the issue of two versus three
choices and found that in fact the trichotomous choice performs marginally better than a
dichotomous format on a number of indicators.
Application to the Coral Reef Survey
Below we provide a brief description of the Coral Reef survey then describe how the survey
design addresses each of the identified criteria for incentive compatibility.
The introduction of the Coral Reef survey presented respondents with an incentive to complete
for the survey in a truthful manner. We tell respondents that the government is deciding whether
to undertake some actions to further protect Hawaiian coral reefs and that it wants the public’s
input in the decision. In this manner the survey is an “advisory referendum.” The survey
provided multiple statements about the consequences of respondent’s choices (e.g., the
government is making a decision; the government wants input from citizens to make its decision;
this decision will have monetary impacts on individuals and individual’s choices will affect the
quality of coral reefs in Hawaii). The survey used a multinomial choice conjoint format to elicit
WTP for two distinct mechanisms that would provide two different levels of reef protection and
one option that combined the two individual options. Along with a status quo option, the four
choices presented to respondents were:


The status quo of no additional reef protection and no additional annual taxes



Increased protection from overfishing through the increase in size of no-fishing zones
around the reef and an increase in annual taxes



Repair of coral reefs damaged from ship strikes and an increase in annual taxes



A combined program of both protection for overfishing and reef repair with increased
annual taxes.

Before respondents were asked to make a decision, they were reminded of their budget
constraints. All four choices are presented to respondents at the same time, and they are
instructed to indicate their preferred choice. Their preferred choice is removed from the set and
then respondents are provided with the three remaining choices and again instructed to select

Page 2
SC11883

Stratus Consulting

Memorandum (10/21/2009)

their preferred choice. Finally, the remaining pair of choices is provided to respondents. In this
manner, a full ranking of respondents preferences is obtained.
Evaluation of Incentive Compatibility Conditions
Consequentiality: Respondents are informed that their responses to the survey will help the
government make a decision about what more, if anything, should be done to protect the coral
reefs around Hawaii. Respondents are also told that their annual federal taxes may increase as a
result of their decision.
Plausibility: The protection and repair mechanisms are both types of actions that have occurred
in the past. Examples are provided of where these types of actions have worked before to protect
and restore coral reefs. Through focus groups, we verified that respondents believed and
accepted the scenarios as plausible.
Use of survey results: Respondents are explicitly told that the results of the survey will inform
government decisions on whether or not they should to do more to protect coral reefs around the
main Hawaiian island. The respondents were told that their responses would influence the
actions taken by the government, which is equivalent to an “advisory referendum.”
Task comprehension: Respondents understanding of the specific task they are being asked to
complete is important. Extensive pretesting through focus groups and cognitive interviews
confirmed that individuals understood how to accurately complete the task. In addition, a warmup question was used to ensure that respondents understood how to make the tradeoff between
two programs before they saw the main choice options. The Coral Reef survey used an “advance
disclosure design” by showing the full choice set that individuals would face at the beginning of
the task and kept attribute levels constant for each specific respondent.
Credibility: Problems can also occur when a respondent is given inconsistent information at
various points in a survey. Examples include providing two different cost numbers in a doublebounded dichotomous-choice elicitation format, or asking respondents about the provision of two
different levels of the same public good at different points in a survey without corresponding
changes in other information to justify those different levels.
In particular, this can be an issue with double bounded contingent valuation questions when
respondents do not know the full set of choices being provided to them (the bid amounts changed
in the second round) and respondents may feel “gamed” such that they may not truthfully reveal
their preferences in the second round. The Coral Reef survey enhanced credibility and task
understanding by providing the full choice set in the initial presentation and keeping attribute
levels between choice sets constant for a given respondent.

Page 3
SC11883

Stratus Consulting

Memorandum (10/21/2009)

The above criteria are successfully met in the overall design of the Coral Reef survey and led to
the conclusion that the survey design does not violate incentive compatibility conditions and
elicits truthful demand revelation.
Bibliography
Bateman, I., M. Cole, P. Cooper, S. Georgiou, D. Hadley, and G.L. Poe. 2004. On visible choice
sets and scope sensitivity. Journal of Environmental Economics and Management 47:71–93.
Carlsson, F. and P. Martinsson. 2001. Do hypothetical and actual marginal willingness to pay
differ in choice experiments? Journal of Environmental Economics and Management 41:179–92.
Carson, R. and T. Groves. 2007. Incentive and informational properties of preference questions.
Environmental & Resource Economics 37:181–210.
Collins, J. and C. Vossler. 2009. Incentive compatibility tests of choice experiment value
elicitation questions. Journal of Environmental Economics and Management 58:226–235.
Ding, M.R. 2007. An incentive-aligned mechanism for conjoint analysis. Journal of Marketing
Research 44:214–23.
Ding, M., R. Grewel, and J. Liechty. 2005. Incentive-aligned conjoint analysis. Journal of
Marketing Research 67(XLII):67–82.

Page 4
SC11883