Attachment A4

Date: May 10, 2011
To:

Nathan Dietz
Corporation for National and Community Service

CC:

JoAnn Jastrzab
Abt Associates

From: Benjamin Phillips, Courtney Kennedy
Abt SRBI
Johnny Blair
Abt Associates
RE:

Results of Round 2 Cognitive Testing

This memo summarizes the results of the second round of cognitive testing for the
November 2011 Civic Engagement Supplement (CES) to the Current Population Survey
(CPS), conducted by Abt Associates on behalf of the Corporation for National and
Community Service (CNCS). The sections of this memo are outlined below, with page
numbers for the beginning of each section listed on the right.
Methods

2

Responses to Survey Questions

2

Responses to Debriefing Items

16

Recommendations

18

Appendix A: Cognitive Testing Protocol

26

Appendix B: Review of Proxy Reporting

38

Abt SRBI Inc. 55 Wheeler Street Cambridge, MA 02138 Phone 617.386.2600 Fax 617.386.2622 www.srbi.com
Cambridge Chicago North Carolina New York City Washington D.C. Florida New Jersey Cincinnati

2
Methods
For round 2 of the Civic Engagement Supplement cognitive tests, a total of 15 interviews
were completed in the period April 28-May 2, 2011 at the Abt Associates Cognitive
Testing Laboratory in Bethesda, MD. Respondents were selectively recruited for
diversity with respect to age, sex, and socioeconomic status from a third-party
contractor. Interviews took approximately one hour and were video recorded for
subsequent analysis. Respondent demographics are summarized below.
Round 1
Round 2
2/28 – 3/1 4/28 – 5/2
Age
18-24
25-34
35-44
45-54
55-64
65+
Sex
Male
Female
Household Income
$50,000 or less
More than $50,000
Education
High school or less
Some college
College graduate
Advanced degree
Adults in household
1
2
3+

1
3
2
6
1
2

0
5
4
2
3
1

8
7

7
8

8
7

8
7

8
3
3
1

4
6
5
0

5
8
2

3
9
3

Responses to Survey Questions
Results are organized by the order in which the questions appeared in the cognitive
testing protocol and refer to the question numbers found in the protocol. Item wording
is recapitulated for ease of reading.

3
Q1

The first question is about LOCAL elections, such as for mayor or a school board.
Do you always vote in local elections, do you sometimes vote, do you rarely vote,
or do you never vote?
(1) Always vote
(2) Sometimes vote
(3) Rarely vote
(4) Never vote

Most respondents (10/15) appeared to answer the question appropriately, giving
appropriate examples of local elections (school board, county, city). These respondents,
however, seemed to be using short reference periods: two years, three years, or the last
election, so responses will tend to be always or never, rather than the desired
approximate frequency. Two respondents didn’t listen to the question fully before
responding and based their answers on national elections. Another did listen but gave
an answer based on the presidential election. One respondent gave an answer based on
a governor’s race. Another respondent thought about whether to answer honestly or
not, because he had not voted.
Q2

I am going to read a list of things some people have done to express their views.
Please tell me whether or not you have done any of the following in the last 12
months, that is since April 2010:
(a)

Contacted or visited a public official - at any level of government - to
express your opinion?
(1) Yes
(2) No

(b)

Bought or boycotted a certain product or service because of the social or
political values of the company that provides it?
(1) Yes
(2) No

One respondent answered “No” to Q2a because she had received no response to her
email and telephone efforts to contact people. Another respondent had contacted
someone, but couldn’t remember who.
One respondent had to stop and think to Q2b, then answered “no,” saying she had
boycotted “but not for that purpose,” explaining she didn’t buy anything edible from
China because she doesn’t trust the safety and “they’re going to own us some day.”
Another respondent felt that “boycotted” was too strong a term.

4
Q3

How often, if at all, have you used the Internet to express your opinions about
POLITICAL or COMMUNITY issues within the last 12 months—basically every day,
a few times a week, a few times a month, once a month, less than once a month,
or not at all?
(1) Basically ever day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

Responses to this item raised a number of issues. Respondents described Internet use
that did not appear to have political or community content or communication with
political or community-related content that did not involve the Internet. Respondents
described one-to-one communication with government officials and employees. Some
respondents also described use of the Internet in ways that seemed to stretch the
boundaries of what could be construed as expressing opinions.
Some respondents provided answers to probes about the nature of their reported use
of the Internet that did not appear to contain any political or community-related
content. There respondents described general online communication (using social
networking sites, Twitter, and blogging) without any mention of specific political or
community related activities. They appeared to miss the restriction to political or
community issues in the question. In one case, a respondent included expression of
opinions about political or community issues through non-Internet activities like calling
or writing letters.
Respondents described considerable usage of the Internet for personal communications
to elected representatives or government agencies: complaining to the “Metro people”
about service; emailing the city councilman; contacting council members about the
library; contacting the White House and governor’s office; emailing police regarding
protection of a “Muslim church” as being a waste of taxes; protesting parking fees at a
library; one mentioned communicating once about “health care issues” without further
details. The usage of the Internet in these cases was incidental, except perhaps for
convenience: they could have been made by phone, fax, mail, or in-person. These
responses did not speak to the unique opportunities for one to many communication
offered by the Internet. Only one respondent’s answer to the follow-up question
described behavior that was unambiguously unique to the Internet: commenting on
websites, social networking sites, Twitter, and blogs regarding oil and President
Obama’s citizenship status. One respondent who had answered the question negatively
described behavior that might have fallen under the question’s intent, emailing brothers
about elected officials.

5
Several respondents described activities that seemed to be at the margins of what could
be considered communication. One had signing online petitions after receiving email
from politicians. A respondent who had not used the Internet for these purposes gave
the example of filling out online surveys as a way of expressing opinion, as well as
emailing friends. Another respondent said he rated college professors online.
Q5

Next, I will give you a list of types of groups or organizations in which people
sometimes participate. Have you participated in any of these groups during the
last 12 months, that is since April 2010:
(a)

A school group, neighborhood, or community association such as PTA or
neighborhood watch group?
(1) Yes
(2) No

(b)

A service or civic organization such as American Legion or Lions Club?
(1) Yes
(2) No

(c)

A sports or recreation organization such as a soccer club or tennis club?
(1) Yes
(2) No

(d)

A church, synagogue, mosque or other religious institution or
organization, NOT COUNTING your attendance at religious services?
(1) Yes
(2) No

(e)

Any other type of organization that I have not mentioned?
(1) Yes
(2) No

Q6

Continue
Skip to Q7

What type of organization is that?
Record verbatim

Q5a was relatively challenging for respondents to answer, hinging on the meaning of
“participate.” Two respondents queried the meaning of “participate”: one was the
president of an organization, the other wanted to know if attendance meant
participation. Two respondents asked if a homeowners’ association qualified, and
decided that it did; other respondents asked about a condo owners’ association and

6
decided that it didn’t count. A respondent who volunteered at school but wasn’t on the
PTA paused for a while before deciding that it counted. Another parent who attended
school commissioners’ meetings decided that it didn’t count because the items of
interest were all of a political nature.
For Q5b, one respondent answered positively based on taking eyeglasses to the Lion’s
Club, presumably for some sort of drive. Another respondent asked if the VFW qualified
and decided that it did not. Another respondent said she volunteers for things like
cleaning up the Bay, but that didn’t count because it was not the same as the examples
given.
For Q5c, a respondent who created fliers for a fundraiser for an afterschool program
that encouraged kids to exercise responded affirmatively.
Two respondents needed Q5d repeated before answering.
Three respondents gave food-related charities for Q5e, one secular group and two
feeding the homeless with their churches.
In general, one respondent said that people might count online groups, such as Yahoo!
Groups.
Q7

In the last 12 months, that is since April 2010, have you been an officer or served
on a committee of any group or organization?
(1) Yes
(2) No

In response to a probe about the type of organizations covered, there appeared to be
some confusion among respondents. One respondent was not sure and said the
question could be worded better and provide an example of a specific organization.
Four respondents interpreted the item in political terms. One respondent thought the
question referred to political organizations and the question was asking if he was a
lobbyist. He felt the question was aimed at understanding how best to reach people
through their associations. Another respondent thought that the question was referring
to organizations with a political dimension—those that had a power structure. A
different respondent saw this as involvement in government, a political campaign, or
volunteering for a campaign. Yet another respondent saw this as being about
Democratic or Republican organizations. One respondent, who was on the board of his
college’s alumni association, hesitated before responding and said the use of “officer”
may be confusing and it would be best to ask if people had served on a committee. One
respondent focused on the use of “served” and felt that the question focused on
policemen, firemen, or members of the military.

7
Other respondents, who interpreted the item in the manner intended, drew on the
previous item for context in defining the types of organizations that this question
covered.
Q11

These next questions ask how often you did something during a TYPICAL MONTH
in the last 12 months, that is since April 2011.
How often did you discuss politics with family or friends—basically every day, a
few times a week, a few times a month, once a month, less than once a month,
or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

In all but one case, no difficulties were reported. One respondent, who discussed
politics once a year with family and a little more frequently with friends, had difficulty
answering this question and eventually selected less than once a month. He would have
preferred numerical ranges (e.g., 1-2 times a month).
Q12

How often did you eat dinner with any of the other members of your household—
basically every day, a few times a week, a few times a month, once a month, less
than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

No problems were reported.

8
Q13

This next question is about friends and family you don’t live with. These questions
ask how often you did something during a TYPICAL MONTH in the last 12 months,
that is since April 2011.
How often did you see or hear from friends or family, whether in-person or not—
basically every day, a few times a week, a few times a month, once a month, less
than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

One respondent failed to register “hear from” and “in-person or not.” She first
answered “a few times a week,” then changed her answer to “basically everyday” when
the interviewer repeated the question. One respondent asked if an ex-husband should
be considered family. Two respondents recommended that the question ask about
friends and family separately; a different respondent had a similar response, giving
different frequencies for family and friends.
In response to a probe about what the question was asking about, five respondents
appeared to miss the fact that the question also asked about friends.
In response to a probe asking about how the main ways the respondent was in touch
with friends and family, responses included in-person, via email, Facebook, and via
phone, whether voice calls or by texting.
There was no evidence of respondents incorrectly including family they lived with in
their responses.
Q15

How often did you see or hear from your neighbors, whether in-person or not—
basically every day, a few times a week, a few times a month, once a month, less
than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

There appeared to be some confusion about “hear from.” Three respondents included
simply hearing sounds from their neighbors in “hear from.” Another person would not

9
include simply saying “hi” and “bye” to his neighbors; he thought the question was
about communication with neighbors and all aspects of community, political views,
knowledge of certain situations, etc.
One respondent asked to have the question repeated.
In response to a probe about what respondents thought the question was asking about,
most respondents interpreted the item as measuring closeness of the neighborhood, its
camaraderie, if the neighborhood was an active place, interaction in the community,
“how tight people are.” Four respondents thought the item referred to how well the
respondent was integrated into the community, knew the community, if the respondent
is a loner, or if the respondent is a “community person.” One respondent thought this
question was trying to find out if he’s “living next to someone who’s on the run from the
law, a fugitive.” However, in response to an interviewer probe about what this question
was asking about, the respondent said “How often I interact with people outside my
family.”
Most respondents described seeing them in-person, whether coming or going from
home or around the neighborhood (e.g., at a grocery store); it was not clear from some
responses whether any communication took place besides merely seeing the neighbor.
A single respondent described telephone and email communication besides in-person
communication. Another respondent described twice monthly social gatherings.
Respondents considered people in their immediate neighborhoods (other residents of
multifamily dwelling units, people on their street, people on their block) to be
neighbors. The most exclusive definitions were “people in my building” and the
immediate community of five townhouses, while the most inclusive was that neighbors
could be five blocks down the street but that “it depends” on details that were left
unspecified. One respondent included former neighbors; another respondent who had
moved inside a year referred to the new neighborhood.
Q16

How often did you and your neighbors do favors for each other? By favors we
mean such things as watching each other’s children, helping with shopping,
house sitting, lending garden or house tools, and other small things to help each
other—basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

10
The only concern expressed regarding this question was by a respondent who felt it was
“convoluted,” with too many examples. He felt it would be better phrased, “How often
do you and your neighbors do favors for each other or help each other out?”
Q18

We’d like to know how much you trust people in your neighborhood. Generally
speaking, would you say that you can trust all the people in your neighborhood,
most of the people, some of the people, or none of the people in your
neighborhood?
(1) All the people
(2) Most of the people
(3) Some of the people
(4) None of the people

This question resulted in noticeable pauses by many respondents and some difficulty
choosing between options. The only respondent who answered it without pausing was
president of the homeowners’ association who knows all his neighbors. One respondent
wavered between “most” and “some” before finally choosing “some” based on not
knowing some neighbors as well as others. In response to a probe about how they chose
their response, respondents based their answers on people they knew in the
neighborhood, previous positive interactions like neighbors cooking for one another, or
previous negative interactions like girls who knocked out a car mirror and didn’t tell
anyone, and on neighborhood events like coming together after a school shooting.
Respondents appeared to focus first on neighbors they knew and then made
adjustments for neighbors they didn’t know based on their general level of trust; one
respondent said she had no reason to trust people she hasn’t met, while another said
she had no reason not to trust the neighbors she hadn’t met yet.
There were wide variations in respondent understandings of “trust.” Responses
included: feeling comfortable with your neighbors; “how close I let someone get to
know me;” “everything, like staying at your house alone, taking care of your kids,
whether they steal from you, would they look out for you;” if you could leave a window
open and if whether a neighbor would do a favor when asked; how safe you feel;
allowing kids to play outside, letting workmen into your home if you’re not home, pet
sitting or watching children; do you trust your neighbors enough to give them your key,
leave money out if people are doing maintenance, take care of your car, pick you up if
you have a medical problem; the ability to rely on someone and take them at face value;
reciprocal trust. Respondents generally talked about trust in abstract terms rather than
relating it to specific events. The exceptions were a respondent who accidentally left the
garage door open and nothing was stolen and a person who had talked to neighbors and
done shopping for them.
In general, respondents used a more expansive definition of “neighborhood” for this
item than for “neighbors” in Q16. These included named neighborhoods (SE
Washington, D.C., Glover Park), a half-mile radius, and in the case of the respondent

11
who had cited the five surrounding townhouses in regards to neighbors, more
townhouses. Others cited more limited areas including a block radius, and apartment or
condo buildings. One respondent recommended a more specific definition of
“neighborhood.”
Q19

In an emergency, some people have friends or family they can rely on; others do
not.
How many friends and family members do you have, if any, whom you could
count on in an emergency? Don’t worry if your answer is not exact, just give me
your best estimate.
Accept integers 0 through 9999

This question also was associated with noticeable pauses by respondents.
Respondents had questions about how to define an “emergency.” One respondent
noted that an “emergency” was “pretty broad,” and could be anything from needing to
go to the ER in the middle of the night to asking a neighbor to take care of a cat because
a family member died to providing support in an “emotional crisis.” Most respondents
appeared to include a fairly broad range of events such as arguments, any type of
situation, picking kids up from school, dropping kids off, watching a dog, having
someone to watch the house when you were out of town, babysitting, providing a ride,
loaning money, forgetting one’s key (with a neighbor having a spare), or (as one
respondent reported) anything I need done right now that I can’t do. Respondents also
mentioned crises such as medical emergencies, blackouts, robberies, or car accidents.
Respondents emphasized reliability in their interpretations of “count on;” implicit in this
was willingness to help regardless of the circumstance or inconvenience to the helper.
Two respondents did, however, have questions relating to the person’s ability—rather
than willingness—to help. Both asked if the question intended to count people who
were long distances away. If the question intended physically proximate people, they
would have provided lower estimates.
No respondents indicated that they suffered anxiety as a result of this question.

12
Q20

I am going to read some ways that people get news and information. Please tell
me how often you did each of the following during a TYPICAL MONTH in the past
year, that is since April 2010:
(a)

Read a newspaper in print or on the Internet—basically every day, a few
times a week, a few times a month, once a month, less than once a
month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

(b)

Read a news magazine such as Newsweek or Time, in print or on the
Internet—basically every day, a few times a week, a few times a month,
once a month, less than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

(c)

Watch the news on television or get news from television Internet sites—
basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

13
(d)

Listen to the news on radio or get news from radio Internet sites—
basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

(e)

Obtain news from any other Internet sources that we have not previously
asked about such as blogs, chat rooms, or independent news services—
basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

For Q20b, one respondent asked if the news magazine had to be Newsweek or Time. For
Q20d, a respondent reported listening (apparently inadvertently) to the news when a
music station covers the news. Another respondent said that she would say “every day,”
except that her car radio was broken. A respondent to Q20e asked what an
“independent news service” was; depending on the definition, the answer might
change.

14
Q21

I am going to name some institutions in this country. For each of these
institutions, would you say you have a great deal of confidence, only some
confidence, or hardly any confidence at all in them?
(a) Corporations
(1) A great deal of confidence
(2) Some confidence
(3) Hardly any confidence at all
(e) The media
(1) A great deal of confidence
(2) Some confidence
(3) Hardly any confidence at all
(i) Public schools
(1) A great deal of confidence
(2) Some confidence
(3) Hardly any confidence at all
(j) Religious institutions
(1) A great deal of confidence
(2) Some confidence
(3) Hardly any confidence at all

For Q21a, one respondent said that it would depend on the corporation: some are good
and others just “want the people in ivory tower *to get+ richer.” Another respondent
appeared to take a similar view: the question immediately reminded her of Enron—“I
now wonder if any are honest; gotta believe some of them are.”
One respondent paused to think when answering Q21e, saying “That’s a hard one. I’m
pessimistic about the media right now.” Another respondent had some confidence and
explained that there were several different media sources but each one was biased in its
own way. In response to a probe about why she answered the way she did, a
respondent felt her negative answer had been too broad: all in all, she didn’t distrust
them; lately, have seen some failures [but] everything has the possibility to fail.
One respondent to Q21i felt the question was really broad and also wanted to know if
she should answer for the quality of teachers or the quality of the system. One
respondent cited this question when asked to respond to the debriefing question
concerning which questions were most difficult to answer, mentioning “public school
leadership.”

15
Q21j appeared to elicit considerable problems. A respondent felt that it was hard to
lump all religious institutions together and was not an easy question to answer. One
respondent based her answer solely on the church she attended and responded “a great
deal of confidence.” Two other respondents explicitly excluded their places of worship
from consideration, answering “some confidence” and “hardly any confidence.” Another
respondent asked for the answer options to be repeated. This question was selected by
two respondents as the most difficult question in the survey to answer in the debriefing
questions.
When probed how they would have answered if they really had no opinion, responses
varied. Two would have replied “some confidence,” another would have replied “hardly
any confidence,” and another said she would say she didn’t understand. One didn’t
know what he would do. Other respondents simply didn’t see how one could not have
an opinion.
Respondents gave various answers to a probe about how they would have answered if
they were ambivalent, trusting some institutions in a given category but not others.
Seven said they would report having “some confidence,” while one would report having
“hardly any confidence,” and another did not know. One respondent said it would
depend on the ratio of trusted to distrusted institutions: if it were 50/50, she would say
“some confidence;” if it were 80/20, she would have answered “a great deal of
confidence.”
When asked about “media,” most respondents included television, radio, print, and
Internet media. However, one respondent who read a newspaper each day didn’t think
about it when answering the question.
Respondents offered a range of religious institutions when asked what they had been
thinking about when answering questions. Nine respondents gave answers indicating
that they had thought only about Christianity; three mentioned other religions (one
mentioned churches and synagogues but explicitly excluded mosques for reasons not
given), while the remaining three did not state there views in a way that it was possible
to determine what they had in mind.
Q22

How would you describe your overall state of health these days: Would you say
it is excellent, very good, good, fair, or poor?
(1) Excellent
(2) Very good
(3) Good
(4) Fair
(5) Poor

No difficulties encountered.

16
Q23

On the following scale, how would you describe the quality of your life overall?
Excellent, good, fair, poor, or very poor?
(1) Excellent
(2) Good
(3) Fair
(4) Poor
(5) Very poor

No difficulties encountered.
Responses to Debriefing Items
Overall, would say the survey questions were easy or difficult to answer?
Follow-up:

Why?

12 respondents indicated the questions were easy while another respondent said they
were neither easy nor difficult. One respondent had concerns with the flow of the
instrument and had no idea where it was going, which bothered her. A lot, she felt, had
nothing to do with civic engagement and the questions were all over the place. Another
respondent did not give an overall assessment but stated the question about religion
was difficult. Another respondent also raised religion as being the most difficult
question, while a third mentioned Q21i, regarding the ambiguity of “public school
leadership.”
One respondent felt “sort of bothered” that the survey was “not cohesive, not leading
to a particular end; I had no idea where it was going.” One respondent asked why Q22
and Q23 were asked. One respondent felt long banks of items were difficult to answer
and wanted the item stem repeated at intervals.
Which questions did you have to think about the most to decide on an answer?
Follow-up:

Why was that?

Q21 attracted the most mentions, with two respondents citing the entire bank of items;
one of these said this was because institutions were broad and had several parts,
presumably indicating that because institutions within these broad groupings differ
from one another, it is difficult answer. Another two respondents mentioned Q21j in
particular; one of these felt it was difficult to answer because grouping all religions
together made it difficult to answer.
Q18 attracted two mentions, one because “when you think about the word ‘trust’ a lot
goes into that” and the other because “trust can mean so many things.” A third
respondent mentioned questions about the neighborhood (it was not clear if questions
about neighbors were also included) because he didn’t know how to answer.

17
Items with one mention were Q1 (difficult to decide whether to be honest about not
voting), Q3 (no explanation given), Q5b (no further explanation given), Q7 (served on a
committee but not as an officer), Q19 (regarding who would be called in an emergency),
and Q20e (what “independent news media” meant).
Which questions or topics were [most/least] interesting to you?
Respondents were far more likely to find questions interesting than uninteresting. The
only item/items cited as being least interesting were “political questions” by one
respondent. On the other side of the ledger, three respondents found all the questions
interesting. Three respondents cited questions about neighbors. Two respondents cited
questions about communications and connections with people. Two respondents
mentioned the question about trust in public schools. One respondent mentioned the
question about trust in religious institutions. Finally, three respondents indicated that
they did not find any questions to be particularly interesting or uninteresting.
Do you think there are questions people would find difficult?
Follow-up:

Which ones?

Follow-up:

Why?

Religion was the most mentioned question, with four respondents citing it as most
difficult; one respondent explained this was because it was “sensitive.” Another topic
sometimes deemed unsuitable for dinner conversations, politics, was cited by two
respondents. Other topics were mentioned once. One respondent was concerned
because he did not want to jeopardize his security clearance and because people from
other countries might find it sensitive. Two respondents mentioned the media, one in
generic terms and the other because “media” was broad and difficult to generalize. One
respondent mentioned public schools, either because people might not be involved or
be reluctant to criticize them. One respondent felt that the item on helping neighbors
was too long. Another thought questions on one’s neighborhood would be difficult. One
mentioned community organizations, because they were difficult to define. Trust also
received a single mention. The question on volunteering was cited once because people
sometimes just do things without being organized in a group. Finally a respondent felt
that a lot of questions were open-ended and could be answered in different ways, e.g.,
one’s neighborhood and the media (these mentions are accounted for above).

18
Sometimes it is necessary to ask people to respond on behalf of other members of the
household if someone isn’t home. Do you think there are questions you would have
trouble answering on someone else’s behalf?
Follow-up:

Which ones?

Follow-up:

Why?

Respondents differed sharply from each other regarding whether they could answer on
someone else’s behalf. Six respondents felt they would not have problems answering on
another person’s behalf, although one qualified this by saying that it would depend on
how long you had been with someone (he had been with his partner for seven years and
could answer on his partner’s behalf. Four respondents indicated that they could
answer most questions; two cited questions regarding trust as being impossible to
answer for another person, two mentioned public schools, one mentioned religion, and
one mentioned questions on involvement and participation. Four respondents said that
they would not answer questions on another’s behalf; three of these were because it
was not the right thing to do, while the other said that she was a very private person
and did not know how other people could answer for her.
Do you think there are questions that people would find sensitive?
Follow-up:

Which ones?

Follow-up:

Why?

The most commonly cited question that people would find sensitive was religion, with
four respondents mentioning it. Three said “political” topics, and another one
mentioned voting. Two respondents felt that people would be sensitive to questions
about health; one explained this was for fear of the information being leaked to
insurance companies. One respondent thought that trust in corporations might be
sensitive. One respondent said all questions would be sensitive. Another said some
questions were very private, but was unable to give any examples. Finally, three
respondents saw no items as being sensitive.
Recommendations
Item-Specific Recommendations
Voting in local elections
The revised wording of Q1 (“The first question is about LOCAL elections, such as for
mayor or a school board. Do you always vote in local elections, do you sometimes vote,
do you rarely vote, or do you never vote?”) appears to have been more successful than
the original wording (“In any election, some people are not able to vote because they
are sick or busy or have some other reason, and others do not want to vote. Thinking

19
about recent local elections, have you voted in all of them, most of them, some of them,
or none?”). Interviewer instructions should be written for the new wording.
Activities expressing social and political views
Items use wording from previous years. Interviewer instructions appear sufficient.
Use of Internet for communication on political and communal issues
The appropriateness of responses to this question depends heavily on question intent,
which we were unable to get a clear understanding of. Much of the usage describes
direct communication between a private individual and an elected representative or
employee of a government agency. Fewer responses describe public speech on
communal or political issues. We recommend that CNCS develop a very clear statement
of intent for use in training interviewers, particularly with respect to the nature of
expressing opinion and the degree to which such speech needs be public. On the nature
of communication, is it necessary that the respondent have composed a
communication? Would signing a petition count? Would filling out an online survey be
sufficient (e.g., a government customer satisfaction survey)? The other important
distinction is to make clear to interviewers whether various types of private
communication count: private communication to friends (e.g., email) and private
communication to politicians or government officials.
CNCS should also consider the status of borderline cases associated with the
convergence of traditional and Internet communication technologies: should a phone
call expressing an opinion initiated by clicking a phone number on a website and carried
out over the Internet without regular telephony count?
It is not clear how the item text could be adjusted in response to respondents who
described use of the Internet that did not involve expressing opinions on political and
community issues or described expression of opinion on political or community issues
that did not involve the Internet. The item is relatively straightforward and already
emphasizes “POLITICAL or COMMUNITY issues.” At the expense adding an additional
item, an item on Internet use in general could precede the item on expression of
opinion, giving respondents an outlet to describe their general use of the Internet and
highlighting the specific type of Internet use this item describes. However, given the
space limitations for this survey, if this item is retained, we recommend CNCS develop
interviewer training material that highlights the tendency for some respondents to
interpret the item more broadly than intended.
Participation in civic organizations
This bank of items is largely unchanged from previous surveys. In both rounds of testing,
respondents had numerous questions for interviewers regarding the types of
organizations that should be considered appropriate and what constituted participation.

20
Respondent interpretations regarding participation appear to be somewhat looser than
the standard called for in interviewer training, which excludes “simply attending
organization or club meetings.” While interviewer instructions appear to be
comprehensive, this bank of items should be flagged for particular attention by
interviewers who should be alert to possible misinterpretation by respondents.
Serving as an officer or committee member of an organization
This item was surprisingly difficult for respondents to interpret, with nearly a third of
Round 2 respondents seeing this in a political light. It is not clear, based on the question
wording and the previous item, as to how these respondents arrived at this
interpretation. We recommend adding emphasis to the closing phrase: “ANY GROUP OR
ORGANIZATION.” The interviewer instructions appear to be comprehensive.
Discussing politics with family and friends
The wording of this item differs slightly from that asked in previous surveys. Wording
changed from “How often were politics discussed when communicating with family and
friends” to “How often did you discuss politics with family or friends.” Respondents did
not appear to encounter difficulties when responding to this item; the one respondent
who paused was eventually able to answer correctly. Interviewer instructions appear to
be appropriate.
Eating dinner with other members of the family
This question was asked on previous surveys. The wording has not been changed. No
problems were noticed during cognitive testing. Interviewer instructions seem suitable.
Seeing or hearing from family and friends
At CNCS’ request, this item combined elements of the following items asked in Round 1:
“During a TYPICAL MONTH in the past year, how often, if at all, did you spend time
visiting friends—basically every day, a few times a week, a few times a month, once a
month, or not at all?” and “During a TYPICAL MONTH in the past year, how often, if at
all, did you communicate with friends and family by email or on the internet—basically
every day, a few times a week, a few times a month, once a month, or not at all?” Some
respondents appear to miss the “see or hear” and “in-person or not.” We had previously
expressed concerns that this item might give rise to underestimates of behavior
compared to asking about in-person and Internet communication separately. These
concerns did not appear to be borne out by cognitive testing, with respondents
describing multiple modes of communication. Unexpectedly, many respondents
appeared to interpret the question as asking only about family. It may be desirable to
remove the mention of the timeframe from the fairly long question preamble (“This
next question is about friends and family you don’t live with. These questions ask how
often you did something during a TYPICAL MONTH in the last 12 months, that is since

21
April 2011.”) Another possible source of this error is the use of “friends or family”
(emphasis added) in the question stem, which carries connotations that respondents
choose one or the other. Changing this to “friends and family” (emphasis added) would
be consistent with other usage in the instrument. Interviewer training should emphasize
the inclusion of both friends and family in responses.
Seeing or hearing from neighbors
This combined a preexisting item, “How often did you talk with any of your neighbors—
basically every day, a few times a week, a few times a month, once a month, or not at
all?” and a new item tested in Round 1, “During a TYPICAL MONTH in the past year, how
often, if at all, were you in the home of a neighbor–basically every day, a few times a
week, a few times a month, once a month, or not at all?” Respondents’ understanding
of this item showed considerable variation and confusion. Because of the natural
tendency to hear noises emanating from one’s neighbors’ properties, a number of
respondents interpreted “hear from” as “hear.” It was also unclear in some responses
whether seeing one’s neighbors included any form of communication. On balance, we
recommend retaining the existing item asking about talking with one’s neighbors, which
did not exhibit problems in Round 1 testing. In addition, we feel that there would be a
relatively high degree of overlap between the talking to neighbors item and visiting
neighbors’ homes item and, should limited space be available, relatively little additional
information would be lost by foregoing the item on visiting with neighbors. Some
respondent understandings of who was a neighbor were possibly broader than question
intent; training material describes the nearest 10 to 20 households to the respondent.
Assuming that 10 to 20 households is consistent with CNCS question intent, training
material can be retained unaltered.
Doing favors for neighbors
This item appeared to function as intended. The one respondent who felt the question
was needlessly convoluted was nevertheless able to answer it correctly. No changes are
needed.
Trust people in neighborhood
There was a wide range in respondent understandings of trust. One group gave
definitions that might be described as “affirmative trust” (to coin a phrase) or trust in
gemeinschaft terms: trusting neighbors to help oneself or the existence or reciprocal
trust. The other group seemed to interpret trust as in terms associated with a
gesellschaft, being the absence of distrust. The gemeinschaft definition of trust in terms
of reciprocal obligations appears to be measured quite effectively by the preceding
question on doing favors. We recommend that CNCS develop interviewer instructions
focusing on trust as the absence of distrust. In addition, interviewer instructions should
provide a definition of “neighborhood.”

22
People to count on in an emergency
Many respondents interpreted emergencies as covering a range of everyday situations
such as losing keys or needing pets minded. If the purpose of the question is to measure
social capital, these definitions of emergencies may be appropriate. However, the
variation in respondent understanding of emergencies will introduce undesirable
variance in responses not associated with substantive variations in social capital. It is,
however, difficult to predict what the effect might be. On the one hand, there is a
stronger social desirability effect for helping people experiencing severe emergencies
(e.g., destruction of one’s house by fire; needing childcare for an emergency trip to
hospital), than more workaday ones (e.g., minding children when the babysitter cancels
unexpectedly). On the other hand, the level of effort required to assist in low
consequence emergencies like forgetting one’s key is probably lower than in high
consequence emergencies like severe illness. We recommend that CNCS define an
emergency based on the level of consequences associated with it and develop
interviewer training material based on this.
Media use
These questions performed well. The only exception to this was with respect to
“independent news services,” which a respondent did not understand. The existing
training material does not include a definition of independent news services and should
be revised to include one.
Trust in institutions
This bank of questions attracted more negative comment than any other. Respondent
feedback included concerns about diversity of institutions within each grouping and the
sensitivity of particular items. Various respondents mentioned that particular groupings
of institutions were very diverse, making it difficult to answer appropriately. Religion
was mentioned in these terms, as were corporations and the media. Public schools were
described by some respondents as being difficult to answer for other reasons: some
people might not have experience with them; one might be dissatisfied with certain
elements of the public school system (e.g., teachers) and not others (e.g., leadership).
Religion was mentioned repeatedly as being sensitive. These comments did not address
the ways in which answering questions on trust in religious institutions would be
sensitive, instead treating anything to do with religion as being fraught.
Construct validity of this item does appear to be problematic as the “institutions”
referred to in the question stem are indeed, as respondents reported, very diverse
groupings of institutions about which it is difficult to report an overall degree of trust.
This item might be better changed from overall trust in institutions to the proportion of
institutions trusted within each grouping. The introduction could be rewritten along the
following lines:

23
I am going to name some types of institutions in this country. For each of these
types of institutions, would you say you trust all of the institutions, most of the
institutions, some of the institutions, or none of the institutions?
(a)

Corporations
(1)
(2)
(3)
(4)

(b)

The media
(1)
(2)
(3)
(4)

(c)

Trust all media outlets
Trust most media outlets
Trust some media outlets
Trust no media outlets

Public schools
(1)
(2)
(3)
(4)

(d)

Trust all corporations
Trust most corporations
Trust some corporations
Trust no corporations

Trust all public schools
Trust most public schools
Trust some public schools
Trust no public schools

Religious institutions
(1)
(2)
(3)
(4)

Trust all religious institutions
Trust most religious institutions
Trust some religious institutions
Trust no religious institutions

Given the level of respondent concern about expressing trust in religious institutions,
CNCS may wish to remove this item. If this bank of items is included in the final
instrument, CNCS will need to develop interviewer instructions.
State of health and quality of life
Respondents were able to answer these last two items appropriately. However, some
respondents wondered why these questions were asked, given the survey’s stated focus
on civic engagement. CNCS may wish to consider adding a transition to these last two
items to make the topic switch less jarring. The final two questions could introduced by
a sentence contextualizing their inclusion in the CES, such as: “People’s ability to
participate in civic life can be affected by their health.”

24
Proxy Reporting
We understand that proxy reporting will be used in the November 2011 CPS and that
the accuracy of proxy reporting for the Civic Engagement Supplement is of some
concern. Appendix B contains a review of literature on proxy reporting and detailed
application of the findings of the literature review to the questions used for Round 2
testing. We summarize Appendix B here.
Items that are attitudinal or are not observed by other household members (e.g., quality
of life) are at the greatest risk of proxy reporting errors. Items that focus on behaviors
that may not be fully observed by proxy reporters (e.g., Internet usage) are at moderate
risk of proxy reporting errors. Items that are easily observable by proxy reporters (e.g.,
eating dinner together/separately), likely to be known to the proxy reporter (e.g.,
serving on a committee), or are specific are at low risk of proxy reporting errors. When
items concern behavior that may be subject to social desirability, proxy reporting may
be more accurate than self-reports.
We review items by degree of a priori concern below. However, given the likely
substantial number of proxy reports, we recommend additional testing to better
understand the biases that may be associated with each item. Such designs could
include separate simultaneous interviews with both self- and proxy reports of couples in
a laboratory setting and/or random assignment designs. Laboratory designs have
increased control over extraneous factors and permit direct comparisons between selfreports and proxy reports, increasing internal validity and enabling the use of more
statistically powerful paired sample statistics. Overall numbers of interviews in
laboratory designs are, however, smaller due to limited capacity, reducing statistical
power. Random-assignment designs dividing households into proxy and self-reporting
conditions are subject to various confounds to internal validity due to limited control
over respondents and risk of nonresponse error due to differential response rates
between conditions, and the use of paired sample statistics is not possible. Larger
sample sizes are possible, increasing statistical power, and external validity is enhanced
by recruitment that mirrors survey conditions.
High risk items
Q18 (trust in neighborhood) and Q21 (trust in institutions)—both items are attitudinal
and it will be difficult for proxy reporters to report accurately.
Moderate risk items
Q19 (number of people relied upon in an emergency)—judgments about whether
someone can be relied upon are predominantly attitudinal, and some error would be
expected. In addition, the observed variation in respondent definitions of what
constituted an emergency would likely introduce additional error. As a result, this item
is probably most problematic of the moderate risk items.

25
Q3 (use of Internet to express opinions), Q11 (discuss politics with family and friends),
and Q20 (sources of news and information)—these items focus on behaviors that are
likely not to be fully observed by proxy reporters, who thus may be unaware of
frequency of behavior, and may therefore lead to underestimates.
Q23 (quality of life)—evaluation of quality of life is quite subjective and we would
expect a substantial amount of error as a result.
Q22 (state of health)—in general, we would expect health to be reasonably well known
to proxy reporters; however, certain aspects of health, such as pain and mental health,
are not as easily observed and there is some person-to-person variation in
interpretation of vague qualifiers like “good” and “fair.” This item is the least
problematic of the moderate risk items.
Low risk items
Q1 (voting in local elections), Q2a (contact public official), Q2b (boycott product or
service), Q5 and Q6 (participation in groups), Q7 (organizational leadership), Q12 (eat
dinner with household members), Q13 (frequency of seeing or hearing from friends or
family), Q15 (frequency of seeing or hearing from neighbors), Q16 (favors for
neighbors)—these items all concern relatively easily observed behaviors and are at low
risk of error from proxy reporting.

26
Appendix A: Cognitive Testing Protocol
Participant ID #:

| | | | | |

Interview Date:

| | | / | | | / | 2 | 0 | 1 | 1 | (mm/dd/yyyy)

Interviewer Initials:

| | |

Start Time: ____________ AM / PM

End Time: ____________ AM / PM

Section 1: Interviewer: Read/ Paraphrase the following text:

Hello. My name is ________________. I work for Abt Associates, a research
company that does work under contract mainly for federal agencies. Thank you
for agreeing to participate in our study.
Human subjects requirement (prior to starting the recorder)
Confidentiality: Before we begin, I need to be sure you understand that our session today is
completely confidential. Your participation in this study is completely voluntary
and you can decline to answer any particular question.
Incentive:

In appreciation for your participation, you will receive an American Express gift
card for $40.

Recording:

So I don’t have to rely on my memory later on, this session is being video
recorded. That way, I can focus today on what you’re saying rather than having
to concentrate on taking notes.

Observers:

Some members of our research team may be observing the interview.
** HAVE PARTICIPANT READ THE CONSENT FORM**
Before you sign this, do you have questions or see anything that is not clear?
** HAVE PARTICIPANT SIGN THE CONSENT FORM**

START RECORDING

27
Describe the interview objectives and procedures. (This should begin a conversational
interaction with the respondent; it is not necessary to read these descriptions verbatim.):
Before I go into the details of what we’ll be doing, I’d like you to tell me what
you were told about why you were asked to come in today.
(INTERVIEWER: Confirm respondent’s understanding if it is correct. If there are
misconceptions, let the respondent know that you’ll clarify some things as you
describe the plan for the interview session.)
In order to help us improve our surveys, we turn to people like you to find out if
our questions make sense and are fairly easy to understand and answer. We
have found that the best way to do that is to actually conduct the survey with
people and see how it works for them. So you will be helping us test a
questionnaire from one of our surveys.
How:

I want you to answer the questions exactly the way you would if an interviewer
had phoned you at home for an interview, but with one major difference: I
would like you to tell me your thoughts as you decide on your answers.

Think aloud:

I would like you to think aloud as you answer the questions. I am interested in
your answers, but I am also interested in the thoughts that occur to you as you
answer the questions. I would like you to tell me everything that you are
thinking and feeling as you go about answering each question.
I don’t want you to try to plan what you say. Whatever you’re thinking as you
decide on your answer is what I’d like to hear.
Usually, it helps to try this out once or twice before we get to the survey
questions.
I’d like for you to think aloud as you decide how to answer the question: “How
many windows are there in your house or apartment?”
(INTERVIEWER: Use this second practice question if necessary: “Thinking
about yesterday, starting with the time you got up until you went to bed, how
many phone calls did you make?”)

Probes:

As we go through the survey, from time to time I’ll ask you some questions
about your answers, or about the questions themselves. Remember, there are
no right or wrong answers.
This is only a draft of the questionnaire, and we expect that it needs
improvements.
I really want to hear your opinions and reactions, so don’t hesitate to speak up
whenever something is unclear, is hard to answer, or doesn’t seem to apply to

28
you. Of course, if you think a question is especially interesting or useful, we’d
also like to hear about that.
Do you have any questions before we begin?

(INTERVIEWER: Wait for respondent to complete thinking aloud or commenting
before asking the probes.)
The next set of questions is about people’s involvement and communication within their
communities.
Q1

The first question is about LOCAL elections, such as for mayor or a school board.
Do you always vote in local elections, do you sometimes vote, do you rarely vote,
or do you never vote?
(1) Always vote
(2) Sometimes vote
(3) Rarely vote
(4) Never vote

Probe:
Q2

Can you tell me which elections you were thinking about when you
answered this question? (Ask about actual election years.)
I am going to read a list of things some people have done to express their views.
Please tell me whether or not you have done any of the following in the last 12
months, that is since April 2010:
(a)

Contacted or visited a public official - at any level of government - to
express your opinion?
(1) Yes
(2) No

(b)

Bought or boycotted a certain product or service because of the social or
political values of the company that provides it?
(1) Yes
(2) No

No probes. Preexisting questions.

29
Q3

How often, if at all, have you used the Internet to express your opinions about
POLITICAL or COMMUNITY issues within the last 12 months—basically every day,
a few times a week, a few times a month, once a month, less than once a month,
or not at all?
(1) Basically ever day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

Probe:

(Ask unless “not at all”) Can you give me an example or two of how you
used the Internet in this way?
Follow-up:

Probe:
Q5

(Ask if necessary) I’m not looking to know what your
specific VIEWS were, but could you tell me what ISSUES
you were talking about?

(Ask if “not at all”) Can you tell me some ways in which you might use
the Internet to express your opinions?
Next, I will give you a list of types of groups or organizations in which people
sometimes participate. Have you participated in any of these groups during the
last 12 months, that is since April 2010:
(a)

A school group, neighborhood, or community association such as PTA or
neighborhood watch group?
(1) Yes
(2) No

(b)

A service or civic organization such as American Legion or Lions Club?
(1) Yes
(2) No

(c)

A sports or recreation organization such as a soccer club or tennis club?
(1) Yes
(2) No

30
(d)

A church, synagogue, mosque or other religious institution or
organization, NOT COUNTING your attendance at religious services?
(1) Yes
(2) No

(e)

Any other type of organization that I have not mentioned?
(1) Yes
(2) No

Continue
Skip to Q7

No probes. Preexisting questions.
Q6

What type of organization is that?
Record verbatim

No probes. Preexisting question.
Q7

In the last 12 months, that is since April 2010, have you been an officer or served
on a committee of any group or organization?
(1) Yes
(2) No

Probe:
Q11

What sorts of groups or organizations do you think this question is
asking about?
These next questions ask how often you did something during a TYPICAL MONTH
in the last 12 months, that is since April 2011.
How often did you discuss politics with family or friends—basically every day, a
few times a week, a few times a month, once a month, less than once a month,
or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

No probes. Preexisting question.

31
Skip to Q13 if one person household.
Q12

How often did you eat dinner with any of the other members of your
household—basically every day, a few times a week, a few times a month, once
a month, less than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

No probes. Preexisting question.
This next question is about friends and family you don’t live with. These questions ask
how often you did something during a TYPICAL MONTH in the last 12 months, that is
since April 2011
Q13

How often did you see or hear from friends or family, whether in-person or
not—basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

Probe:

Would you tell me, in your own words, what this question wants to find
out?

Probe:

What were the main ways you were in contact with them?

32
Q15

How often did you see or hear from your neighbors, whether in-person or not—
basically every day, a few times a week, a few times a month, once a month, less
than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

Probe:

Would you tell me, in your own words, what this question wants to find
out?

Probe:

What were the main ways you were in contact with your neighbors?

Probe:

Can you tell me who you considered to be neighbors when you
answered this question?

Probe:

How long have you lived at your current address?
Follow-up:

Q16

(Ask if less than a year) Did you think mostly about your
current neighborhood, your last neighborhood, or both
neighborhoods?

How often did you and your neighbors do favors for each other? By favors we
mean such things as watching each other’s children, helping with shopping,
house sitting, lending garden or house tools, and other small things to help each
other—basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

No probes. Preexisting question.

33
Q18

We’d like to know how much you trust people in your neighborhood. Generally
speaking, would you say that you can trust all the people in your neighborhood,
most of the people, some of the people, or none of the people in your
neighborhood?
(1) All the people
(2) Most of the people
(3) Some of the people
(4) None of the people

Probe:

(Unless the respondent answered “None”) How did you decide on
[choice] rather than [next closest option]?

Probe:

What did you have in mind when thinking about trust?

Probe:

What do you think of as “your neighborhood”?

Q19

In an emergency, some people have friends or family they can rely on; others do
not.
How many friends and family members do you have, if any, whom you could
count on in an emergency? Don’t worry if your answer is not exact, just give me
your best estimate.
Accept integers 0 through 9999

Probe:

What sorts of things do you think the question means by “emergency”?

Probe:

What does “count on” mean to you in this question?

Probe:

How many of the [number >1 mentioned] are [friends/family]?

Q20

I am going to read some ways that people get news and information. Please tell
me how often you did each of the following during a TYPICAL MONTH in the past
year, that is since April 2010:
(a)

Read a newspaper in print or on the Internet—basically every day, a few
times a week, a few times a month, once a month, less than once a
month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

34
(b)

Read a news magazine such as Newsweek or Time, in print or on the
Internet—basically every day, a few times a week, a few times a month,
once a month, less than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

(c)

Watch the news on television or get news from television Internet sites—
basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

(d)

Listen to the news on radio or get news from radio Internet sites—
basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

35
(e)

Obtain news from any other Internet sources that we have not previously
asked about such as blogs, chat rooms, or independent news services—
basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Less than once a month
(6) Not at all

No probes. Preexisting questions.
Q21

I am going to name some institutions in this country. For each of these
institutions, would you say you have a great deal of confidence, only some
confidence, or hardly any confidence at all in them?

[INTERVIEWER: Repeat the response options unless the respondent starts to answer
before you read them; or if the respondent appears to forget the options.]
(a) Corporations
(1) A great deal of confidence
(2) Some confidence
(3) Hardly any confidence at all
(e) The media
(1) A great deal of confidence
(2) Some confidence
(3) Hardly any confidence at all
(i) Public schools
(1) A great deal of confidence
(2) Some confidence
(3) Hardly any confidence at all
(j) Religious institutions
(1) A great deal of confidence
(2) Some confidence
(3) Hardly any confidence at all

36
Probe:

(Ask if “some confidence” reported for one or more institutions) If you
really didn’t have an opinion about *insert from list that was answered
“some”+, how would you have answered?

Probe:

What were you thinking about when you said you had [response
choice] in [insert from list]?

Probe:

How would you answer about [insert from list] if you had a great deal of
confidence in some [insert from list] but hardly any confidence in
others?

Probe:

When I asked about “the media,” what came to mind?

Probe:

How about when I asked about “religious institutions”? What came to
mind for religious institutions?

Q22

How would you describe your overall state of health these days: Would you say
it is excellent, very good, good, fair, or poor?
(1) Excellent
(2) Very good
(3) Good
(4) Fair
(5) Poor

No probes. Preexisting question.
Q23

On the following scale, how would you describe the quality of your life overall?
Excellent, good, fair, poor, or very poor?
(1) Excellent
(2) Good
(3) Fair
(4) Poor
(5) Very poor

No probes.

37
DEBRIEFING QUESTIONS
Overall, would say the survey questions were easy or difficult to answer?
Follow-up:

Why?

Which questions did you have to think about the most to decide on an answer?
Follow-up:

Why was that?

Which questions or topics were [most/least] interesting to you?
Do you think there are questions people would find difficult?
Follow-up:

Which ones?

Follow-up:

Why?

Sometimes it is necessary to ask people to respond on behalf of other members of the
household if someone isn’t home. Do you think there are questions you would have
trouble answering on someone else’s behalf?
Follow-up:

Which ones?

Follow-up:

Why?

Do you think there are questions that people would find sensitive?
Follow-up:

Which ones?

Follow-up:

Why? [This is of particular concern to Census. Please get
respondents to elaborate on their concerns.]

38
Appendix B: Review of Proxy Reporting
A sizable proportion of the responses to the supplement will be reported by a proxy
adult. This document gives a brief overview of the proxy versus self response literature
and then discusses the implications for each proposed item in the 2011 Civic
Engagement Supplement.
Minimal Effects of Proxy Reporting on Voter Turnout in CPS Supplements
The majority of the research literature on proxy reporting focuses on health surveys
rather than surveys related to civic engagement. One exception is a study by Highton
(2005) that investigates proxy responding for voting in the Current Population Survey
Voter Supplements of 1992, 1996, and 2000. For about 40% of the CPS sample, turnout
is reported by proxy (one member of a household reports for another member).
Highton’s results are generally optimistic. He found that although proxy-reported
turnout is 4 percentage points lower than self-reported turnout, the individual-level
correlates of turnout and interstate turnout differences appear mostly similar for the
two measures. That said, for other measures, especially attitudinal ones, there is
theoretical and empirical support for the idea that proxy reports may be less accurate
than self reports in some situations.
Theoretical Differences between Self and Proxy Reports
There are several reasons why proxy reporting may be less accurate than self reporting.
Proxy reports may be incomplete because the proxy may simply lack knowledge about
the event or characteristic in question. In some cases, proxy reports may be biased, such
as if the proxy is only able to recall serious or major incidences of an event in question.
There are also several reasons why proxy reporting may be more accurate than self
reporting in some situations. Perhaps the most compelling reason is that respondents
may not feel as much social desirability pressure when responding about someone else
as when responding about themselves. Another reason is that for some domains, the
proxy respondent may be the “household expert.”
Anchoring in attitudinal proxy reports
Schwarz and Wellens (1997) suggest that proxy respondents draw on different
information than self-respondents in carrying out the cognitive tasks required in
surveys. This is partly because proxy respondents, in general, do not have as much
information about the target as self-respondents do. Some researchers (Bickart et al.
1994) have posited that for attitudinal questions, proxy respondents may use
information about their own attitude in forming the proxy report. They may use their
own attitude as an initial estimate and adjust it according to their perception of
similarities and differences between themselves and the target respondent.
“Anchoring” strategies of this type have been show to result in bias because the
adjustments are usually insufficient (Tversky and Kahneman 1973). For example, Davis

39
et al. (1986) found that respondent anchored heavily on their own preferences when in
predicting their spouse’s preference for consumer products. Interestingly, the authors
noted that the proxy respondents would have increased the accuracy of their reports if
they anchored even more heavily on their self report because of the similarities
between spouses in terms of their preferences. As Bickart et al. (1994) note, “this
suggests that anchoring may compensate for individual’s tendency to focus on
dispositional determinant of other’s behaviors or attitudes at the expense of situational
influences, which they are more likely to consider in self-reports (Watson 1982). Thus, at
least when a proxy reporter and the target are likely to be similar, we may want to
encourage proxy respondents to use an anchoring strategy.”
Empirical Differences between Self and Proxy Reports
In spite of doubts about the quality of proxy reports, past research comparing self and
proxy responses has not found consistent differences in favor of self-responses over
proxy responses. This failure may have resulted from (1) nonrandom selection of the
respondent, which only a few studies have addressed (Lee et al. 2004; Moore 1988;
Mathiowetz and Groves 1985; Blair et al. 1991); (2) failure to distinguish different types
of proxies such as spouses, other family members, more distant relatives, roommates,
and neighbors, whose depth and breadth of shared information are likely to vary
dramatically; and (3) variation in the survey topics on which self- and proxy reports were
compared. As Moore (1988) points out, many of the comparisons reported in the
literature are difficult to interpret because proxies have been used only when the
respondent is unavailable. Thus, reporting and sample biases are intermixed. Further, in
most of these studies a validation source for the behaviors and events of interest is not
available.
Summary
In general, proxy reports are likely to be more accurate if the information requested:
-

is observable (e.g., employment status [Deighton 1976; Roman and Woltman
1980])
is likely to be known by the proxy respondent (e.g., spouse’s occupation and
mode of transportation to work [Martin and Butcher 1982])
is specific (Bickart et al. 1991)

In general, proxy reports are likely to be less accurate if the information requested…
-

is attitudinal
is not-directly observable (e.g., quality of life [Tamim et al. 2002], disabilities [Lee
et al. 2004], crime victimization [Turner 1972])

40
Can the Supplement Questions Be Answered Accurately by Proxy Respondents?
The classification of “high” “medium” or “low” concern is a preliminary assessment to
make a distinction between the most/least troublesome questions in terms of proxy
error based on the literature reviewed above.
The next set of questions is about people’s involvement and communication within their
communities.
Q1

The first question is about LOCAL elections, such as for mayor or a school board.
Do you always vote in local elections, do you sometimes vote, do you rarely vote,
or do you never vote?

Concerns about proxy responding: Low concern. We have minimal concern because
Highton (2005) investigated proxy reporting of voter turnout in CPS supplements and
found that although proxy-reported turnout is 4 percentage points lower than selfreported turnout, the individual-level correlates of turnout and inter-state turnout
differences appear mostly similar for the two measures.
Q2

I am going to read a list of things some people have done to express their views.
Please tell me whether or not you have done any of the following in the last 12
months that is since April 2010:
(a)

Contacted or visited a public official - at any level of government - to
express your opinion?

Concerns about proxy responding: Low concern. If someone in the household felt
strongly enough about an issue to contact an official, it seems likely that they would
have discussed the matter with the other members of their household. That said, in
some households such discussion may not take place and so proxy respondents may
underreport this behavior. Another possibility, however, is that some view this activity
as socially desirable and so proxy reporting may actually be more accurate in some
cases because social desirability pressure is generally believed to be lower for proxy
respondents than self-respondents.
(b)

Bought or boycotted a certain product or service because of the social or
political values of the company that provides it?

Concerns about proxy responding: Low concern. Similar to item (a) if someone in the
household felt strongly enough about a product/service to boycott it, it seems likely that
they would have discussed the matter with the other members of their household.
However, in some households such discussion may not take place and so proxy
respondents may underreport this behavior somewhat. We may expect proxy underreporting to be somewhat more severe here than in (a) because boycotting is a more
passive, less observable action than contacting an official. So the chance that the other

41
household members are aware of the boycott may be lower. That said, to the extent
that this is seen as a socially desirable activity, the proxy reports may be more accurate
in a sense.
Q3

How often, if at all, have you used the Internet to express your opinions about
POLITICAL or COMMUNITY issues within the last 12 months—basically every day,
a few times a week, a few times a month, once a month, less than once a month,
or not at all?

Concerns about proxy responding: Moderate concern. While some Internet use for this
purpose will be known to other household members, it seems plausible that the other
household members would not be aware of some of the activities in question. For
example, if the target respondent emails a friend or posts a comment on a website, they
might not necessarily tell all of the other adults in the household. Thus, there seems to
be a potential here for the proxy respondents to underreport this activity. However, to
the extent that this is seen as a socially desirable activity, the proxy reports may be
more accurate.
Q5

Q6

Next, I will give you a list of types of groups or organizations in which people
sometimes participate. Have you participated in any of these groups during the
last 12 months, that is since April 2010:
(a)

A school group, neighborhood, or community association such as PTA or
neighborhood watch group?

(b)

A service or civic organization such as American Legion or Lions Club?

(c)

A sports or recreation organization such as a soccer club or tennis club?

(d)

A church, synagogue, mosque or other religious institution or
organization, NOT COUNTING your attendance at religious services?

(e)

Any other type of organization that I have not mentioned?

What type of organization is that?

Concerns about proxy responding: Low concern. Participation in groups like this is
generally observable by other household members. It seems reasonable to assume that
people are generally aware of the groups, teams, etc. to which their household
members belong. Also, to the extent that participation in such groups is seen as a
socially desirable activity, the proxy reports may be more accurate.

42
Q7

In the last 12 months, that is since April 2010, have you been an officer or served
on a committee of any group or organization?

Concerns about proxy responding: Low concern. Activities like serving on committees
are generally observable by other household members although there may be some
instances where the proxy respondent is not aware of such service.
Q11

These next questions ask how often you did something during a TYPICAL MONTH
in the last 12 months, that is since April 2011. How often did you discuss politics
with family or friends—basically every day, a few times a week, a few times a
month, once a month, less than once a month, or not at all?

Concerns about proxy responding: Moderate concern. Similar to Q3, there is some
concern here that the proxy respondent may be unaware of a substantial number of
instances of this behavior. They would generally know about political conversations with
family but not necessarily conversations with friends.
Q12

How often did you eat dinner with any of the other members of your
household—basically every day, a few times a week, a few times a month, once
a month, less than once a month, or not at all?

Concerns about proxy responding: Low concern. We expect very few problems from
proxy reporting here because the behavior in question would typically involved both the
target and proxy respondent.
This next question is about friends and family you don’t live with. These
questions ask how often you did something during a TYPICAL MONTH in the last
12 months, that is since April 2011.
Q13

How often did you see or hear from friends or family, whether in-person or
not—basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?

Concerns about proxy responding: Low concern. We would expect proxy respondents to
be aware of most though not all instances of this type of behavior. In general, the error
from proxy reporting is not expected to be very large on this measure, and to the extent
that this is seen as a socially desirable activity, the proxy reports may be more.
Q15

How often did you see or hear from your neighbors, whether in-person or not—
basically every day, a few times a week, a few times a month, once a month, less
than once a month, or not at all?

Concerns about proxy responding: Low concern. Same evaluation as Q13.

43
Q16

How often did you and your neighbors do favors for each other? By favors we
mean such things as watching each other’s children, helping with shopping,
house sitting, lending garden or house tools, and other small things to help each
other—basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?

Concerns about proxy responding: Low concern. The concerns here are highly similar to
those shown for Q13 and Q15. That said, this kind of interaction seems likely to be even
more salient than the activities asked about in Q13 and Q15, and so proxy reporting may
be more accurate here.
Q18

We’d like to know how much you trust people in your neighborhood. Generally
speaking, would you say that you can trust all the people in your neighborhood,
most of the people, some of the people, or none of the people in your
neighborhood?

Concerns about proxy responding: High concern. Trust is much more attitudinal than the
construct measured above. It may be difficult then for proxy respondents to report
accurately on the trust felt by the target. The literature suggests that they will use their
own trust level as an “anchor” in formulating the report about the target, but this may
still result in error.
Q19

In an emergency, some people have friends or family they can rely on; others do
not.
How many friends and family members do you have, if any, whom you could
count on in an emergency? Don’t worry if your answer is not exact, just give me
your best estimate.

Concerns about proxy responding: Moderate concern. Judgments about whether
someone can be relied upon are attitudinal, and so we may expect some error in the
proxy reports. The proxy reports may tend to underestimate the number that would be
reported by the target respondent if there are people who the target knows well and
would rely on but the proxy respondent does no necessarily know well.
Q20

I am going to read some ways that people get news and information. Please tell
me how often you did each of the following during a TYPICAL MONTH in the past
year, that is since April 2010:
(a)

Read a newspaper in print or on the Internet—basically every day, a few
times a week, a few times a month, once a month, less than once a
month, or not at all?

44
(b)

Read a news magazine such as Newsweek or Time, in print or on the
Internet—basically every day, a few times a week, a few times a month,
once a month, less than once a month, or not at all?

(c)

Watch the news on television or get news from television Internet sites—
basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?

(d)

Listen to the news on radio or get news from radio Internet sites—
basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?

(e)

Obtain news from any other Internet sources that we have not previously
asked about such as blogs, chat rooms, or independent news services—
basically every day, a few times a week, a few times a month, once a
month, less than once a month, or not at all?

Concerns about proxy responding: Moderate concern. There is some concern for these
items that proxy respondents would have incomplete knowledge of the news sources
used by the target respondent. That said, to the extent that following news is socially
desirable, the proxy reporting may be more accurate here.
Q21

I am going to name some institutions in this country. For each of these
institutions, would you say you have a great deal of confidence, only some
confidence, or hardly any confidence at all in them?
(a) Corporations
(e) The media
(i) Public schools
(j) Religious institutions

Concerns about proxy responding: High concern. We would expect proxy responses to
be quite flawed for these items. Confidence is an attitude and it seems likely that proxy
respondents would at best have only an educated guess about the confidence level that
the target respondent feels toward each of these institutions.
Q22

How would you describe your overall state of health these days: Would you say
it is excellent, very good, good, fair, or poor?

Concerns about proxy responding: Moderate concern. We would expect proxy reports
about overall health to be reasonably accurate because people in the same household
would generally be expected to know the health status of each other. The proxy
responses would contain some noise, however, because vague quantifiers like “good”

45
versus “fair” are often interpreted differently by different people. It is also true that
some aspects of health status (e.g., pain, mental health) are not necessarily observable
and so may not be accounted for in a proxy response.
Q23

On the following scale, how would you describe the quality of your life overall?
Excellent, good, fair, poor, or very poor?

Concerns about proxy responding: Moderate concern. This question is expected to be
more problematic than Q22 because an evaluation of one’s quality of life is more
subjective than their overall health status. Proxy responses are expected to contain a
fair amount of error on this item.
Selected References
Bickart, B., Menon, G., Schwarz, N. and Blair, J. 1994, "The Use of Anchoring Strategies in
Constructing Proxy Reports of Attitudes," International Journal of Public Opinion
Research, Vol. 6, No. 4, 375-379.
Bickart, B., Blair, J., Sudman, S., Menon, G. 1991. ”An Experimental Study of the Effects
of Level of Participation on Proxy Reports of Vacation Planning.” Proceedings of
the American Statistical Association, Alexandria, Va.
Highton, B. 2005. “Self-Reported Versus Proxy-Reported Voter Turnout in the Current
Population Survey.” Public Opinion Quarterly, 69: 113-123.
Mathiowetz, N.A. and Groves, R.M. .1985. “The Effect of Respondent Rules on Health
Survey Reports.” American Journal of Public Health, 75, 639–644.
Moore, J.C. 1988. “Self/Proxy Status and Survey Response Quality.” Journal of Official
Statistics, 4, 155–172.
Schwarz, N. and Wellens, T. 1997. “Cognitive Dynamics of Proxy Responding: The
Diverging Perspectives of Actors and Observers.” Journal of Official Statistics, 13,
159–179.
Tversky, A. and Kahneman, D. 1974. “Judgment under Uncertainty: Heuristics and
Biases.” Science, 185, 1124–1131.