Attachment A3

Date: March 14, 2011
To:

Nathan Dietz
Corporation for National and Community Service

CC:

JoAnn Jastrzab
Abt Associates
Courtney Kennedy
Abt Associates

From: Benjamin Phillips
Abt SRBI
Johnny Blair
Abt Associates
RE:

Results of Respondent Debriefing Items

This memo summarizes answers to the respondent debriefing items for cognitive testing
for the November 2011 Civic Engagement Supplement (CES) to the Current Population
Survey (CPS), conducted by Abt Associates on behalf of the Corporation for National and
Community Service (CNS). The cognitive testing protocol is found in Appendix A.
Results
Overall, the survey was well received. Respondents felt most questions were easy to
answer and generally reported being interested in question topics. There was little
consensus on which questions might be sensitive. The item most consistently
mentioned as problematic was Q21 (trust in various institutions), with a number of
respondents feeling that it was difficult to provide an answer for categories of
institutions where they may be confident in some but not all institutions and difficulties
with certain categories of institutions asked about.
Overall, would say the survey questions were easy or difficult to answer?
Respondents felt that the majority of the questions were generally easy overall
(“Majority pretty much easy;” “Did a good job;” “Very easy;” “Good overall;” “Easy and
clear;” “Easy, made sense and were clear;” “Mostly easy.”) Exceptions that were
specifically mentioned were length/comprehensibility (mentioned by two respondents),
difficult comparisons, and being overly broad (mentioned by single respondents). One
respondent who mentioned length and comprehensibility felt that some questions
Abt SRBI Inc. 55 Wheeler Street Cambridge, MA 02138 Phone 617.386.2600 Fax 617.386.2622 www.srbi.com
Cambridge Chicago North Carolina New York City Washington D.C. Florida New Jersey Cincinnati

2
“need to break down, get to the point”—what did the questions want to find out, giving
as an example Q5, particularly regarding Q5(a) community organizations. The other
respondent felt that “some questions were too long,” with the question about “labor
organizations” (presumably Q21(d)) as an example. The respondent who raised
problems with comparisons felt it was “difficult to compare one thing to another,”
giving Q21, confidence in institutions in the country, as an example: it may not “be a
true picture with ‘some.’” A similar point was made by another respondent, who felt he
had to average his response options. These tie in with feedback about Q18 (trust some
neighbors and not others) and Q21 (feel one way about some institutions and not
others). The respondent who felt questions were overly broad thought that they could
be “more specific.”
Which questions did you have to think about the most to decide on an answer?
Respondents gave various answers. The respondents noted above who referred to
community organizations (Q5(a)) and labor organizations (Q21(d), “very lengthy”) again
referenced these questions. Two respondents mentioned questions on media
(presumably Q21(e)-(g)). One respondent felt that “There is so much media to consider.
It is hard to break it down.” The other had problems with Q21(g), noting that s/he was
“not too fond of blogging. Sometimes amusing, but not information you can use.” In the
Q21 series another respondent felt that s/he didn’t know anything about the scientific
community (Q21(h)). This respondent also had difficulties answering about her/his
neighborhood, having just moved there. Another respondent had questions about the
type of contact intended by some questions: In Q5 (participation in various kind of
organizations), what did “participation” entail? In Q15, what did “communication” with
neighbors entail, a simple nod hello? (The item actually referred to talking with
neighbors. The respondent lived in a small neighborhood where there was not a lot of
contact.)
Which questions or topics were [most/least] interesting to you?
Respondents were very positive about the topics, with most replying that the questions
were “most”/“mostly” interesting. The exception was a respondent who felt they were
“all interesting.” It should be noted, however, that respondents were recruited from a
pool that had expressed interest in participation in surveys. In general, respondents
mentioned questions about the social or political topics as being of interest, appreciated
the lack of “personal” questions (however, respondents were not asked the core CPS
items), and appeared to like the fact that these were different from typical survey
questions. Questions singled out as for comment were items about respondents’
neighborhoods (three respondents found them interesting, one felt they were the least
interesting of those asked) and visiting neighbors’ homes (Q17; one respondent).
Questions about how people obtained news were also of interest to one respondent.
Another respondent mentioned being most interested in the confidence questions
(Q21).

3
Do you think there are there questions people would find difficult?
There were few consistent themes in answers about which questions people might find
it difficult to respond to. One response shared by two respondents was that the survey
would only be difficult for non-English speakers/people with English as a second
language. Another two mentioned the need to think about responses: “Some people
might have to think a minute;” “Depends on how much people think about these
things.” A single respondent mentioned religious institutions: “Maybe religions. I go by
what I have seen. Another respondent thought questions about neighbors might be
difficult “because people have gotten out of the habit of knowing their neighbors.” I
can’t speak for anyone else.” Q5(b), participation in civil service organizations like the
American Legion and Lions Club, was mentioned by one respondent who was concerned
that people would not know the American Legion. This respondent also mentioned
Q21(g) regarding “citizen media.” A respondent, apparently in response to Q8 (work on
a community project), felt that “the community part” was difficult: “I was thinking as an
individual, do I do things for community vs. being part of church when they do things for
the community.” Finally, a respondent saw multiple choice questions as a problem,
saying that there were “too many” and “it would be hard for people to narrow them
down.”
Do you think there are questions that people would find sensitive?
Several questions were singled out as being sensitive by some respondents. Two
respondents mentioned questions about the number of people one could rely on in an
emergency (Q19): “You’re asking them to size up, define the relationship. You’re telling
on yourself [by saying] who you count on.” Two respondents felt questions about
neighbors might be sensitive. One suggested that this might be because people might
not know if they’re not involved in the neighborhood. The other suggested reaffirming
to respondents that their answers were confidential. Two respondents felt questions on
political issues might be sensitive. One simply mentioned “political activities,” while the
other mentioned volunteering for a campaign. Single respondents mentioned questions
about religious issues, about friends, and about participating in charitable walks.
Another respondent articulated concerns about social desirability in general terms,
feeling that “people would feel bad saying that they hadn’t done anything.” Four
respondents explicitly stated that no items were sensitive.

4
Appendix A: Cognitive Testing Protocol
Testing Protocol
Participant ID #:

| | | | | |

Interview Date:

| | | / | | | / | 2 | 0 | 1 | 1 | (mm/dd/yyyy)

Interviewer Initials:

| | |

Start Time: ____________ AM / PM

End Time: ____________ AM / PM

Section 1: Interviewer: Read/ Paraphrase the following text:

Hello. My name is ________________. I work for Abt Associates, a research
company that does work under contract mainly for federal agencies. Thank you
for agreeing to participate in our study.
Human subjects requirement (prior to starting the recorder)
Confidentiality: Before we begin, I need to be sure you understand that our session today is
completely confidential. Your participation in this study is completely voluntary
and you can decline to answer any particular question.
Incentive:

In appreciation for your participation, you will receive an American Express gift
card for $40.

Recording:

So I don’t have to rely on my memory later on, this session is being video
recorded. That way, I can focus today on what you’re saying rather than having
to concentrate on taking notes.

Observers:

Some members of our research team may be observing the interview.
** HAVE PARTICIPANT READ THE CONSENT FORM**
Before you sign this, do you have questions or see anything that is not clear?
** HAVE PARTICIPANT SIGN THE CONSENT FORM**

START RECORDING

5
Describe the interview objectives and procedures. (This should begin a conversational
interaction with the respondent; it is not necessary to read these descriptions verbatim.):
Before I go into the details of what we’ll be doing, I’d like you to tell me what
you were told about why you were asked to come in today.
(INTERVIEWER: Confirm respondent’s understanding if it is correct. If there are
misconceptions, let the respondent know that you’ll clarify some things as you
describe the plan for the interview session.)
In order to help us improve our surveys, we turn to people like you to find out if
our questions make sense and are fairly easy to understand and answer. We
have found that the best way to do that is to actually conduct the survey with
people and see how it works for them. So you will be helping us test a
questionnaire from one of our surveys.
How:

I want you to answer the questions exactly the way you would if an interviewer
had come to your home for an interview, but with one major difference: I would
like you to tell me your thoughts as you decide on your answers.

Think aloud:

I would like you to think aloud as you answer the questions. I am interested in
your answers, but I am also interested in the thoughts that occur to you as you
answer the questions. I would like you to tell me everything that you are
thinking and feeling as you go about answering each question.
I don’t want you to try to plan what you say. Whatever you’re thinking as you
decide on your answer is what I’d like to hear.
Usually, it helps to try this out once or twice before we get to the survey
questions.
I’d like for you to think aloud as you decide how to answer the question: “How
many windows are there in your house or apartment?”
(INTERVIEWER: Use this second practice question if necessary: “Thinking
about yesterday, starting with the time you got up until you went to bed, how
many phone calls did you make?”)

Probes:

As we go through the survey, from time to time I’ll ask you some questions
about your answers, or about the questions themselves. Remember, there are
no right or wrong answers.
I really want to hear your opinions and reactions, so don’t hesitate to speak up
whenever something is unclear, is hard to answer, or doesn’t seem to apply to
you.
Do you have any questions before we begin?

6
(INTERVIEWER: Wait for respondent to complete thinking aloud or commenting
before asking the probes.)
Q1

In any election, some people are not able to vote because they are sick or busy
or have some other reason, and others do not want to vote. Thinking about
recent local elections, have you voted in all of them, most of them, some of
them, or none?
(1) All of them
(2) Most of them
(3) Some of them
(4) None
Probe: What do you think of as being a “local election”?
Probe: What did you consider as being “recent” in this question?

Q2

I am going to read a list of things some people have done to express their views.
Please tell me whether or not you have done any of the following in the last 12
months, that is between November 2010 and now:
(a)

Contacted or visited a public official - at any level of government - to
express your opinion?
(1) Yes
(2) No

(b)

Bought or boycotted a certain product or service because of the social or
political values of the company that provides it?
(1) Yes
(2) No

No probes. Preexisting questions.
(c)

Gave money to a candidate, party, or political organization?
(1) Yes
(2) No

Probe:

IF YES: Can you tell me about what gave to?

Probe:

IF NO: What kinds of things do you think this question is asking about?

7
Q3

I’m going to read to you a list of different Internet technologies. Please say
whether you have used each of these to express your opinions about political or
social or community issues within the last 12 months:

(INTERVIEWER: Repeat as necessary: Have you used this within the last 12 months to
express your opinions about political or social or community issues?)
(a)

Email
(1) Yes
(2) No

(b)

Text messaging
(1) Yes
(2) No

(c)

Chat room
(1) Yes
(2) No

(d)

A social networking site like Facebook, MySpace, or Twitter
(1) Yes
(2) No

(e)

My own blog
(1) Yes
(2) No

(f)

Commenting or writing on someone else’s blog
(1) Yes
(2) No

Probe:

For selected items (or a subset): Can you tell me what you used [
for?

]

Generic follow-up probe (if necessary): What (other) kinds of
[social / community] issues have you expressed opinions about?
Probe:

For unselected items (or subset): Have you used [ ] for something
other than giving your opinions about a political or social issue?

8
Q4

Have you watched an online video that supports or opposes a candidate, party,
political organization, or public policy campaign in the past 12 months, that is
between November 2010 and now?
(1) Yes
(2) No

Probe:

Would you tell me, in your own words, what this question wants to find
out?

Probe:

What do you think an online video means in this question?

Q5

Next, I will give you a list of types of groups or organizations in which people
sometimes participate. Have you participated in any of these groups during the
last 12 months, that is between November 2010 and now:
(a)

A school group, neighborhood, or community association such as PTA or
neighborhood watch groups?
(1) Yes
(2) No

(b)

A service or civic organization such as American Legion or Lions Club?
(1) Yes
(2) No

(c)

A sports or recreation organization such as a soccer club or tennis club?
(1) Yes
(2) No

(d)

A church, synagogue, mosque or other religious institution or
organization, NOT COUNTING your attendance at religious services?
(1) Yes
(2) No

(e)

Any other type of organization that I have not mentioned?
(1) Yes
(2) No

Continue
Skip to Q7

No probes. Preexisting questions.

9
Q6

What type of organization is that?
Record verbatim

No probes. Preexisting question.
Q7

In the last 12 months, between November 2010 and now, have you been an
officer or served on a committee of any group or organization?
(1) Yes
(2) No

No probes. Preexisting question.
Q8

Have you worked on a community project in the past 12 months?
(1) Yes
(2) No

Probe:

IF YES: Can you tell me a little about the project(s) you worked on?

Probe:

IF NO: What sorts of things do you think the question means by
“community project”?

Q9

Within the past twelve months, have you personally walked, ran, or bicycled or
done anything else for a charitable cause? This is separate from sponsoring or
giving money to this type of event.
(1) Yes
(2) No

Probe:

IF YES: Can you tell me about what you did?
Follow-up probe (if not volunteered): What was the charitable
cause?

Probe:

IF NO: What kinds of events do you think this question is asking about?

10
Q10

In the last 12 months, have you volunteered for a political organization or
candidates running for office?
(1) Yes
(2) No

Probe:

IF YES: Can you describe what you did as a volunteer?

Probe:

IF NO: What do you think the question means by “volunteered for a
political organization”?

Q11

These next questions ask how often you did something during a TYPICAL MONTH
in the past year. How often were politics discussed when communicating with
family and friends, -- basically every day, a few times a week, a few times a
month, once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Not at all

Skip to
Q13
if one
person
household

No probes. Preexisting question.
Q12

How often did you eat dinner with any of the other members of your household
-- basically every day, a few times a week, a few times a month, once a month, or
not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Not at all

No probes. Preexisting question.

11
Q13

During a TYPICAL MONTH in the past year, how often, if at all, did you spend
time visiting friends–basically every day, a few times a week, a few times a
month, once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Not at all

Probe:
Q14

Would you tell me, in your own words, what this question wants to find
out?
During a TYPICAL MONTH in the past year, how often, if at all, did you
communicate with friends and family by email or on the internet -- basically
every day, a few times a week, a few times a month, once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Not at all

No probes. Preexisting question.
Q15

How often did you talk with any of your neighbors -- basically every day, a few
times a week, a few times a month, once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Not at all

No probes. Preexisting question.

12
Q16

How often did you and your neighbors do favors for each other? By favors we
mean such things as watching each other’s children, helping with shopping,
house sitting, lending garden or house tools, and other small acts of kindness -basically every day, a few times a week, a few times a month, once a month, or
not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Not at all

No probes. Preexisting question.
Q17

During a TYPICAL MONTH in the past year, how often, if at all, were you in the
home of a neighbor–basically every day, a few times a week, a few times a
month, once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Not at all

Probe:

How long have you lived at your current address?
Follow-up [if less than a year]: Did you think mostly about your
current neighborhood, your last neighborhood, or both
neighborhoods?

Probe:

Would you tell me, in your own words, what this question wants to find
out?

13
Q18

We’d like to know how much you trust people in your neighborhood. Generally
speaking, would you say that you can trust them a lot, some, only a little, or not
at all?
(1) A lot
(2) Some
(3) Only a little
(4) Not at all

Probe:

(Unless the respondent answered “Not at all”) How did you decide on
[choice] rather than [next closest option]?

Probe:

What sorts of things come to mind when you hear “trust the people in
your neighborhood”?

Probe:

What do you think of as “your neighborhood”?

Probe:

What comes to mind when you hear “people in your neighborhood”?

Q19

How many friends or family members do you have, if any, whom you could count
on in an emergency? Don’t worry if your answer is not exact, just give me your
best estimate.
Accept integers 0 through 9999

Probe:

What sorts of things do you think the question means by “emergency”?

Probe:

What does “count on” mean to you in this question?

Probe:

How many of the [number >1 mentioned] are [friends/family]?

Probe:

Do any live outside your neighborhood?

Q20

I am going to read some ways that people get news and information. Please tell
me how often you did each of the following during a TYPICAL MONTH in the past
year:
(a)

Read a newspaper in print or on the Internet - basically every day, a few
times a week, a few times a month, once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Not at all

14
(b)

Read news magazines such as Newsweek or Time, in print or on the
Internet - basically every day, a few times a week, a few times a month,
once a month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Not at all

(c)

Watch the news on television or get news from television internet sites basically every day, a few times a week, a few times a month, once a
month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Not at all

(d)

Listen to the news on radio or get news from radio internet sites basically every day, a few times a week, a few times a month, once a
month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Not at all

(e)

Obtain news from any other Internet sources that we have not previously
asked about such as blogs, chat rooms, or independent news services basically every day, a few times a week, a few times a month, once a
month, or not at all?
(1) Basically every day
(2) A few times a week
(3) A few times a month
(4) Once a month
(5) Not at all

15
No probes. Preexisting questions.
Q21

I am going to name some institutions in this country. As far as the people
running these institutions are concerned, would you say you have a great deal of
confidence, only some confidence, or hardly any confidence at all in them?

[INTERVIEWER: Repeat the response options unless the respondent starts to answer
before you read them; or if the respondent appears to forget the options.]
(a) Banks and financial institutions
(1) Great deal of confidence
(2) Some confidence
(3) Hardly any confidence
(b) Major companies
(1) Great deal of confidence
(2) Some confidence
(3) Hardly any confidence
(c) Small and local businesses
(1) Great deal of confidence
(2) Some confidence
(3) Hardly any confidence
(d) Organized labor
(1) Great deal of confidence
(2) Some confidence
(3) Hardly any confidence
(e) The print media, such as newspapers and news magazines
(1) Great deal of confidence
(2) Some confidence
(3) Hardly any confidence
(f) Electronic news media, such as television and cable news
(1) Great deal of confidence
(2) Some confidence
(3) Hardly any confidence

16
(g) Independent or citizen media, such as an online blog
(1) Great deal of confidence
(2) Some confidence
(3) Hardly any confidence
(h) The scientific community
(1) Great deal of confidence
(2) Some confidence
(3) Hardly any confidence
(i) Public schools
(1) Great deal of confidence
(2) Some confidence
(3) Hardly any confidence
(j) Religious institutions
(1) Great deal of confidence
(2) Some confidence
(3) Hardly any confidence
Probe:

Who would you think of as “the people running” the scientific
community?

Probe:

(Use especially if respondent usually chose the middle category) If you
really didn’t have an opinion about the people running [insert from list
that was answered “some”], how would you have answered?

Probe:

What were you thinking when you said you had [response choice] in
[insert from list]?

Probe:

What do you think the question means by “citizen media”?

Probe:

How would you answer about [insert from list] if you had a great deal of
confidence in some of the people running [insert from list] but hardly
any confidence in others?

17
Q22

How would you describe your overall state of health these days: Would you say
it is excellent, very good, good, fair, or poor?
(1) Excellent
(2) Very good
(3) Good
(4) Fair
(5) Poor

No probes. Preexisting question.
Q23

On the following scale, how would you describe the quality of your life overall?
Excellent, good, fair, poor, or very poor?
(1) Excellent
(2) Good
(3) Fair
(4) Poor
(5) Very poor

Probe:

What did you think the question means by “the quality of your life
overall”?

DEBRIEFING QUESTIONS
Overall, would say the survey questions were easy or difficult to answer?
Follow-up:

Why?

Which questions did you have to think about the most to decide on an answer?
Follow-up:

Why was that?

Which questions or topics were [most/least] interesting to you?
Do you think there are questions people would find difficult?
Follow-up:

Which ones?

Follow-up:

Why?

Do you think there are questions that people would find sensitive?
Follow-up:

Which ones?

Follow-up:

Why?