OMB Part B_070816

Contents
B.

Collection of Information Employing Statistical Methods................................................. B-1
B.1.
Respondent Universe .......................................................................................... B-1
B.2.
Procedures for the Collection of Information/Limitations of the Study ............. B-1
B.3.
Methods to Maximize Response Rates and Deal With Issues of Nonresponse .. B-3
B.4.
Tests of Procedures or Methods.......................................................................... B-4
B.5.
Names and Telephone Numbers of Individuals Consulted ................................. B-4

Abt Associates Inc.

Evaluation of Youth Corps: 18-Month Follow-up: Part B

B-i

B.

Collection of Information Employing Statistical
Methods

B.1. Respondent Universe
The Corporation’s contractor, Abt Associates Inc., will collect information for the Evaluation of
Youth Corps: 18-Month Follow-up on behalf of the Corporation. The contractor is responsible for the
design and administration of the surveys that will be used to collect information about youth
corpsmembers.
In 2006, a stratified random sample of 22 youth corps programs across the country were selected and
agreed to participate in the youth corps study. The 22 youth corps were randomly selected from a
universe of 100 programs and a sampling frame of 60 programs that were eligible for the youth corps
study.1 Corps enrolled and served over 21,034 corpsmembers in 2006. Study participants were
selected from applicants to the programs selected for participation in the study. A representative
sample of 2,267 eligible corpsmembers was randomly assigned to either participate in a youth corps
program (treatment) or to be part of the control group. The statistical methods for this collection were
approved by OMB under #3045-0119. This information collection request is for the 18-month followup for participants in the study.
Exhibit B-1 shows the number of youth corps programs in the universe, the number of youth
corpsmembers enrolled in the 2006 program year, the number of youth corps that were sampled and
participated in the study, and the number of members in both the treatment and control groups.
Exhibit B-1
Population and Sample Size
Population

Sample

Number of
Programs

Number of Members

Number of
Programs

Number of
Members

100

21,034

22

2,267

A random sample of eligible program applicants was selected from each selected program to yield a
total sample of 2,267 youth corps eligible applicants at baseline.

B.2. Procedures for the Collection of Information/Limitations of
the Study
Abt Associates project staff will coordinate with their subcontractor, Guideline Inc., to conduct the
follow-up survey. All study participants who completed baseline surveys in 2006 and 2007 will be
contacted by telephone to participate in the follow-up survey, which will be conducted using a
1

Eligible corps had to be active members of The Corps Network and to have served a minimum of 50
corpsmembers in 2005.

Abt Associates Inc.

Evaluation of Youth Corps: 18-Month Follow-up: Part B

B-1

computer-assisted telephone interviewing (CATI) system. CATI allows telephone interviewers to
read the survey questions on their computer screens and record the survey responses using their
keyboards. This technology speeds the conduct of the interviewing by managing question skips and
other technical issues associated with the survey questionnaire. It also assures higher quality data by
eliminating a separate step for data entry. Conducting the survey by telephone will also be least
burdensome to respondents, since they are not required to complete any paperwork and they can
complete the survey in their own home.
B.2.1. Statistical Methodology for Stratification and Sample Selection

The approach for the study includes a sample of approximately 2,267 applicants from a randomly
selected stratified sample of 22 youth corps programs. Program applicants in the 2006–2007 program
year were randomly assigned either to participate in a youth corps program (treatment group) or to a
control group. The 22 corps were randomly chosen from all eligible, Corps Network-affiliated youth
corps programs. The program group includes full-time, part-time, and limited-part-time
corpsmembers in their initial year of service. The methodology for stratification and sample selection
was approved by OMB under #3045-0119. This collection is for the 18-month follow-up for
participants in the study.
B.2.2. Estimation Procedure

For producing population-based estimates of totals, percentages and means, each respondent member
will have a sampling weight. This weight combines the base weight that is the inverse of the
probability of selection of a member and an adjustment for non-response. This adjustment is to
account for members who are in the sample but do not respond to the follow-up survey. Every effort
will be made to minimize the non-sampling errors in the estimates by maximizing the response rates
and taking steps to reduce response errors.
In addition to the program sample of youth corpsmembers, the study also drew a sample of control
group members in order to assess the impact of the program. An impact study is generally so defined
because it attempts to identify program effects that cause the changes in outcomes. Thus, a study of
impacts requires the identification of an appropriate control group. Control group members are
similar to the treatment group, and were eligible for the youth corps program, but through random
assignment are excluded from participation in the youth corps for the study period.
Because this study relies on random assignment, estimation of program impacts is relatively simple
and straightforward. Random assignment produces treatment and control group members who are
comparable in every way except for receipt of treatment. Therefore, we can interpret any statistically
significant differences in outcomes as effects of participation in youth corps. Abt Associates Inc. will
use a standard difference-in-means approach to yield average impact estimates of participation. This
approach compares average outcomes of treatment and control group members; given random
assignment, any differences can be attributable to the program. To the extent possible, we will also
examine program effects on various subgroups by splitting the sample and estimating the effects on
each subgroup using data only for that subgroup. Subgroups of interest may include race/ethnicity,
gender, age, and education. Finally, we will explore the differential effects of selected program
experiences, including program intensity (part-time vs. full-time duration) and AmeriCorps funding
status (AmeriCorps funded vs. non-AmeriCorps funded participants); this analysis will be conducted

Abt Associates Inc.

Evaluation of Youth Corps: 18-Month Follow-up: Part B

B-2

on the treatment group alone, since we will not have comparable data from the control group.2 We
plan to consult with the members of the expert panel (Exhibit A-1, page A-6 in Part A) to provide
input on any key analytic decisions.
B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

A sample size of 2,267 was randomly assigned during the baseline data collection period. Therefore,
we will attempt to collect follow-up data on this sample. We estimate in Exhibit B-2 the sample sizes
necessary to conduct subgroup analysis. As indicated in the exhibit, there are reasonable gains in
precision as measured by minimum detectable effect sizes (MDEs), as the sample size increases from
500 to 2,000 per group. Therefore, our ability to conduct subgroup analysis by the levels of
participation (full-time, part-time and limited part-time) and AmeriCorps funding status (AmeriCorps
funded vs. non-AmeriCorps funded participants) will need to be determined based on the response
rates for these subgroups after follow-up.
Exhibit B-2
Minimum Detectable Effect Sizes
Sample Size, Treatment Group

a

Sample Size, Control/Comparison Group

Minimum Detectable Effect Size (MDES)a

500

500

.078

750

750

.064

1000

1000

.055

1500

1500

.045

2000

2000

.039

Based on 80 percent power at the .05 significance level (two-tail test), 80 percent survey response rate.

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems requiring specialized sampling procedures.
B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

The follow-up survey is a one-time data collection necessary to determine the differences between
youth corpsmembers and their counterparts in the control group 18 months after they were randomly
assigned to one of these two groups.

B.3. Methods to Maximize Response Rates and Deal With Issues
of Nonresponse
All eligible youths who completed the baseline survey during the youth corps application process will
be asked to complete the follow-up survey. We expect the fact that respondents have already
completed one similar survey and are familiar with the study to encourage most respondents to
participate. The instrument was developed in coordination with The Corps Network and the
2

The feasibility of various subgroup analyses will be determined by the size of the sample, and the size of
the subgroups being explored.

Abt Associates Inc.

Evaluation of Youth Corps: 18-Month Follow-up: Part B

B-3

Corporation, with consideration to length and reading level so it is simple for youth to complete. In
addition, the instrument was pre-tested with a diverse group of four current corpmembers to ensure
usability.
Guideline Inc. will contact all individuals 10 months after completion of the baseline survey to verify
their current contact information. This will increase the likelihood of locating these individuals to
conduct the follow-up survey 18 months after completion of the baseline survey. When conducting
the follow-up survey, a minimum of seven telephone calls will be made to individuals who may be
difficult to reach. Phone numbers will be called at different times of the day and on different days to
increase the likelihood of completing the telephone interview. In addition, a 1-800 telephone number
will be left with hard-to-reach individuals for callbacks at their convenience. This methodology and
level of calling persistence improves the response rate and minimizes non-response bias.
We expect to achieve at least an 80 percent response rate on completion of the follow-up survey.

B.4. Tests of Procedures or Methods
The survey instrument has been drafted and has undergone two reviews: (1) an internal review
conducted by Abt Associates Inc.’s Internal Review Board and (2) a pre-test with four former youth
corpsmembers. In order to accurately determine the burden placed on respondents as well as further
test the clarity of the survey questions, a pre-test was conducted in which a total of four former youth
corpsmembers from diverse backgrounds responded to the survey to assess the reliability of the
instrument. Revisions were made to the follow-up instrument in response to comments received from
both of these reviews.
Modifications to the length, content, and structure of the survey have been made based on the results
of the survey pre-test interviews. Respondents provided generally positive feedback indicating that
they could readily answer the questions and that the time to complete the survey was not onerous
(about 60 minutes).

B.5. Names and Telephone Numbers of Individuals Consulted
The information for this study is being collected by Abt Associates Inc., a research and consulting
firm, on behalf of the Corporation for National and Community Service. With Corporation oversight,
Abt Associates Inc. is responsible of the study design, instrument development, data collection,
analysis, and report preparation.
The instrument for this study and the plans for statistical analyses were developed by Abt Associates
Inc. The staff team is composed of JoAnn Jastrzab, project director; Dr. Carrie Markovitz, task
leader; and a team of senior-level staff. In addition, members of the expert panel consulted on the
study design and instrument development (see Exhibit A-1, page A-6 in Part A). Contact information
for these individuals is provided below.
Name

Number

JoAnn Jastrzab
Carrie Markovitz

617-349-2372
301-634-1807

Abt Associates Inc.

Evaluation of Youth Corps: 18-Month Follow-up: Part B

B-4