New GPA OMB Clearance Request 1.21.10 - Part B

OMB Clearance Request

B. DESCRIPTION OF STATISTICAL METHODS
1.

Universe and Respondent Selection
All three surveys proposed in this study are targeting the entire universe of respondents
(i.e., census). This is because the number of cases is small, and/or sampling would not be
appropriate given the proposed analyses. In the case of the program fellows, a census of
approximately 1800 fellows is necessary to be able to report the results by cohort (i.e., year
of fellowship). Given the study encompasses eight program years, this yields
approximately 225 fellows per year. This is a necessary number to be able to conduct
many analyses by cohort. For example, conducting a cross-tabulation of race/ethnicity by
cohort even with the entire census will yield several cells with a very small number of
cases (under 50). Should we use a sample, we would not be able to conduct any analyses
by cohort, even though there are strong theoretical reasons to expect that the program
outcomes of fellows do vary by cohort (e.g., career trajectory).
The frames for all three surveys are derived from the U.S. Department of Education’s
International Resource Information System (IRIS) and includes all 29 GPA Project
Directors who received GPA funding since 2002 as well as the complete list of all 20022009 program fellows (approximately 1800 individuals).
These frames are as exhaustive as frames can be in that they represent the entire population
of interest. Because participants in the GPA program self-select themselves into applying,
it is possible that the universe of participants is not similar to the larger population of
students with interest in foreign languages and area studies across the United States.
However, because there is no such frame and the entire population of GPA potential
applicants is undefinable, this limitation will be noted in the final report.

2.

Procedures for Collecting Information
The information for this study will be gathered through three surveys (see Appendix A).
Two of these surveys (2000-2008 fellows and 2009 fellows) will be administered via the
World Wide Web using Survey Gizmo, a commercial survey dissemination and

12

OMB Clearance Request

management website. Because of the small number of respondents (N=29), the GPA
Project Director survey will be administered as an electronic Word document disseminated
via email. All respondents will be able to request a pencil and paper version of the study.
Respondents to all three surveys will be contacted about the study via email. AIR has
already worked with GPA Project Directors to publicize the study and emphasize its
importance to respondents. All respondents will receive a pre-notification email about the
study emphasizing its importance and requesting their participation. Three to five email
follow-ups over a period of two months (depending on response rates monitored on a
weekly basis) will follow up with respondents to encourage participation (see Appendix B
for contact and follow-up emails).

3.

Methods to Maximize Response Rates
As discussed above, methods to maximize response rates have already been implemented
by publicizing the study early on to respondents via email to Project Directors. In addition,
the follow-up procedures discussed above will be implemented to follow up with
respondents who have not filled out the surveys.
To ensure a contact rate near 100 percent, AIR updated all email contact information
through cross-checking them with the latest available contact data included in IRIS and by
obtaining available updated contact information from GPA Project Directors.
Because the study has been well publicized already and the respondents are professionals
with a vested interest in the topic of the study and in the future of the GPA program, we
expect a high response rate.
It is expected that the response rate for the GPA Project Directors will be 100 percent
because they are all current or past grantees of the U.S. Department of Education with a
vested interest in the assessment (17 are current and 12 are past Project Directors). For the
12 past Project Directors, we still expect them to participate in the study because while they
may not be currently funded, it is the Department of Education’s expectation that they are
sustaining the activities started with a GPA grant and will most likely apply for another

13

OMB Clearance Request

grant in the near future. Past Project Directors still have access to all their program records
via the IRIS system.
The expected response rates for the two fellows’ surveys are 85 percent. This response rate
will be achieved through a combination of updated contact information, contact emails
emphasizing the importance of the project and endorsed by influential scholars in the field
of foreign language teaching, and the multiple follow-ups from the contractor mentioned
above.
Non-response bias due to a less than 100% response rate will be examined by conducting a
non-response bias analysis based on the information available from the frame. However, it
must be kept in mind that only three variables are available on the frame for each
respondent: year of fellowship; institutional affiliation; and country of fellowship. The
paucity of data seriously limits the extent of the non-response bias analysis that can be
conducted. While no statistical adjustments are planned given the study is based on a
census, an analysis reporting on the frame totals as compared to the respondent totals on
the above three variables will be reported in the final report and appropriate caution in the
interpretation of findings will be discussed.

4.

Tests of Procedures
Because the questions in these surveys capture implementation and are process-oriented,
they have not been cognitively tested. They have, however, been vetted by a committee of
experts, and were developed, in part, based on a review of similar questionnaires used
previously by the U.S. Department of Education.
Usability of the website for the surveys will be examined by AIR’s subcontractor Firepig
Partners who specializes in on-line survey design and administration. Based on their
recommendations, revisions to the web designs will be implemented.

5.

Contacts for Statistical Aspects and Data Collection
The following individuals were involved in the design and statistical aspects of the study
and its data collection:

14

OMB Clearance Request

•

Stephane Baldi, Principal Research Scientist, American Institutes for Research

•

Tanya Taylor, Research Analyst, American Institutes for Research

•

Martha Abbott, Director of Education, American Council on the Teaching of Foreign
Languages (ACTFL)

•

Gerald Lampe, Former Deputy Director of the National Foreign Language Center
(NFLC)

•

Scott McGinnis, Professor and Academic Advisor, Defense Language Institute.

15