Document
Appendices A-C NAEP 2019 & 2020 Supplemental Documents
ICR 201908-1850-006 · OMB 1850-0928 · Object 94448201.
⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0928 can be found here:
Document [pdf]
Download: pdf | txt
NATIONAL CENTER FOR EDUCATION STATISTICS NATIONAL ASSESSMENT OF EDUCATIONAL P ROGRESS National Assessment of Educational Progress (NAEP) 2019 and 2020 Long-Term Trend (LTT) 2020 Update 2 Appendices A-C Appendix A: External Advisory Committees Appendix B1: NAEP 2013 Weighting Procedures Appendix B2: Long-Term Trend (LTT) 2012 Weighting Procedures Appendix C1: NAEP 2019 Sampling Memo Appendix C2: LTT 2020 Sampling Memo OMB# 1850-0928 v.16 March 2019 No changes since v.15 Table of Contents Appendix A: External Advisory Committees (no changes)………………………………………................3 Appendix B1: NAEP 2013 Weighting Procedures (no changes)……………………………………..........18 Appendix B2: Long-term trend 2012 Weighting Procedures (new)……………………………….............68 Appendix C1: NAEP 2019 Sampling Memo (no changes)……………………………………………….106 Appendix C2: LTT 2020 Sampling Memo (new)..…………………………………………………….....128 Appendices A-C NAEP 2019-2020 2 NATIONAL CENTER FOR EDUCATION STATISTICS NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS National Assessment of Educational Progress (NAEP) 2019 and 2020 Appendix A External Advisory Committees OMB# 1850-0928 v.15 September 2018 No changes since v.10 Appendices A-C NAEP 2019-2020 3 Appendix A-1: NAEP Design and Analysis Committee Name Affiliation Betsy Becker Florida State University, FL Peter Behuniak University of Connecticut, CT Lloyd Bond University of North Carolina, Greensboro, NC (Emeritus)/Carnegie Foundation (retired) Derek Briggs University of Colorado, CO Steve Elliott Arizona State University, AZ Ben Hansen University of Michigan, MI Matthew Johnson Columbia University, NY Brian Junker Carnegie Mellon University, PA David Kaplan University of Wisconsin-Madison, WI Kenneth Koedinger Carnegie Mellon University, PA Sophia Rabe-Hesketh University of California, Berkeley, CA Michael Rodriguez University of Minnesota, MN S.Lynne Stokes Southern Methodist University, TX Chun Wang University of Minnesota, MN Appendices A-C NAEP 2019-2020 4 Appendix A-2: NAEP Validity Studies Panel Name Affiliation Peter Behuniak University of Connecticut, CT George Bohrnstedt American Institutes for Research, Washington, DC Jim Chromy RTI International (Emeritus Fellow), Raleigh, NC Phil Daro Strategic Education Research (SERP) Richard Duran University of California, Berkeley, CA David Grissmer University of Virginia, VA Larry Hedges Northwestern University, IL Sami Kitmitto American Institutes for Research, San Mateo, CA Ina Mullis Boston College, MA Scott Norton Council of Chief State School Officers, Washington, DC Jim Pellegrino University of Illinois at Chicago/Learning Sciences Research Institute, IL Gary Phillips American Institutes for Research, Washington, DC Lorrie Shepard University of Colorado at Boulder, CO Fran Stancavage American Institutes for Research, San Mateo, CA David Thissen University of North Carolina at Chapel Hill, NC Sheila Valencia University of Washington, WA Ting Zhang American Institutes for Research, Washington, DC Appendices A-C NAEP 2019-2020 5 Appendix A-3: NAEP Quality Assurance Technical Panel Name Affiliation Jamal Abedi University of California, Davis, CA Chuck Cowan Analytic Focus LLC, San Antonio, TX Gail Goldberg Gail Goldberg Consulting, Ellicott City, MD Brian Gong National Center for the Improvement of Educational Assessment, Dover, NH Richard Luecht University of North Carolina-Greensboro, NC Jim Pellegrino University of Illinois at Chicago/Learning Sciences Research Institute, IL Mark Reckase Michigan State University, MI Michael (Mike) Russell Boston College, MA Phoebe Winter Consultant, Chesterfield, VA Richard Wolfe University of Toronto (Emeritus), Ontario, Canada Appendices A-C NAEP 2019-2020 6 Appendix A-4: NAEP National Indian Education Study Technical Review Panel Name Affiliation Doreen E. Brown ASD Education Center, Anchorage, AK Robert B.Cook Native American Initiative/Teach for America, Summerset, SD Steve Andrew Culpepper University of Illinois at Urbana-Champaign, IL Susan C. Faircloth University of North Carolina Wilmington, NC Jeremy MacDonald Rocky Boy Elementary, Box, Elder, MT Holly Jonel Mackey University of Oklahoma, OK Jeannette Muskett Miller Tohatchi High School, Tohatchi, NM Sedelta Oosahwee National Education Association, DC Debora Norris Salt River Pima-Maicopa Indian Community Martin Reinhardt Northern Michigan University, MI Tarajean Yazzie-Mintz Wakanyeja ECE Initative/American Indian College Fund, Denver, CO Appendix A-5: Geography Standing Committee Name Affiliation Sarah Bednarz Texas A&M University, TX Osa Brand National Council for Geographic Education, Washington, DC Seth Dixon Rhode Island College, RI Charlie Fitzpatrick ESRI Schools, Arlington, VA Ruth Luevanos Pacoima Middle School, Pacoima, CA Joe Stoltman Western Michigan University, MI Kelly Swanson Johnson Senior High, St. Paul, MN Appendices A-C NAEP 2019-2020 7 Appendix A-6: NAEP Civics Standing Committee Name Affiliation Patricia Avery University of Minnesota, MN Christopher Elnicki Cherry Creek School District, Greenwood Village, CO Fay Gore North Carolina Public Schools, Raleigh, NC Barry Leshinsky Challenger, NC Middle School, Huntsville, AL Peter Levine CIRCLE (Center for Information & Research on Civic Learning and Engagement), Medford, MA Clarissa Peterson DePauw University, IN Terri Richmond Golden Valley High School, Bakersfield, CA Jackie Viana Miami-Dade County Schools, Miami, FL Appendix A-7 NAEP Economics Standing Committee Name Affiliation Kris Bertelsen Little Rock Branch-Federal Reserve Bank of St. Louis, Little Rock, AR William Bosshardt Florida Atlantic University, FL Stephen Buckles Vanderbilt University, TN Andrea Caceres-Santamaria Seminole Ride Community High School, FL Steven L. Cobb University of North Texas, TX Kristen S. McDaniel Wisconsin Dept. of Public Instruction, WI Richard MacDonald St. Cloud State University, MN Kevin Smith Renaissance High School, Detroit, MI William Walstad University of Nebraska–Lincoln, NE Appendices A-C NAEP 2019-2020 8 Appendix A-8: NAEP Mathematics Standing Committee Name Affiliation Scott Baldridge Louisiana State University, LA Carl Cowen Indiana University–Purdue University, IN Kathleen Heid Pennsylvania State University, PA Mark Howell Gonzaga College High School, Washington, DC Carolyn Maher Rutgers University, NJ Michele Mailhot Maine Department of Education, Augusta, ME Matthew Owens Spring Valley High School, Columbia, SC Carole Philip Alice Deal Middle School, Washington, DC Kayonna Pitchford University of North Carolina, NC Melisa M. Ramos Trinidad Educación Bilingüe Luis Muñoz Iglesias, Cidra, PR Allan Rossman College of Science and Mathematics-CalPoly, CA Carolyn Sessions Louisiana Department of Education, LA Lya Snell Georgia Department of Education, GA Ann Trescott Stella Maris Academy, La Jolla, CA Vivian Valencia Espanola Public Schools, NM Appendices A-C NAEP 2019-2020 9 Appendix A-9: NAEP Reading Standing Committee Name Affiliation Peter Afflerbach University of Maryland, MD Patricia Alexander University of Maryland, MD Alison Bailey University of California, LA, CA Katrina Boone Kentucky Department of Education, KY Margretta Browne Richard Montgomery High School, Silver Spring, MD Julie Coiro University of Rhode Island, RI Bridget Dalton University of Colorado Boulder, CO Jeanette Mancilla-Martinez Vanderbilt University, TN Pamela Mason Harvard Graduate School of Education, MA P. David Pearson University of California, Berkeley, CA Frank Serafini Arizona State University, AZ Kris Shaw Kansas State Department of Education, KS Diana Townsend University of Nevada, Reno, NV Victoria Young Texas Education Agency, Austin, TX Appendices A-C NAEP 2019-2020 10 Appendix A-10: NAEP Science Standing Committee Name Affiliation Alicia Cristina Alonzo Michigan State University, MI George Deboer American Association for the Advancement of Science, Washington, DC Alex Decaria Millersville University, PA Crystal Edwards Lawrence Township Public Schools, Lawrenceville, NJ Ibari Igwe Shrewd Learning, Elkridge, MD Michele Lombard Kenmore Middle School, Arlington, VA Emily Miller Consultant, WI Blessing Mupanduki Department of Defense, Washington, DC Amy Pearlmutter Littlebrook Elementary School, Princeton, NJ Brian Reiser Northwestern University, Evanston, IL Michal Robinson Alabama Department of Education, Montgomery, AL Gloria Schmidt Darby Junior High School, Fort Smith, AR Steve Semken Arizona State University, Tempe, AZ Roberta Tanner Board of Science Education, Longmont, CO David White Lamoille North Supervisory Union School District, Hyde Park, VT Appendices A-C NAEP 2019-2020 11 Appendix A-11: NAEP Survey Questionnaires Standing Committee Name Affiliation Angela Duckworth University of Pennsylvania, PA Hunter Gehlbach Harvard University, MA Camille Farrington University of Chicago, Chicago, IL Gerunda Hughes Howard University, DC David Kaplan University of Wisconsin-Madison, WI Henry Levin Teachers College, Columbia University, NY Stanley Presser University of Maryland, MD Augustina Reyes University of Houston, Houston, TX Leslie Rutkowski Indiana University Bloomington, IN Jonathon Stout Lock Haven University, PA Roger Tourangeau Westat, Rockville, MD Akane Zusho Fordham University, NY Appendices A-C NAEP 2019-2020 12 Appendix A-12: NAEP Technology and Engineering Literacy Standing Committee Name Affiliation Keith Barton Indiana University Bloomington, IN John Behrens Pearson eLEADS Center, Mishawaka, IN Brooke Bourdelat-Parks Biological Sciences Curriculum Study, Colorado Springs, CO Barbara Bratzel Shady Hill School, Cambridge, MA Lewis Chappelear James Monroe High School, North Hills, CA Britte Haugan Cheng SRI International, Menlo Park, CA Meredith Davis North Carolina State University, NC Chris Dede Harvard Graduate School of Education, MA Richard Duran University of California, Santa Barbara, CA Maurice Frazier Oscar Smith High School, Chesapeake, VA Camilla Gagliolo Arlington Public Schools, Arlington, VA Christopher Hoadley New York University, NY Eric Klopfer Massachusetts Institute of Technology, MA Beth McGrath Stevens Institute of Technology, NJ Greg Pearson National Academy of Engineering, Washington, DC John Poggio University of Kansas, KS Erin Reilly University of Southern California, CA Troy Sadler University of Missouri Science Education Center, Columbia, MO Kimberly Scott Arizona State University, AZ Teh-Yuan Wan New York State Education Department, Albany, NY Appendices A-C NAEP 2019-2020 13 Appendix A-13: NAEP U.S. History Standing Committee Name Affiliation Keith Barton Indiana University Bloomington, IN Michael Bunitsky Frederick County Public Schools, Frederick, MD Teresa Herrera Shenandoah Middle School, Miami, FL Cosby Hunt Center for Inspired Teaching, Washington, DC Helen Ligh Macy Intermediate School, Monterey, CA Amanda Prichard Green Mountain High School, Lakewood, CO Kim Rasmussen Auburn Washburn Unified School District, Topeka, KS Diana Turk New York University, New York, NY Appendix A-14: NAEP Mathematics Translation Review Committee Name Affiliation Mayra Aviles Puerto Rico Department of Education, PR David Feliciano P.S./M.S 29, The Melrose School, Bronx, NY Yvonne Fuentes Author and Spanish Linguist, Carrollton, GA Marco Martinez-Leandro Sandia High School, NM Jose Antonio (Tony) Paulino Nathan Straus Preparatory School, NY Evelisse Rosado Rivera Teacher, PMB 35 HC, PR Myrna Rosado-Rasmussen Austin Independent School District, TX Gloria Rosado Vazquez Teacher, HC-02, PR Enid Valle Kalamazoo College, Kalamazoo, MI Appendices A-C NAEP 2019-2020 14 Appendix A-15: NAEP Science Translation Review Committee Name Affiliation Daniel Berdugo Teacher, PS 30X Wilton, NY Yvonne Fuentes Author and Spanish Linguist, Carrollton, GA Myrna Rosado- Rasmussen Austin Independent School District, Austin, TX Enid Valle Kalamazoo College, Kalamazoo, MI Appendix A-16: NAEP Grade 8 Social Science Translation Review Committee Name Affiliation Yvonne Fuentes Author and Spanish Linguist, Carrollton, GA Jose Antonio Paulino Middle School Teacher, Nathan Strauss Preparatory School, NY Dagoberto Eli Ramierz Bilingual Education Expert, Palmhurst, TX Enid Valle Kalamazoo College, Kalamazoo, MI Appendix A17: NAEP Grade 4 and 8 Survey Questionnaires and eNAEP DBA System Translation Committee Name Affiliation Daniel Berdugo PS 30X Wilton, Bronx, NY Yvonne Fuentes Carrollton, GA Marco Martinea-Leandro Sandia High School. Albuquerque, NM Jose Antonio (Tony) Paulino Nathan Straus Preparatory School, New York, NY Evelisse Rosado Rivera PMB 36 HC 72, Naranjito, PR Myrna Rosado-Rasmussen Austin Independent School District, Austin, TX Gloria M. Rosado Vazquez HC – 02 Barranquitas, PR Enid Valle Kalamazoo College, Kalamazoo, MI Appendices A-C NAEP 2019-2020 15 Appendix A-18: NAEP Writing Standing Committee Name Affiliation Margretta Browne Montgomery County Public Schools, Silver Spring, MD Dina Decristofaro Scituate Middle School, RI Elyse Eidman-Aadahl National Writing Project, Berkeley, CA Nikki Elliot-Schuman Smarter Balanced Assessment Consortium Charles MacArthur University of Delaware, Newark, DE Michael McCloskey Johns Hopkins University, Baltimore, MD Norma Mota-Altman San Gabriel High School, Alhambra, CA Sandra Murphy University of California, Davis, Walnut Creek, CA Peggy O’Neill Loyola University Maryland, MD Laura Roop University of Pittsburgh School of Education, PA Drew Sterner Tamanend Middle School, Warrington, PA Sherry Swain National Writing Project, Berkeley, CA Jason Torres-Rangel University of California, CA Victoria Young Texas Education Agency, Austin, TX Appendices A-C NAEP 2019-2020 16 Appendix A-19: NAEP Principals’ Panel Standing Committee Name Affiliation David Atherton Clear Creek Middle School, Gresham, OR Ardith Bates Gladden Middle School, Chatsworth, GA Williams Carozza Harold Martin Elementary School, Hopkinton, NH Diane Cooper St. Joseph’s Academy, Clayton, MO Brenda Creel Alta Vista Elementary School, Cheyenne, WY Rita Graves Pin Oak Middle School, Bellaire, TX Don Hoover Lincoln Junior High School, Springdale, AR Stephen Jackson (Formerly with) Paul Laurence Dunbar High School, Washington, DC Anthony Lockhart Lake Shore Middle School, Belle Glade, FL Susan Martin Berrendo Middle School, Roswell, NM Lillie McMillan Porter Elementary School, San Diego, CA Kourtney Miller Chavez Prep Middle School, Washington, DC Jason Mix Howard Lake–Waverly–Winsted High School, Howard Lake, MN Leon Oo-Sah-We Ch’ooshgai Community School, Tohatchi, NM Sylvia Rodriguez Vargas Atlanta Girls’ School, Atlanta Georgia, GA Appendices A-C NAEP 2019-2020 17 NATIONAL CENTER FOR EDUCATION STATISTICS NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS National Assessment of Educational Progress (NAEP) 2019 and 2020 Appendix B1 NAEP 2013 Weighting Procedures OMB# 1850-0928 v.15 September 2018 No changes since v.10 Appendices A-C NAEP 2019-2020 18 NAEP Technical Documentation Website NAEP Technical DocumentationWeighting Procedures for the 2013 Assessment NAEP assessments use complex sample designs to Computation of Full-Sample Weights create student samples that generate population Computation of Replicate Weights for and subpopulation estimates with reasonably high Variance Estimation precision. Student sampling weights ensure valid inferences from the student samples to their Quality Control on Weighting respective populations. In 2013, weights were Procedures developed for students sampled at grades 4, 8, and 12 for assessments in mathematics and reading. Each student was assigned a weight to be used for making inferences about students in the target population. This weight is known as the final full-sample student weight and contains the following major components: the student base weight; school nonresponse adjustments; student nonresponse adjustments; school weight trimming adjustments; student weight trimming adjustments; and student raking adjustment. The student base weight is the inverse of the overall probability of selecting a student and assigning that student to a particular assessment. The sample design that determines the base weights is discussed in the NAEP 2013 sample design section. The student base weight is adjusted for two sources of nonparticipation: school level and student level. These weighting adjustments seek to reduce the potential for bias from such nonparticipation by increasing the weights of students from participating schools similar to those schools not participating; and increasing the weights of participating students similar to those students from within participating schools who did not attend the assessment session (or makeup session) as scheduled. Furthermore, the final weights reflect the trimming of extremely large weights at both the school and student level. These weighting adjustments seek to reduce variances of survey estimates. An additional weighting adjustment was implemented in the state and Trial Urban District Assessment (TUDA) samples so that estimates for key student-level characteristics were in agreement across assessments in reading and mathematics. This adjustment was implemented using a raking procedure. In addition to the final full-sample weight, a set of replicate weights was provided for each student. These replicate weights are used to calculate the variances of survey estimates using the jackknife repeated replication method. The methods used to derive these weights were aimed at reflecting the features of the sample design, so that when the jackknife variance estimation procedure is implemented, approximately unbiased estimates of sampling variance are obtained. In addition, the various weighting procedures were repeated on each set of replicate weights to appropriately reflect the impact of the weighting adjustments on the sampling variance of a survey estimate. A finite population correction (fpc) factor was incorporated into the replication scheme so that it could be reflected in the variance estimates for the reading and mathematics assessments. See Computation of Replicate Weights for Variance Estimation for details. Quality control checks were carried out throughout the weighting process to ensure the accuracy of the full-sample and replicate weights. See Quality Control for Weighting Procedures for the various checks implemented and main findings of interest. In the linked pages that follow, please note that Vocabulary, Reading Vocabulary, and Meaning Vocabulary refer to the same reporting scale and are interchangeable. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/naep_assessment_weighting_procedures.aspx Appendices A-C NAEP 2019-2020 19 NAEP Technical Documentation Website NAEP Technical Documentation Computation of FullSample W eights for the 2013 Assessment The full-sample or final student weight is the sampling weight used to derive NAEP student estimates of population and subpopulation characteristics for a specified grade (4, 8, or 12) and assessment subject (reading or mathematics). The full-sample student weight reflects the number of students that the sampled student represents in the population for purposes of estimation. The summation of the final student weights over a particular student group provides an estimate of the total number of students in that group within the population. Computation of Base Weights School and Student Nonresponse Weight Adjustments School and Student Weight Trimming Adjustments Student Weight Raking Adjustment The full-sample weight, which is used to produce survey estimates, is distinct from a replicate weight that is used to estimate variances of survey estimates. The full-sample weight is assigned to participating students and reflects the student base weight after the application of the various weighting adjustments. The full-sample weight for student k from school s in stratum j (FSTUWGTjsk) can be expressed as follows: where STU_BWTjsk is the student base weight; SCH_NRAFjs is the school-level nonresponse adjustment factor; STU_NRAFjsk is the student-level nonresponse adjustment factor; SCH_TRIMjs is the school-level weight trimming adjustment factor; STU_TRIMjsk is the student-level weight trimming adjustment factor; and STU_RAKEjsk is the student-level raking adjustment factor. School sampling strata for a given assessment vary by school type and grade. See the links below for descriptions of the school strata for the various assessments. Public schools at grades 4 and 8 Public schools at grade 12 Private schools at grades 4, 8 and 12 http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/computation_of_full_sample_weights_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 20 NAEP Technical Documentation Website NAEP Technical Documentation Computation of Base Weights for the 2013 Assessment Every sampled school and student received a base weight equal to the reciprocal of its probability of selection. Computation of a school base weight varies by School Base Weights Student Base Weights type of sampled school (original or substitute); and sampling frame (new school frame or not). Computation of a student base weight reflects the student's overall probability of selection accounting for school and student sampling; assignment to session type at the school- and student-level; and the student's assignment to the reading or mathematics assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/computation_of_base_weights_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 21 NAEP Technical Documentation Website NAEP Technical Documentation School Base Weights for the 2013 Assessment The school base weight for a sampled school is equal to the inverse of its overall probability of selection. The overall selection probability of a sampled school differs by type of sampled school (original or substitute); sampling frame (new school frame or not). The overall selection probability of an originally selected school in a reading or mathematics sample is equal to its probability of selection from the NAEP public/private school frame. The overall selection probability of a school from the new school frame in a reading or mathematics sample is the product of two quantities: the probability of selection of the school's district into the new-school district sample, and the probability of selection of the school into the new school sample. Substitute schools are preassigned to original schools and take the place of original schools if they refuse to participate. For weighting purposes, they are treated as if they were the original schools that they replaced; so substitute schools are assigned the school base weight of the original schools. Learn more about substitute schools for the 2013 private school national assessment and substitute schools for the 2013 twelfth grade public school assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/school_base_weights_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 22 NAEP Technical Documentation Website NAEP Technical Documentation Student Base Weights for the 2013 Assessment Every sampled student received a student base weight, whether or not the student participated in the assessment. The student base weight is the reciprocal of the probability that the student was sampled to participate in the assessment for a specified subject. The student base weight for student k from school s in stratum j (STU_BWTjsk) is the product of seven weighting components and can be expressed as follows: where SCH_BWTjs is the school base weight; SCHSsessionassignmentESWTjs is the school-level session assignment weight that reflects the conditional probability, given the school, that the particular session type was assigned to the school; WINSCHWTjs is the within-school student weight that reflects the conditional probability, given the school, that the student was selected for the NAEP assessment; STUSESWTjsk is Stu_bookmarkthe student-level session assignment weight that reflects the conditional probability, given that the particular session type was assigned to the school, that the student was assigned to the session type; SUBJFACsubjfacjsk is the subject spiral adjustment factor that reflects the conditional probability, given that the student was assigned to a particular session type, that the student was assigned the specified subject; SUBADJjs is the substitution adjustment factor to account for the difference in enrollment size between the substitute and original school; and YRRND_AFjs is the year-round adjustment factor to account for students in yearround schools on scheduled break at the time of the NAEP assessment and thus not available to be included in the sample. The within-school student weight (WINSCHWTjs) is the inverse of the student sampling rate in the school. The subject spiral adjustment factor (SUBJFACjsk) adjusts the student weight to account for the spiral pattern used in distributing reading or mathematics booklets to the students. The subject factor varies by grade, subject, and school type (public or private), and it is equal to the inverse of the booklet proportions (reading or mathematics) in the overall spiral for a specific sample. For cooperating substitutes of nonresponding original sampled schools, the substitution adjustment factor (SUBADJjs) is equal to the ratio of the estimated grade enrollment for the original sampled school to the estimated grade enrollment for the substitute school. The student sample from the substitute school then "represents" the set of grade-eligible students from the original sampled school. The year-round adjustment factor (YRRND_AFjs) adjusts the student weight for students in yearround schools who do not attend school during the time of the assessment. This situation typically arises in overcrowded schools. School administrators in year-round schools randomly assign students to portions of the year in which they attend school and portions of the year in which they do not attend. At the time of assessment, a certain percentage of students (designated as OFF js) do not attend school and thus cannot be assessed. The YRRND_AFjs for a school is calculated as 1/(1OFF js/100). http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/student_base_weights_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 23 NAEP Technical Documentation Website NAEP Technical Documentation School and Student Nonr esponse Weight Adjustments for the 2013 Assessment Nonresponse is unavoidable in any voluntary survey of a human population. Nonresponse School Nonresponse Weight leads to the loss of sample data that must be compensated for in the weights of the responding Adjustment sample members. This differs from ineligibility, for which no adjustments are necessary. The Student Nonresponse Weight purpose of the nonresponse adjustments is to reduce the mean square error of survey estimates. Adjustment While the nonresponse adjustment reduces the bias from the loss of sample, it also increases variability among the survey weights leading to increased variances of the sample estimates. However, it is presumed that the reduction in bias more than compensates for the increase in the variance, thereby reducing the mean square error and thus improving the accuracy of survey estimates. Nonresponse adjustments are made in the NAEP surveys at both the school and the student levels: the responding (original and substitute) schools receive a weighting adjustment to compensate for nonresponding schools, and responding students receive a weighting adjustment to compensate for nonresponding students. The paradigm used for nonresponse adjustment in NAEP is the quasi-randomization approach (Oh and Scheuren 1983). In this approach, school response cells are based on characteristics of schools known to be related to both response propensity and achievement level, such as the locale type (e.g., large principal city of a metropolitan area) of the school. Likewise, student response cells are based on characteristics of the schools containing the students and student characteristics, which are known to be related to both response propensity and achievement level, such as student race/ethnicity, gender, and age. Under this approach, sample members are assigned to mutually exclusive and exhaustive response cells based on predetermined characteristics. A nonresponse adjustment factor is calculated for each cell as the ratio of the sum of adjusted base weights for all eligible units to the sum of adjusted base weights for all responding units. The nonresponse adjustment factor is then applied to the base weight of each responding unit. In this way, the weights of responding units in the cell are "weighted up" to represent the full set of responding and nonresponding units in the response cell. The quasi-randomization paradigm views nonresponse as another stage of sampling. Within each nonresponse cell, the paradigm assumes that the responding sample units are a simple random sample from the total set of all sample units. If this model is valid, then the use of the quasi-randomization weighting adjustment will eliminate any nonresponse bias. Even if this model is not valid, the weighting adjustments will eliminate bias if the achievement scores are homogeneous within the response cells (i.e., bias is eliminated if there is homogeneity either in response propensity or in achievement levels). See, for example, chapter 4 of Little and Rubin (1987). http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/school_and_student_nonresponse_weight_adjustments_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 24 NAEP Technical Documentation Website NAEP Technical Documentation School Nonresponse Weight Adjustment The school nonresponse adjustment procedure inflates the weights of cooperating schools to account for eligible noncooperating schools for which no substitute schools participated. The adjustments are computed within nonresponse cells and are based on the assumption that the cooperating and noncooperating schools within the same cell are more similar to each other than to schools from different cells. School nonresponse adjustments were carried out separately by sample; that is, by Development of Initial School Nonresponse Cells Development of Final School Nonresponse Cells School Nonresponse Adjustment Factor Calculation sample level (state, national), school type (public, private), and grade (4, 8, 12). http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/school_nonresponse_weight_adjustment_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 25 NAEP Technical Documentation Website NAEP Technical Documentation Development of Initial School Nonresponse Cells The cells for nonresponse adjustments are generally functions of the school sampling strata for the individual samples. School sampling strata usually differ by assessment subject, grade, and school type (public or private). Assessment subjects that are administered together by way of spiraling have the same school samples and stratification schemes. Subjects that are not spiraled with any other subjects have their own separate school sample. In NAEP 2015, all operational assessments were spiraled together. The initial nonresponse cells for the various NAEP 2015 samples are described below. Public School Samples for Reading and Mathematics at Grades 4 and 8 For these samples, initial weighting cells were formed within each jurisdiction using the following nesting cell structure: Trial Urban District Assessment (TUDA) district vs. the balance of the state for states with TUDA districts, urbanicity (urban-centric locale) stratum; and race/ethnicity classification stratum, or achievement level, or median income, or grade enrollment. In general, the nonresponse cell structure used race/ethnicity classification stratum as the lowest level variable. However, where there was only one race/ethnicity classification stratum within a particular urbanicity stratum, categorized achievement, median income, or enrollment data were used instead. Public School Sample at Grade 12 The initial weighting cells for this sample were formed using the following nesting cell structure: census division stratum, urbanicity stratum (urban-centric locale), and race/ethnicity classification stratum. Private School Samples at Grades 4, 8 and 12 The initial weighting cells for these samples were formed within each grade using the following nesting cell structure: affiliation, census division stratum, urbanicity stratum (urban-centric locale), and race/ethnicity classification stratum. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/development_of_initial_school_nonresponse_cells_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 26 NAEP Technical Documentation Website NAEP Technical Documentation Development of Final School Nonresponse Cells Limits were placed on the magnitude of cell sizes and adjustment factors to prevent unstable nonresponse adjustments and unacceptably large nonresponse factors. All initial weighting cells with fewer than six cooperating schools or adjustment factors greater than 3.0 for the full sample weight were collapsed with suitable adjacent cells. Simultaneously, all initial weighting cells for any replicate with fewer than four cooperating schools or adjustment factors greater than the maximum of 3.0 or two times the full sample nonresponse adjustment factor were collapsed with suitable adjacent cells. Initial weighting cells were generally collapsed in reverse order of the cell structure; that is, starting at the bottom of the nesting structure and working up toward the top level of the nesting structure. Public School Samples at Grades 4 and 8 For the grade 4 and 8 public school samples, cells with the most similar race/ethnicity classification within a given jurisdiction/Trial Urban District Assessment (TUDA) district and urbanicity (urban-centric locale) stratum were collapsed first. If further collapsing was required after all levels of race/ethnicity strata were collapsed, cells with the most similar urbanicity strata were combined next. Cells were never permitted to be collapsed across jurisdictions or TUDA districts. Public School Sample at Grades 12 For the grade 12 public school sample, race/ethnicity classification cells within a given census division stratum and urbanicity stratum were collapsed first. If further collapsing was required after all levels of race/ethnicity classification were collapsed, cells with the most similar urbanicity strata were combined next. Any further collapsing occurred across census division strata but never across census regions. Private School Samples at Grades 4, 8, and 12 For the private school samples, cells with the most similar race/ethnicity classification within a given affiliation, census division, and urbanicity stratum were collapsed first. If further collapsing was required after all levels of race/ethnicity strata were collapsed, cells with the most similar urbanicity classification were combined. Any further collapsing occurred across census division strata but never across affiliations. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/development_of_final_school_nonresponse_cells_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 27 NAEP Technical Documentation Website NAEP Technical Documentation School Nonresponse Adjustment Factor Calculation In each final school nonresponse adjustment cell c, the school nonresponse adjustment factor SCH_NRAFc was computed as follows: where Sc is the set of all eligible sampled schools (cooperating original and substitute schools and refusing original schools with noncooperating or no assigned substitute) in cell c, Rc is the set of all cooperating schools within Sc, SCH_BWTs is the school base weight, SCH_TRIMs is the school-level weight trimming factor, SCHSESWTs is the school-level session assignment weight, and Xs is the estimated grade enrollment corresponding to the original sampled school. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/school_nonresponse_adjustment_factor_calculation_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 28 NAEP Technical Documentation Website NAEP Technical Documentation Student Nonresponse Weight Adjustment The student nonresponse adjustment procedure inflates the Development of Initial Student Nonresponse weights of assessed students to account for eligible sampled Cells students who did not participate in the assessment. These Development of Final Student Nonresponse inflation factors offset the loss of data associated with absent Cells students. The adjustments are computed within nonresponse cells and are based on the assumption that the assessed and Student Nonresponse Adjustment Factor absent students within the same cell are more similar to one Calculation another than to students from different cells. Like its counterpart at the school level, the student nonresponse adjustment is intended to reduce the mean square error and thus improve the accuracy of NAEP assessment estimates. Also, like its counterpart at the school level, student nonresponse adjustments were carried out separately by sample; that is, by grade (4, 8, 12), school type (public, private), and assessment subject (mathematics, reading, science, meaning vocabulary). http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/student_nonresponse_weight_adjustment_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 29 NAEP Technical Documentation Website NAEP Technical Documentation Development of Initial Student Nonresponse Cells for the 2013 Assessment Initial student nonresponse cells are generally created within each sample as defined by grade, school type (public, private), and assessment subject. However, when subjects are administered together by way of spiraling, the initial student nonresponse cells are created across the subjects in the same spiral. The rationale behind this decision is that spiraled subjects are in the same schools and the likelihood of whether an eligible student participates in an assessment is more related to its school than the subject of the assessment booklet. In NAEP 2013, there was only one spiral, with the reading and mathematics assessments spiraled together. The initial student nonresponse cells for the various NAEP 2013 samples are described below. Nonresponse adjustment procedures are not applied to excluded students because they are not required to complete an assessment. Public School Samples for Reading and Mathematics at Grades 4 and 8 The initial student nonresponse cells for these samples were defined within grade, jurisdiction, and Trial Urban District Assessment (TUDA) district using the following nesting cell structure: students with disabilities (SD)/English language learners (ELL) by subject, school nonresponse cell, age (classified into "older"1 student and "modal age or younger" student), gender, and race/ethnicity. The highest level variable in the cell structure separates students who were classified either as having disabilities (SD) or as English language learners (ELL) from those who are neither, since SD or ELL students tend to score lower on assessment tests than non-SD/non-ELL students. In addition, the students in the SD or ELL groups are further broken down by subject, since rules for excluding students from the assessment differ by subject. Non-SD and non-ELL students are not broken down by subject, since the exclusion rules do not apply to them. Public School Samples for Reading and Mathematics at Grade 12 The initial weighting cells for these samples were formed hierarchically within state for the state-reportable samples and the balance of the country for remaining states as follows: SD/ELL, school nonresponse cell, age (classified into "older"1 student and "modal age or younger" student), gender, and race/ethnicity. Private School Samples for Reading and Mathematics at Grades 4, 8, and 12 The initial weighting cells for these private school samples were formed hierarchically within grade as follows: SD/ELL, school nonresponse cell, age (classified into "older"1 student and "modal age or younger" student), gender, and race/ethnicity. Although exclusion rules differ by subject, there were not enough SD or ELL private school students to break out by subject as was done for the public schools. 1Older students are those born before October 1, 2002, for grade 4; October 1, 1998, for grade 8; and October 1, 1994, for grade 12. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/development_of_initial_student_nonresponse_cells_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 30 NAEP Technical Documentation Website NAEP Technical Documentation Development of Final Student Nonresponse Cells for the 2013 Assessment Similar to the school nonresponse adjustment, cell and adjustment factor size constraints are in place to prevent unstable nonresponse adjustments or unacceptably large adjustment factors. All initial weighting cells with either fewer than 20 participating students or adjustment factors greater than 2.0 for the full sample weight were collapsed with suitable adjacent cells. Simultaneously, all initial weighting cells for any replicate with either fewer than 15 participating students or an adjustment factor greater than the maximum of 2.0 or 1.5 times the full sample nonresponse adjustment factor were collapsed with suitable adjacent cells. Initial weighting cells were generally collapsed in reverse order of the cell structure; that is, starting at the bottom of the nesting structure and working up toward the top level of the nesting structure. Race/ethnicity cells within SD/ELL groups, school nonresponse cell, age, and gender classes were collapsed first. If further collapsing was required after collapsing all race/ethnicity classes, cells were next combined across gender, then age, and finally school nonresponse cells. Cells are never collapsed across SD and ELL groups for any sample. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/development_of_final_student_nonresponse_cells_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 31 NAEP Technical Documentation Website NAEP Technical DocumentationStudent Nonr esponse Adjustment Factor Calculation In each final student nonresponse adjustment cell c for a given sample, the student nonresponse adjustment factor STU_NRAFc was computed as follows: where Sc is the set of all eligible sampled students in cell c for a given sample, Rc is the set of all assessed students within Sc, STU_BWTk is the student base weight for a given student k, SCH_TRIMk is the school-level weight trimming factor for the school associated with student k, SCH_NRAFk is the school-level nonresponse adjustment factor for the school associated with student k, and SUBJFACk is the subject factor for a given student k. The student weight used in the calculation above is the adjusted student base weight, without regard to subject, adjusted for school weight trimming and school nonresponse. Nonresponse adjustment procedures are not applied to excluded students because they are not required to complete an assessment. In effect, excluded students were placed in a separate nonresponse cell by themselves and all received an adjustment factor of 1. While excluded students are not included in the analysis of the NAEP scores, weights are provided for excluded students in order to estimate the size of this group and its population characteristics. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/student_nonresponse_adjustment_factor_calculation_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 32 NAEP Technical Documentation Website NAEP Technical Documentation School and Student Weight Trimming Adjustments for the 2013 Assessment Weight trimming is an adjustment procedure that involves detecting and reducing extremely large Trimming of School weights. "Extremely large weights" generally refer to large sampling weights that were not Base Weights anticipated in the design of the sample. Unusually large weights are likely to produce large Trimming of Student sampling variances for statistics of interest, especially when the large weights are associated with Weights sample cases reflective of rare or atypical characteristics. To reduce the impact of these large weights on variances, weight reduction methods are typically employed. The goal of employing weight reduction methods is to reduce the mean square error of survey estimates. While the trimming of large weights reduces variances, it also introduces some bias. However, it is presumed that the reduction in the variances more than compensates for the increase in the bias, thereby reducing the mean square error and thus improving the accuracy of survey estimates (Potter 1988). NAEP employs weight trimming at both the school and student levels. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/school_and_student_weight_trimming_adjustments_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 33 NAEP Technical Documentation Website NAEP Technical DocumentationTrimming of School Base Weights Large school weights can occur for schools selected from the NAEP new-school sampling frame and for private schools. New schools that are eligible for weight trimming are schools with a disproportionately large student enrollment in a particular grade from a school district that was selected with a low probability of selection. The school base weights for such schools may be large relative to what they would have been if they had been selected as part of the original sample. To detect extremely large weights among new schools, a comparison was made between a new school's school base weight and its ideal weight (i.e., the weight that would have resulted had the school been selected from the original school sampling frame). If the school base weight was more than three times the ideal weight, a trimming factor was calculated for that school that scaled the base weight back to three times the ideal weight. The calculation of the school-level trimming factor for a new school s is expressed in the following formula: where EXP_WTs is the ideal base weight the school would have received if it had been on the NAEP public school sampling frame, and SCH_BWTs is the actual school base weight the school received as a sampled school from the new school frame. Thirty-seven (37) schools out of 377 selected from the new-school sampling frame had their weights trimmed: eight at grade 4, 29 at grade 8, and zero at grade 12. Private schools eligible for weight trimming were Private School Universe Survey (PSS) nonrespondents who were found subsequently to have either larger enrollments than assumed at the time of sampling, or an atypical probability of selection given their affiliation, the latter being unknown at the time of sampling. For private school s, the formula for computing the school-level weight trimming factor SCH_TRIMs is identical to that used for new schools. For private schools, EXP_WTs is the ideal base weight the school would have received if it had been on the NAEP private school sampling frame with accurate enrollment and known affiliation, and SCH_BWTs is the actual school base weight the school received as a sampled private school. No private schools had their weights trimmed. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/trimming_of_school_base_weights_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 34 NAEP Technical Documentation Website NAEP Technical DocumentationTrimming of Student Weights Large student weights generally come from compounding nonresponse adjustments at the school and student levels with artificially low school selection probabilities, which can result from inaccurate enrollment data on the school frame used to define the school size measure. Even though measures are in place to limit the number and size of excessively large weights—such as the implementation of adjustment factor size constraints in both the school and student nonresponse procedures and the use of the school trimming procedure—large student weights can occur due to compounding effects of the various weighting components. The student weight trimming procedure uses a multiple median rule to detect excessively large student weights. Any student weight within a given trimming group greater than a specified multiple of the median weight value of the given trimming group has its weight scaled back to that threshold. Student weight trimming was implemented separately by grade, school type (public or private), and subject. The multiples used were 3.5 for public school trimming groups and 4.5 for private school trimming groups. Trimming groups were defined by jurisdiction and Trial Urban District Assessment (TUDA) districts for the public school samples at grades 4 and 8; by dichotomy of low/high percentage of Black and Hispanic students (15 percent and below, above 15 percent) for the public school sample at grade 12; and by affiliation (Catholic, Non-Catholic) for private school samples at grades 4, 8 and 12. The procedure computes the median of the nonresponse-adjusted student weights in the trimming group g for a given grade and subject sample. Any student k with a weight more than M times the median received a trimming factor calculated as follows: where M is the trimming multiple, MEDIANg is the median of nonresponse-adjusted student weights in trimming group g, and STUWGTgk is the weight after student nonresponse adjustment for student k in trimming group g. In the 2013 assessment, relatively few students had weights considered excessively large. Out of the approximately 840,000 students included in the combined 2013 assessment samples, 226 students had their weights trimmed. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/trimming_of_student_weights_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 35 NAEP Technical Documentation Website NAEP Technical DocumentationStudent Weight Raking Adjustment for the 2013 Assessment Development of Final Raking Dimensions Weighted estimates of population totals for student-level subgroups for a given grade will vary across subjects even Raking Adjustment Control Totals though the student samples for each subject generally come from the same schools. These differences are the Raking Adjustment Factor Calculation result of sampling error associated with the random assignment of subjects to students through a process known as spiraling. For state assessments in particular, any difference in demographic estimates between subjects, no matter how small, may raise concerns about data quality. To remove these random differences and potential data quality concerns, a new step was added to the NAEP weighting procedure starting in 2009. This step adjusts the student weights in such a way that the weighted sums of population totals for specific subgroups are the same across all subjects. It was implemented using a raking procedure and applied only to state-level assessments. Raking is a weighting procedure based on the iterative proportional fitting process developed by Deming and Stephan (1940) and involves simultaneous ratio adjustments to two or more marginal distributions of population totals. Each set of marginal population totals is known as a dimension, and each population total in a dimension is referred to as a control total. Raking is carried out in a sequence of adjustments. Sampling weights are adjusted to one marginal distribution and then to the second marginal distribution, and so on. One cycle of sequential adjustments to the marginal distributions is called an iteration. The procedure is repeated until convergence is achieved. The criterion for convergence can be specified either as the maximum number of iterations or an absolute difference (or relative absolute difference) from the marginal population totals. More discussion on raking can be found in Oh and Scheuren (1987). For NAEP 2013, the student raking adjustment was carried out separately in each state for the reading and mathematics public school samples at grades 4 and 8, and in the 13 states with state-reportable samples for the reading and mathematics public school samples at grade 12. The dimensions used in the raking process were National School Lunch Program (NSLP) eligibility, race/ethnicity, SD/ELL status, and gender. The control totals for these dimensions were obtained from the NAEP student sample weights of the reading and mathematics samples combined. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/student_weight_raking_adjustment_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 36 NAEP Technical Documentation Website NAEP Technical Documentation Development of Final Raking Dimensions The raking procedure involved four dimensions. The variables used to define the dimensions are listed below along with the categories making up the initial raking cells for each dimension. National School Lunch Program (NSLP) eligibility 1. Eligible for free or reduced-price lunch 2. Otherwise Race/Ethnicity 1. White, not Hispanic 2. Black, not Hispanic 3. Hispanic 4. Asian 5. American Indian/Alaska Native 6. Native Hawaiian/Pacific Islander 7. Two or More Races SD/ELL status 1. SD, but not ELL 2. ELL, but not SD 3. SD and ELL 4. Neither SD nor ELL Gender 1. Male 2. Female In states containing districts that participated in Trial Urban District Assessments (TUDA) districts at grades 4 and 8, the initial cells were created separately for each TUDA district and the balance of the state. Similar to the procedure used for school and student nonresponse adjustments, limits were placed on the magnitude of the cell sizes and adjustment factors to prevent unstable raking adjustments that could have resulted in unacceptably large or small adjustment factors. Levels of a dimension were combined whenever there were fewer than 30 assessed or excluded students (20 for any of the replicates) in a category, if the smallest adjustment was less than 0.5, or if the largest adjustment was greater than 2 for the full sample or for any replicate. If collapsing was necessary for the race/ethnicity dimension, the following groups were combined first: American Indian/Alaska Native with Black, not Hispanic; Hawaiian/Pacific Islander with Black, not Hispanic; Two or More Races with White, not Hispanic; Asian with White, not Hispanic; and Black, not Hispanic with Hispanic. If further collapsing was necessary, the five categories American Indian/Alaska Native; Two or More Races; Asian; Native Hawaiian/Pacific Islander; and White, not Hispanic were combined. In some instances, all seven categories had to be collapsed. If collapsing was necessary for the SD/ELL dimension, the SD/not ELL and SD/ELL categories were combined first, followed by ELL/not SD if further collapsing was necessary. In some instances, all four categories had to be collapsed. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/development_of_final_raking_dimensions_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 37 NAEP Technical Documentation Website NAEP Technical DocumentationRaking Adjustment Control Totals for the 2013 Assessment The control totals used in the raking procedure for NAEP 2013 grades 4, 8, and 12 were estimates of the student population derived from the set of assessed and excluded students pooled across subjects. The control totals for category c within dimension d were computed as follows: where Rc(d) is the set of all assessed students in category c of dimension d, Ec(d) is the set of all excluded students in category c of dimension d, STU_BWTk is the student base weight for a given student k, SCH_TRIMk is the school-level weight trimming factor for the school associated with student k, SCH_NRAFk is the school-level nonresponse adjustment factor for the school associated with student k, STU_NRAFk is the student-level nonresponse adjustment factor for student k, and SUBJFACk is the subject factor for student k. The student weight used in the calculation of the control totals above is the adjusted student base weight, without regard to subject, adjusted for school weight trimming, school nonresponse, and student nonresponse. Control totals were computed for the full sample and for each replicate independently. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/raking_adjustment_control_totals_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 38 NAEP Technical Documentation W ebsite NAEP Technical DocumentationRaking Adjustment Factor Calculation for the 2013 Assessment For assessed and excluded students in a given subject, the raking adjustment factor STU_RAKEk was computed as follows: First, the weight for student k was initialized as follows: where STU_BWTk is the student base weight for a given student k, SCH_TRIMk is the school-level weight trimming factor for the school associated with student k, SCH_NRAFk is the school-level nonresponse adjustment factor for the school associated with student k, STU_NRAFk is the student-level nonresponse adjustment factor for student k, and SUBJFACk is the subject factor for student k. Then, the sequence of weights for the first iteration was calculated as follows for student k in category c of dimension d: For dimension 1: For dimension 2: For dimension 3: For dimension 4: Appendices A-C NAEP 2019-2020 39 where Rc(d) is the set of all assessed students in category c of dimension d, Ec(d) is the set of all excluded students in category c of dimension d, and Total c(d) is the control total for category c of dimension d. The process is said to converge if the maximum difference between the sum of adjusted weights and the control totals is 1.0 for each category in each dimension. If after the sequence of adjustments the maximum difference was greater than 1.0, the process continues to the next iteration, cycling back to the first dimension with the initial weight for student k equaling STUSAWTkadj(4) from the previous iteration. The process continued until convergence was reached. Once the process converged, the adjustment factor was computed as follows: where STUSAWTk is the weight for student k after convergence. The process was done independently for the full sample and for each replicate. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/raking_adjustment_factor_calculation_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 40 NAEP Technical Documentation W ebsite NAEP Technical DocumentationComputation of Replicate W eights for the 2013 Assessment Defining Variance Strata and Forming In addition to the full-sample weight, a set of 62 replicate Replicates weights was provided for each student. These replicate Computing School-Level Replicate Factors weights are used in calculating the sampling variance of estimates obtained from the data, using the jackknife repeated Computing Student-Level Replicate replication method. The method of deriving these weights was Factors aimed at reflecting the features of the sample design appropriately for each sample, so that when the jackknife Replicate Variance Estimation variance estimation procedure is implemented, approximately unbiased estimates of sampling variance are obtained. This section gives the specifics for generating the replicate weights for the 2013 assessment samples. The theory that underlies the jackknife variance estimators used in NAEP studies is discussed in the section Replicate Variance Estimation. In general, the process of creating jackknife replicate weights takes place at both the school and student level. The precise implementation differs between those samples that involve the selection of Primary Sampling Units (PSUs) and those where the school is the first stage of sampling. The procedure for this second kind of sample also differed starting in 2011 from all previous NAEP assessments. The change that was implemented permitted the introduction of a finite population correction factor at the school sampling stage, developed by Rizzo and Rust (2011). In assessments prior to 2011, this adjustment factor has always been implicitly assumed equal to 1.0, resulting in some overestimation of the sampling variance. For each sample, the calculation of replicate weighting factors at the school level was conducted in a series of steps. First, each school was assigned to one of 62 variance estimation strata. Then, a random subset of schools in each variance estimation stratum was assigned a replicate factor of between 0 and 1. Next, the remaining subset of schools in the same variance stratum was assigned a complementary replicate factor greater than 1. All schools in the other variance estimation strata were assigned a replicate factor of exactly 1. This process was repeated for each of the 62 variance estimation strata so that 62 distinct replicate factors were assigned to each school in the sample. This process was then repeated at the student level. Here, each individual sampled student was assigned to one of 62 variance estimation strata, and 62 replicate factors with values either between 0 and 1, greater than 1, or exactly equal to 1 were assigned to each student. For example, consider a single hypothetical student. For replicate 37, that student’s student replicate factor might be 0.8, while for the school to which the student belongs, for replicate 37, the school replicate factor might be 1.6. Of course, for a given student, for most replicates, either the student replicate factor, the school replicate factor, or (usually) both, is equal to 1.0. A replicate weight was calculated for each student, for each of the 62 replicates, using weighting procedures similar to those used for the full-sample weight. Each replicate weight contains the school and student replicate factors described above. By repeating the various weighting procedures on each set of replicates, the impact of these procedures on the sampling variance of an estimate is appropriately reflected in the variance estimate. Each of the 62 replicate weights for student k in school s in stratum j can be expressed as follows: where STU_BWTjsk is the student base weight; SCH_REPFACjs(r) is the school-level replicate factor for replicate r; SCH_NRAFjs(r) is the school-level nonresponse adjustment factor for replicate r; STU_REPFACjsk(r) is the student-level replicate factor for replicate r; STU_NRAFjsk(r) is the student-level nonresponse adjustment factor for replicate r; SCH_TRIMjs is the school-level weight trimming adjustment factor; STU_TRIMjsk is the student-level weight trimming adjustment factor; and STU_RAKEjsk(r) is the student-level raking adjustment factor for replicate r. Specific school and student nonresponse and student-level raking adjustment factors were calculated separately for each replicate, thus the use of the index (r), and applied to the replicate student base weights. Computing separate nonresponse and raking adjustment factors for each replicate allows resulting variances from the use of the final student replicate weights to reflect components of variance due to these various weight adjustments. Appendices A-C NAEP 2019-2020 41 School and student weight trimming adjustments were not replicated, that is, not calculated separately for each replicate. Instead, each replicate used the school and student trimming adjustment factors derived for the full sample. Statistical theory for replicating trimming adjustments under the jackknife approach has not been developed in the literature. Due to the absence of a statistical framework, and since relatively few school and student weights in NAEP require trimming, the weight trimming adjustments were not replicated. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/computation_of_replicate_weights_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 42 NAEP Technical Documentation Website NAEP Technical Documentation Defining Variance Strata and Forming Replicates for the 2013 Assessment In the NAEP 2013 assessment, replicates were formed separately for each sample indicated by grade (4, 8, 12), school type (public, private), and assessment subject (mathematics, reading). To reflect the school-level finite population corrections in the variance estimators for the two-stage samples used for the mathematics and reading assessments, replication was carried out at both the school and student levels. The first step in forming replicates was to create preliminary variance strata in each primary stratum. This was done by sorting the appropriate sampling unit (school or student) in the order of its selection within the primary stratum and then pair off adjacent sampling units into preliminary variance strata. Sorting sample units by their order of sample selection reflects the implicit stratification and systematic sampling features of the sample design. Within each primary stratum with an even number of sampling units, all of the preliminary variance strata consisted of pairs of sampling units. However, within primary strata with an odd number of sampling units, all but one variance strata consisted of pairs of sampling units, while the last one consisted of three sampling units. The next step is to form the final variance strata by combining preliminary strata if appropriate. If there were more than 62 preliminary variance strata within a primary stratum, the preliminary variance strata were grouped to form 62 final variance strata. This grouping effectively maximized the distance in the sort order between grouped preliminary variance strata. The first 62 preliminary variance strata, for example, were assigned to 62 different final variance strata in order (1 through 62), with the next 62 preliminary variance strata assigned to final variance strata 1 through 62, so that, for example, preliminary variance stratum 1, preliminary variance stratum 63, preliminary variance stratum 125 (if in fact there were that many), etc., were all assigned to the first final variance stratum. If, on the other hand, there were fewer than 62 preliminary variance strata within a primary stratum, then the number of final variance strata was set equal to the number of preliminary variance strata. For example, consider a primary stratum with 111 sampled units sorted in their order of selection. The first two units were in the first preliminary variance stratum; the next two units were in the second preliminary variance stratum, and so on, resulting in 54 preliminary variance strata with two sample units each (doublets). The last three sample units were in the 55th preliminary variance stratum (triplet). Since there are no more than 62 preliminary variance strata, these were also the final variance strata. Within each preliminary variance stratum containing a pair of sampling units, one sampling unit was randomly assigned as the first variance unit and the other as the second variance unit. Within each preliminary variance stratum containing three sampling units, the three first-stage units were randomly assigned variance units 1 through 3. Reading and Mathematics Assessments At the school-level for these samples, formation of preliminary variance strata did not pertain to certainty schools, since they are not subject to sampling variability, but only to noncertainty schools. The primary stratum for noncertainty schools was the highest school-level sampling stratum variable listed below, and the order of selection was defined by sort order on the school sampling frame. Trial Urban District Assessment (TUDA) districts, remainder of states (for states with TUDAs), or entire states for the public school samples at grades 4, 8, and 12; and Private school affiliation (Catholic, non-Catholic) for the private school samples at grades 4, 8, and 12. At the student-level, all students were assigned to variance strata. The primary stratum was school, and the order of selection was defined by session number and position on the administration schedule. Within each pair of preliminary variance strata, one first-stage unit, designated at random, was assigned as the first variance unit and the other first-stage unit as the second variance unit. Within each triplet preliminary variance stratum, the three schools were randomly assigned variance units 1 through 3. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/defining_variance_strata_and_forming_replicates_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 43 NAEP Technical Documentation W ebsite NAEP Technical DocumentationComputing SchoolLevel Replicate Factors for the 2013 Assessment The replicate variance estimation approach for the mathematics and reading assessments involved finite population corrections at the school level. The calculation of school-level replicate factors for these assessments depended upon whether or not a school was selected with certainty. For certainty schools, the school-level replicate factors for all replicates are set to unity – this is true regardless of whether or not the variance replication method uses finite population corrections – since certainty schools are not subject to sampling variability. Alternatively, one can view the finite population correction factor for such schools as being equal to zero. Thus, for each certainty school in a given assessment, the school-level replicate factor for each of the 62 replicates (r = 1, ..., 62) was assigned as follows: where SCH_REPFACjs(r) is the school-level replicate factor for school s in primary stratum j for the r-th replicate. For noncertainty schools, where preliminary variance strata were formed by grouping schools into pairs or triplets, school-level replicate factors were calculated for each of the 62 replicates based on this grouping. For schools in variance strata comprising pairs of schools, the school-level replicate factors,SCH_REPFACjs(r),r = 1,..., 62, were calculated as follows: where min(πj1, πj2) is the smallest school probability between the two schools comprising Rjr , Rjr is the set of schools within the r-th variance stratum for primary stratum j, and Ujs is the variance unit (1 or 2) for school s in primary stratum j. For noncertainty schools in preliminary variance strata comprising three schools, the school-level replicate factors SCH_REPFACjs(r), r = 1,..., 62 were calculated as follows: For school s from primary stratum j, variance stratum r, while for r' = r + 31 (mod 62): Appendices A-C NAEP 2019-2020 44 and for all other r* other than r and r': where min(πj1, πj2,πj3) is the smallest school probability among the three schools comprising Rjr , Rjr is the set of schools within the r-th variance stratum for primary stratum j, and Ujs is the variance unit (1, 2, or 3) for school s in primary stratum j. In primary strata with fewer than 62 variance strata, the replicate weights for the “unused” variance strata (the remaining ones up to 62) for these schools were set equal to the school base weight (so that those replicates contribute nothing to the variance estimate). http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/computing_school_level_replicate_factors_for_the_2013_assessment_.aspx Appendices A-C NAEP 2019-2020 45 NAEP Technical Documentation W ebsite NAEP Technical DocumentationComputing StudentLevel Replicate Factors for the 2013 Assessment For the mathematics and reading assessments, which involved school-level finite population corrections, the studentlevel replication factors were calculated the same way regardless of whether or not the student was in a certainty school. For students in student-level variance strata comprising pairs of students, the student-level replicate factors, STU_REPFACjsk(r), r = 1,..., 62, were calculated as follows: where πs is the probability of selection for school s, Rjsr is the set of students within the r-th variance stratum for school s in primary stratum j, and Ujsk is the variance unit (1 or 2) for student k in school s in stratum j. For students in variance strata comprising three students, the student-level replicate factors STU_REPFACjsk(r), r = 1,..., 62, were calculated as follows: while for r' = r + 31 (mod 62): and for all other r* other than r and r': where πs is the probability of selection for school s, Rjsr is the set of students within the r-th replicate stratum for school s in stratum j, and Ujsk is the variance unit (1, 2, or 3) for student k in school s in stratum j. Note, for students in certainty schools, where πs = 1, the student replicate factors are 2 and 0 in the case of pairs, and 1.5, 1.5, and 0 in the case of triples. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/computing_student_level_replicate_factors_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 46 NAEP Technical Documentation Website NAEP Technical Documentation Replicate Variance Estimation for the 2013 Assessment Variances for NAEP assessment estimates are computed using the paired jackknife replicate variance procedure. This technique is applicable for common statistics, such as means and ratios, and differences between these for different subgroups, as well as for more complex statistics such as linear or logistic regression coefficients. In general, the paired jackknife replicate variance procedure involves initially pairing clusters of first-stage sampling units to form H variance strata (h = 1, 2, 3, ...,H) with two units per stratum. The first replicate is formed by assigning, to one unit at random from the first variance stratum, a replicate weighting factor of less than 1.0, while assigning the remaining unit a complementary replicate factor greater than 1.0, and assigning all other units from the other (H - 1) strata a replicate factor of 1.0. This procedure is carried out for each variance stratum resulting in H replicates, each of which provides an estimate of the population total. In general, this process is repeated for subsequent levels of sampling. In practice, this is not practicable for a design with three or more stages of sampling, and the marginal improvement in precision of the variance estimates would be negligible in all such cases in the NAEP setting. Thus in NAEP, when a two-stage design is used – sampling schools and then students – beginning in 2011 replication is carried out at both stages. (See Rizzo and Rust (2011) for a description of the methodology.) When a three-stage design is used, involving the selection of geographic Primary Sampling Units (PSUs), then schools, and then students, the replication procedure is only carried out at the first stage of sampling (the PSU stage for noncertainty PSUs, and the school stage within certainty PSUs). In this situation, the school and student variance components are correctly estimated, and the overstatement of the between-PSU variance component is relatively very small. The jackknife estimate of the variance for any given statistic is given by the following formula: where represents the full sample estimate of the given statistic, and represents the corresponding estimate for replicate h. Each replicate undergoes the same weighting procedure as the full sample so that the jackknife variance estimator reflects the contributions to or reductions in variance resulting from the various weighting adjustments. The NAEP jackknife variance estimator is based on 62 variance strata resulting in a set of 62 replicate weights assigned to each school and student. The basic idea of the paired jackknife variance estimator is to create the replicate weights so that use of the jackknife procedure results in an unbiased variance estimator for simple totals and means, which is also reasonably efficient (i.e., has a low variance as a variance estimator). The jackknife variance estimator will then produce a consistent (but not fully unbiased) estimate of variance for (sufficiently smooth) nonlinear functions of total and mean estimates such as ratios, regression coefficients, and so forth (Shao and Tu, 1995). The development below shows why the NAEP jackknife variance estimator returns an unbiased variance estimator for totals and means, which is the cornerstone to the asymptotic results for nonlinear estimators. See for example Rust (1985). This paper also discusses why this variance estimator is generally efficient (i.e., more reliable than alternative approaches requiring similar computational resources). The development is done for an estimate of a mean based on a simplified sample design that closely approximates the sample design for first-stage units used in the NAEP studies. The sample design is a stratified random sample with H strata with population weights Wh, stratum sample sizes nh, and stratum sample means . The population estimator and standard unbiased variance estimator are: with Appendices A-C NAEP 2019-2020 47 The paired jackknife replicate variance estimator assigns one replicate h=1,…, H to each stratum, so that the number of replicates equals H. In NAEP, the replicates correspond generally to pairs and triplets (with the latter only being used if there are an odd number of sample units within a particular primary stratum generating replicate strata). For pairs, the process of generating replicates can be viewed as taking a simple random sample (J) of size nh/2 within the replicate stratum, and assigning an increased weight to the sampled elements, and a decreased weight to the unsampled elements. In certain applications, the increased weight is double the full sample weight, while the decreased weight is in fact equal to zero. In this simplified case, this assignment reduces to replacing with , the latter being the sample mean of the sampled nh/2 units. Then the replicate estimator corresponding to stratum r is The r-th term in the sum of squares for is thus: In stratified random sampling, when a sample of size nr /2 is drawn without replacement from a population of size nr ,, the sampling variance is See for example Cochran (1977), Theorem 5.3, using nr, as the “population size,” nr /2 as the “sample size,” and sr 2 as the “population variance” in the given formula. Thus, Taking the expectation over all of these stratified samples of size nr /2, it is found that In this sense, the jackknife variance estimator “gives back” the sample variance estimator for means and totals as desired under the theory. In cases where, rather than doubling the weight of one half of one variance stratum and assigning a zero weight to the other, the weight of one unit is multiplied by a replicate factor of (1+δ), while the other is multiplied by (1- δ), the result is that In this way, by setting δ equal to the square root of the finite population correction factor, the jackknife variance estimator is able to incorporate a finite population correction factor into the variance estimator. Appendices A-C NAEP 2019-2020 48 In practice, variance strata are also grouped to make sure that the number of replicates is not too large (the total number of variance strata is usually 62 for NAEP). The randomization from the original sample distribution guarantees that the sum of squares contributed by each replicate will be close to the target expected value. For triples, the replicate factors are perturbed to something other than 1.0 for two different replicate factors, rather than just one as in the case of pairs. Again in the simple case where replicate factors that are less than 1 are all set to 0, with the replicate weight factors calculated as follows. For unit i in variance stratum r where weight wi is the full sample base weight. Furthermore, for r' = r + 31 (mod 62): And for all other values r*, other than r and r´,wi(r*) = 1. In the case of stratified random sampling, this formula reduces to replacing replicate r, where with for is the sample mean from a “2/3” sample of 2nr /3 units from the nr sample units in the replicate stratum, and replacing with for replicate r', where is the sample mean from another overlapping “2/3” sample of 2nr /3 units from the nr sample units in the replicate stratum. The r-th and r´-th replicates can be written as: From these formulas, expressions for the r-th and r´-th components of the jackknife variance estimator are obtained (ignoring other sums of squares from other grouped components attached to those replicates): These sums of squares have expectations as follows, using the general formula for sampling variances: Appendices A-C NAEP 2019-2020 49 Thus, as desired again. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/replicate_variance_estimation_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 50 NAEP Technical Documentation Website NAEP Technical Documentation Quality Control on Weighting Procedures for the 2013 Assessment Given the complexity of the weighting procedures utilized in NAEP, a range of quality control (QC) checks was conducted throughout the weighting process to identify potential problems with collected student-level demographic data or with specific weighting procedures. The QC processes included Final Participation, Exclusion, and Accommodation Rates Nonresponse Bias Analyses checks performed within each step of the weighting process; checks performed across adjacent steps of the weighting process; review of participation, exclusion, and accommodation rates; checking demographic data of individual schools; comparisons with 2011 demographic data; and nonresponse bias analyses. To validate the weighting process, extensive tabulations of various school and student characteristics at different stages of the process were conducted. The school-level characteristics included in the tabulations were minority enrollment, median income (based on the school ZIP code area), and urban-centric locale. At the student level, the tabulations included race/ethnicity, gender, relative age, students with disability (SD) status, English language learners (ELL) status, and participation status in National School Lunch Program (NSLP). http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/quality_control_on_weighting_procedures_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 51 NAEP Technical Documentation Website NAEP Technical Documentation Final Participation, Exclusion, and Accommodation Rates for the 2013 Assessment Final participation, exclusion, and accommodation rates are presented in quality control tables for each grade and subject by geographic domain and school type. School-level participation rates have been calculated according to National Center for Education Statistics (NCES) standards as they have been for previous assessments. School-level participation rates were below 85 percent for private schools at all three grades (4, 8, and 12). Student-level participation rates were also below 85 percent for grade 12 public school student sample overall and in specific states: Connecticut, Florida, Illinois, Iowa, Massachusetts, New Hampshire, New Jersey, and West Virginia. As required by NCES standards, nonresponse bias analyses were conducted on each reporting group falling below the 85 percent participation threshold. Grade 4 Mathematics Grade 4 Reading Grade 8 Mathematics Grade 8 Reading Grade 12 Mathematics Grade 12 Reading http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/final_participation_exclusion_and_accommodation_rates_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 52 NAEP Technical Documentation Website NAEP Technical Documentation Participation, Exclusion, and Accommodation Rates for Grade 4 Mathematics for the 2013 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 4 mathematics assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 4 mathematics assessment, by school type and jurisdiction: 2013 School type and jurisdiction All National all1 Northeast all Midwest all South all West all National public Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Number of schools in original sample, rounded 8,760 8,590 School participation rates (percent) before substitution (weighted by base weight and enrollment) 97.30 97.27 School participation rates (percent) before substitution (weighted by base weight only) 90.45 90.32 Number of students sampled, rounded 214,900 209,800 1,480 2,190 2,740 2,120 8,060 95.63 97.27 98.20 96.86 99.69 85.22 88.80 93.44 91.04 99.54 120 200 120 120 300 120 120 100 140 100.00 99.48 100.00 100.00 99.17 100.00 97.22 100.00 100.00 240 170 120 130 200 120 140 150 160 130 160 100.00 100.00 100.00 100.00 97.98 100.00 100.00 100.00 100.00 100.00 100.00 Weighted percent of students excluded 1.40 1.41 Weighted student participation rates (percent) after makeups 94.57 94.57 Weighted percent of students accommodated 13.55 13.44 34,500 47,300 73,600 51,800 202,700 1.29 1.32 1.37 1.62 1.52 93.85 94.84 94.71 94.57 94.49 15.68 12.87 14.38 10.98 14.22 100.00 96.56 100.00 100.00 98.75 100.00 97.25 100.00 100.00 3,200 3,100 3,400 3,400 9,000 3,400 3,200 3,400 2,300 1.10 1.14 1.20 1.24 1.93 1.15 1.36 2.10 1.37 94.82 93.18 95.07 94.66 94.79 92.34 93.85 94.36 95.09 5.15 21.85 12.97 15.16 8.78 12.11 15.52 13.58 17.59 100.00 100.00 100.00 100.00 98.40 100.00 100.00 100.00 100.00 100.00 100.00 6,900 5,300 3,500 3,500 5,100 3,300 3,100 3,400 4,700 3,300 3,400 1.84 1.43 1.25 1.29 1.00 1.52 0.70 1.62 1.45 1.08 2.11 94.11 94.18 94.70 95.24 94.40 95.18 95.16 94.79 94.67 94.49 93.95 20.24 11.22 10.64 9.58 15.44 17.03 14.50 15.16 11.30 18.38 17.44 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity schools. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Mathematics Assessment. Appendices A-C NAEP 2019-2020 53 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 170 190 190 130 120 130 200 170 120 130 100.00 100.00 100.00 100.00 100.00 100.00 99.85 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 98.28 100.00 100.00 100.00 120 150 160 160 100.00 99.69 98.84 100.00 270 210 140 130 170 120 120 190 120 310 120 220 110 120 150 190 200 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 4,700 5,200 4,600 3,500 3,300 3,600 3,400 3,500 3,500 3,400 0.99 2.03 1.96 1.37 0.76 1.41 1.68 1.72 1.41 1.22 94.22 93.74 94.14 94.85 95.44 95.42 93.92 95.37 95.75 93.74 17.30 17.18 11.02 10.62 6.73 11.20 8.56 14.37 22.90 14.78 100.00 99.48 96.79 100.00 3,300 4,200 4,500 4,800 1.17 1.22 1.23 1.24 94.85 95.06 92.27 94.19 16.62 16.90 20.02 14.17 99.86 100.00 100.00 100.00 100.00 100.00 100.00 99.19 100.00 100.00 100.00 100.00 100.00 100.00 3,700 4,700 3,600 3,500 4,500 3,400 3,200 2.56 1.33 1.85 2.12 1.64 1.12 1.08 95.57 94.29 94.35 94.18 94.30 94.98 96.08 9.78 13.52 13.95 15.23 12.95 15.17 11.87 100.00 100.00 100.00 99.08 100.00 100.00 99.09 100.00 100.00 100.00 100.00 100.00 100.00 99.32 100.00 100.00 99.35 100.00 100.00 100.00 3,400 3,400 9,200 3,600 3,000 3,300 3,600 3,200 4,400 3,500 1.42 1.34 1.65 1.25 1.37 1.51 2.17 1.71 1.79 1.01 95.36 94.21 95.36 94.79 95.04 94.35 93.50 94.77 95.42 94.65 10.56 13.54 17.92 12.66 15.72 13.07 14.12 10.03 16.21 12.76 DoDEA2 120 99.23 98.08 Trial Urban (TUDA) Districts and Other Jurisdictions Albuquerque 50 100.00 100.00 Atlanta 60 100.00 100.00 Austin 60 100.00 100.00 Baltimore 70 100.00 100.00 City Boston 80 100.00 100.00 Charlotte 50 100.00 100.00 Chicago 100 100.00 100.00 Cleveland 90 100.00 100.00 Dallas 50 100.00 100.00 Detroit 70 100.00 100.00 Fresno 50 100.00 100.00 Hillsborough 60 100.00 100.00 Houston 80 100.00 100.00 3,700 1.66 95.05 12.20 1,700 2,000 1,700 1,600 1.15 0.98 2.04 1.59 94.71 95.42 93.69 94.32 20.47 9.76 30.80 19.27 2,000 1,700 2,500 1,500 1,700 1,300 1,800 1,700 2,600 3.69 1.19 1.07 4.26 2.33 4.88 0.90 1.17 1.88 93.72 94.18 94.85 93.62 95.79 90.92 93.58 95.74 96.62 19.59 12.81 19.30 22.29 35.42 14.80 7.51 23.30 27.25 School type and jurisdiction Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Wyoming 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity schools. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Mathematics Assessment. Appendices A-C NAEP 2019-2020 54 School type and jurisdiction Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 50 100.00 100.00 80 90 70 80 100.00 100.00 100.00 100.00 60 50 90 Jefferson County, KY Los Angeles Miami Milwaukee New York City Philadelphia San Diego District of Columbia (TUDA) National private Catholic Non-Catholic private Puerto Rico Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 1,700 1.74 94.66 11.61 100.00 100.00 100.00 100.00 2,500 2,300 1,500 2,500 1.96 2.35 3.40 1.33 95.80 95.07 94.68 91.74 9.83 28.05 26.55 27.56 100.00 100.00 100.00 100.00 100.00 100.00 1,600 1,500 1,500 3.45 1.48 1.97 94.71 95.18 95.52 15.82 11.80 18.06 410 71.19 64.52 3,300 0.08 95.61 4.38 130 280 88.65 56.94 89.70 52.97 1,700 1,600 0.06 0.11 95.60 95.62 4.95 3.92 170 100.00 100.00 5,100 0.24 94.47 27.19 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity schools. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Mathematics Assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/participation_exclusion_and_accommodation_rates_for_grade_4_mathematics_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 55 NAEP Technical Documentation Website NAEP Technical Documentation Participation, Exclusion, and Accommodation Rates for Grade 4 Reading for the 2013 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 4 reading assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 4 r jurisdiction: 2013 School type and jurisdiction All National all1 Northeast all Midwest all South all West all National public Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota eading assessment, by school type and Number of schools in original sample, rounded 8,590 8,590 School participation rates (percent) before substitution (weighted by base weight and enrollment) 97.27 97.27 School participation rates (percent) before substitution (weighted by base weight only) 90.32 90.32 Number of students sampled, rounded 216,400 216,400 1,480 2,190 2,740 2,120 8,060 95.63 97.27 98.20 96.86 99.69 85.22 88.80 93.44 91.04 99.54 120 200 120 120 300 120 120 100 140 100.00 99.48 100.00 100.00 99.17 100.00 97.22 100.00 100.00 240 170 120 130 200 120 140 150 160 130 160 170 190 190 130 100.00 100.00 100.00 100.00 97.98 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 Weighted percent of students excluded 2.52 2.52 Weighted student participation rates (percent) after makeups 94.78 94.78 Weighted percent of students accommodated 12.17 12.17 35,600 48,700 76,000 53,500 209,100 1.72 2.01 3.39 2.13 2.69 93.97 95.04 95.00 94.71 94.70 15.30 12.22 12.25 9.92 12.87 100.00 96.56 100.00 100.00 98.75 100.00 97.25 100.00 100.00 3,400 3,300 3,500 3,600 9,300 3,500 3,400 3,500 2,400 1.14 1.45 1.08 1.11 2.50 1.52 1.58 4.70 1.65 95.49 93.65 95.46 95.16 94.88 93.66 94.29 94.34 94.46 5.39 20.65 13.24 15.34 7.73 12.61 15.33 10.38 17.41 100.00 100.00 100.00 100.00 98.40 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 7,100 5,400 3,600 3,600 5,200 3,500 3,200 3,500 4,800 3,400 3,500 4,900 5,300 4,800 3,600 2.96 4.90 1.80 1.49 1.24 2.43 1.08 1.82 2.99 1.16 1.69 12.86 2.66 3.81 2.71 93.98 95.34 93.97 94.99 95.13 94.40 95.11 95.07 94.97 94.73 93.65 94.40 93.77 94.64 94.93 19.02 8.13 10.48 9.32 14.76 16.31 14.42 13.41 9.74 18.61 17.87 5.70 15.53 9.66 9.61 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity schools. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Reading Assessment. Appendices A-C NAEP 2019-2020 56 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 120 130 200 170 120 130 100.00 100.00 99.85 100.00 100.00 100.00 100.00 100.00 98.28 100.00 100.00 100.00 120 150 160 160 100.00 99.69 98.84 100.00 270 210 140 130 170 120 120 190 120 310 120 220 110 120 150 190 200 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 3,400 3,700 3,500 3,600 3,700 3,500 0.53 1.23 2.86 3.57 1.50 2.56 94.99 95.26 94.40 95.83 95.10 93.45 6.85 11.16 7.33 14.26 22.73 13.48 100.00 99.48 96.79 100.00 3,400 4,300 4,600 5,000 1.72 1.02 1.35 1.80 94.87 94.55 93.06 94.88 15.27 15.04 20.15 13.06 99.86 100.00 100.00 100.00 100.00 100.00 100.00 99.19 100.00 100.00 100.00 100.00 100.00 100.00 3,800 4,800 3,700 3,700 4,600 3,500 3,300 4.06 2.61 1.72 2.49 2.29 1.34 1.73 96.28 94.58 94.58 93.98 94.42 94.78 94.64 8.73 12.80 14.35 12.20 12.53 14.43 9.74 100.00 100.00 100.00 99.08 100.00 100.00 99.09 100.00 100.00 100.00 100.00 100.00 100.00 99.32 100.00 100.00 99.35 100.00 100.00 100.00 3,500 3,500 9,500 3,700 3,100 3,400 3,700 3,300 4,500 3,600 2.22 3.10 4.90 3.05 1.17 1.54 2.81 1.78 1.61 1.25 95.69 95.34 95.50 93.71 95.05 94.93 93.71 93.62 94.97 94.38 9.26 12.29 14.40 10.29 15.65 12.21 12.45 8.89 16.63 13.00 DoDEA2 120 99.23 98.08 Trial Urban (TUDA) Districts and Other Jurisdictions Albuquerque 50 100.00 100.00 Atlanta 60 100.00 100.00 Austin 60 100.00 100.00 Baltimore 70 100.00 100.00 City Boston 80 100.00 100.00 Charlotte 50 100.00 100.00 Chicago 100 100.00 100.00 Cleveland 90 100.00 100.00 Dallas 50 100.00 100.00 Detroit 70 100.00 100.00 Fresno 50 100.00 100.00 Hillsborough 60 100.00 100.00 Houston 80 100.00 100.00 Jefferson 50 100.00 100.00 County, KY Los Angeles 80 100.00 100.00 Miami 90 100.00 100.00 Milwaukee 70 100.00 100.00 New York 80 100.00 100.00 City 3,800 5.95 95.48 7.39 1,800 2,000 1,700 1,700 0.74 1.12 3.90 15.85 93.43 95.96 94.12 93.62 17.51 9.39 27.06 4.33 2,000 1,700 2,600 1,500 1,700 1,300 1,800 1,800 2,700 1,800 4.33 0.90 1.45 4.70 17.11 5.51 2.36 1.07 6.41 5.28 94.03 94.49 94.58 94.08 96.08 92.09 94.94 94.92 96.63 95.03 17.64 11.72 18.56 22.22 24.30 13.44 6.04 23.00 23.90 7.56 2,500 2,400 1,500 2,500 2.10 4.51 4.08 1.62 94.63 95.37 93.65 92.44 10.75 26.36 25.71 27.13 School type and jurisdiction Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Wyoming 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity schools. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Reading Assessment. Appendices A-C NAEP 2019-2020 57 School type and jurisdiction Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 60 50 90 100.00 100.00 100.00 100.00 100.00 100.00 410 71.19 130 280 88.65 56.94 Philadelphia San Diego District of Columbia (TUDA) National private Catholic Non-Catholic private Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 1,600 1,600 1,600 3.83 2.32 2.26 94.61 94.74 94.50 15.31 10.45 17.21 64.52 3,400 0.53 95.85 4.05 89.70 52.97 1,700 1,600 0.23 0.79 95.75 95.96 3.84 4.22 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity schools. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Reading Assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/participation_exclusion_and_accommodation_rates_for_grade_4_reading_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 58 NAEP Technical Documentation Website NAEP Technical Documentation Participation, Exclusion, and Accommodation Rates for Grade 8 Mathematics for the 2013 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 8 mathematics assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 8 mathematics assessment, by school type and jurisdiction: 2013 School type and jurisdiction All National all1 Northeast all Midwest all South all West all National public Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Number of schools in original sample, rounded 7,370 7,240 School participation rates (percent) before substitution (weighted by base weight and enrollment) 96.97 96.94 School participation rates (percent) before substitution (weighted by base weight only) 84.74 84.59 Number of students sampled, rounded 201,500 195,600 1,160 1,920 2,380 1,720 6,760 93.53 97.62 97.75 97.42 99.48 75.06 85.21 86.70 89.08 99.61 110 150 120 110 260 120 110 70 90 100.00 99.91 99.03 100.00 100.00 100.00 98.00 100.00 100.00 230 130 60 100 190 110 120 130 140 150 120 160 140 170 100.00 100.00 100.00 100.00 100.00 97.06 100.00 100.00 99.04 100.00 100.00 100.00 100.00 100.00 Weighted percent of students excluded 1.47 1.48 Weighted student participation rates (percent) after makeups 93.14 93.15 Weighted percent of students accommodated 11.88 11.79 32,700 44,100 68,800 48,000 189,400 1.60 1.42 1.51 1.41 1.59 92.00 93.69 93.24 93.28 93.02 15.85 11.78 11.59 9.25 12.25 100.00 98.79 99.16 100.00 100.00 100.00 97.87 100.00 100.00 3,000 3,000 3,200 3,200 8,400 3,100 3,100 3,200 2,100 1.04 1.08 1.30 1.93 1.49 1.12 2.05 1.31 0.96 94.23 91.72 93.42 95.00 93.59 93.47 92.44 90.65 91.26 5.14 18.75 10.71 13.92 7.91 11.50 13.92 14.90 20.71 100.00 100.00 100.00 100.00 100.00 96.65 100.00 100.00 99.21 100.00 100.00 100.00 100.00 100.00 6,400 4,800 3,200 3,100 4,800 3,000 3,100 3,300 4,300 3,200 2,900 4,400 4,800 4,200 1.70 1.55 1.67 1.06 1.01 1.64 0.77 1.67 2.08 1.06 1.33 1.74 2.01 2.46 91.06 93.38 90.26 94.15 94.48 92.49 93.74 93.94 94.54 94.14 92.79 92.08 91.98 92.93 15.32 9.82 12.28 8.42 13.83 13.95 13.28 11.23 10.09 14.26 15.99 13.33 16.11 10.55 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity schools. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Mathematics Assessment. Appendices A-C NAEP 2019-2020 59 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 130 110 130 150 130 90 90 98.99 100.00 100.00 99.80 100.00 100.00 100.00 99.67 100.00 100.00 98.82 100.00 100.00 100.00 110 120 160 140 100.00 99.68 93.08 100.00 190 200 130 130 160 60 110 150 110 230 120 120 110 120 110 170 100 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 2,900 3,200 3,100 3,200 3,100 3,300 3,200 1.70 0.80 1.28 1.44 1.85 1.04 1.06 91.58 93.80 94.25 92.28 93.41 92.80 91.60 9.16 6.51 10.57 9.20 12.02 11.91 15.99 100.00 99.02 95.81 100.00 3,100 4,000 4,300 4,500 1.64 1.57 1.90 1.29 92.26 93.07 91.15 92.95 16.38 12.00 19.38 13.74 99.92 100.00 100.00 100.00 100.00 100.00 100.00 99.44 100.00 100.00 100.00 100.00 100.00 100.00 3,700 4,500 3,100 3,100 4,300 3,200 3,200 2.93 1.51 1.63 1.47 1.70 1.11 1.33 94.98 93.07 92.97 92.91 92.17 93.93 94.19 11.44 13.54 14.09 10.88 14.66 15.92 9.86 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 3,200 3,200 8,800 3,300 3,000 3,200 3,100 3,200 4,300 3,300 1.30 1.77 1.92 1.53 0.83 1.05 2.03 1.69 1.51 1.50 94.44 92.81 93.82 92.07 93.91 93.39 90.87 92.62 94.25 93.66 8.66 9.81 12.13 10.15 15.36 12.18 11.47 9.02 14.73 12.51 DoDEA2 70 99.40 96.83 Trial Urban (TUDA) Districts and Other Jurisdictions Albuquerque 30 100.00 100.00 Atlanta 30 100.00 100.00 Austin 30 100.00 100.00 Baltimore 60 100.00 100.00 City Boston 40 100.00 100.00 Charlotte 40 100.00 100.00 Chicago 100 100.00 100.00 Cleveland 90 100.00 100.00 Dallas 40 100.00 100.00 Detroit 50 100.00 100.00 Fresno 20 100.00 100.00 Hillsborough 50 100.00 100.00 Houston 50 100.00 100.00 Jefferson 40 100.00 100.00 County, KY Los Angeles 70 100.00 100.00 Miami 80 100.00 100.00 Milwaukee 60 100.00 100.00 2,600 1.15 94.47 9.23 1,400 1,600 1,600 1,300 1.53 0.72 1.88 1.70 90.76 91.57 90.97 89.54 14.44 11.10 20.60 19.73 1,800 1,500 2,300 1,500 1,600 1,100 1,400 1,600 2,400 1,600 2.55 1.29 1.28 2.62 2.44 4.29 1.74 1.35 2.21 1.65 91.61 90.94 94.80 91.57 93.81 91.58 92.52 93.78 92.37 93.37 20.88 10.11 17.19 28.48 18.35 15.07 7.06 20.46 14.67 12.72 2,200 2,300 1,500 1.54 2.25 4.10 94.39 92.63 91.60 10.83 18.78 25.55 School type and jurisdiction Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Wyoming 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity schools. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Mathematics Assessment. Appendices A-C NAEP 2019-2020 60 School type and jurisdiction Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 90 99.00 97.58 50 30 40 100.00 100.00 100.00 400 New York City Philadelphia San Diego District of Columbia (TUDA) National private Catholic Non-Catholic private Puerto Rico Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 2,400 1.72 91.78 26.10 100.00 100.00 100.00 1,400 1,300 1,100 3.74 2.32 1.69 92.67 92.60 90.15 20.69 11.81 22.20 69.63 60.45 3,400 0.26 94.74 6.54 130 270 87.18 53.51 84.76 48.11 1,800 1,600 0.26 0.26 95.73 93.50 5.50 7.51 130 100.00 100.00 5,900 0.03 92.75 23.05 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity schools. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Mathematics Assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/participation_exclusion_and_accommodation_rates_for_grade_8_mathematics_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 61 NAEP Technical Documentation Website NAEP Technical Documentation Participation, Exclusion, and Accommodation Rates for Grade 8 Reading for the 2013 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 8 reading assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 8 r jurisdiction: 2013 School type and jurisdiction All National all1 Northeast all Midwest all South all West all National public Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota eading assessment, by school type and Number of schools in original sample, rounded 7,240 7,240 School participation rates (percent) before substitution (weighted by base weight and enrollment) 96.94 96.94 School participation rates (percent) before substitution (weighted by base weight only) 84.59 84.59 Number of students sampled, rounded 199,100 199,100 1,160 1,920 2,380 1,720 6,760 93.53 97.62 97.75 97.42 99.48 75.06 85.21 86.70 89.08 99.61 110 150 120 110 260 120 110 70 90 100.00 99.91 99.03 100.00 100.00 100.00 98.00 100.00 100.00 230 130 60 100 190 110 120 130 140 150 120 160 140 170 130 100.00 100.00 100.00 100.00 100.00 97.06 100.00 100.00 99.04 100.00 100.00 100.00 100.00 100.00 98.99 Weighted percent of students excluded 2.15 2.15 Weighted student participation rates (percent) after makeups 93.11 93.11 Weighted percent of students accommodated 10.76 10.76 33,300 45,100 69,900 48,900 192,900 1.55 1.93 2.60 2.08 2.32 91.80 93.48 93.39 93.21 92.93 15.53 11.08 9.99 8.32 11.16 100.00 98.79 99.16 100.00 100.00 100.00 97.87 100.00 100.00 3,100 3,100 3,300 3,200 8,500 3,200 3,100 3,200 2,100 1.14 1.40 1.47 1.96 2.52 1.15 2.13 3.49 1.82 94.26 91.91 93.67 93.21 93.42 93.46 91.38 91.59 91.33 4.83 18.39 9.67 13.36 6.74 10.89 13.88 12.23 19.57 100.00 100.00 100.00 100.00 100.00 96.65 100.00 100.00 99.21 100.00 100.00 100.00 100.00 100.00 99.67 6,500 4,900 3,300 3,200 4,900 3,100 3,100 3,300 4,300 3,300 3,000 4,400 4,900 4,300 3,000 1.86 3.80 1.93 1.61 1.44 1.90 1.27 1.72 3.28 1.24 1.55 9.41 2.15 3.53 2.33 91.72 93.67 90.58 93.64 93.76 93.12 93.44 93.42 93.93 93.78 92.34 93.77 91.82 93.66 91.30 15.15 8.18 12.33 7.76 12.94 13.75 12.16 11.72 8.47 14.15 15.16 5.45 15.04 9.68 8.43 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity schools. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Reading Assessment. Appendices A-C NAEP 2019-2020 62 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 110 130 150 130 90 90 100.00 100.00 99.80 100.00 100.00 100.00 100.00 100.00 98.82 100.00 100.00 100.00 110 120 160 140 100.00 99.68 93.08 100.00 190 200 130 130 160 60 110 150 110 230 120 120 110 120 110 170 100 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 3,200 3,100 3,200 3,200 3,400 3,200 0.70 1.02 2.29 2.99 1.00 2.93 93.72 92.55 91.61 92.32 92.19 91.46 6.55 10.62 7.51 10.14 10.91 14.28 100.00 99.02 95.81 100.00 3,200 4,000 4,400 4,600 2.64 1.70 0.96 1.72 92.01 93.39 90.46 92.51 14.78 10.00 20.03 12.29 99.92 100.00 100.00 100.00 100.00 100.00 100.00 99.44 100.00 100.00 100.00 100.00 100.00 100.00 3,800 4,600 3,200 3,200 4,300 3,300 3,200 4.30 2.22 1.39 1.45 1.78 1.37 1.88 94.07 93.08 93.43 92.62 91.94 92.96 94.03 9.52 13.08 12.42 11.30 14.51 15.18 7.48 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 3,300 3,200 8,900 3,400 3,100 3,300 3,200 3,200 4,400 3,400 2.95 3.13 3.51 3.05 0.92 1.40 2.46 1.82 1.61 1.14 95.01 93.54 93.78 93.00 92.93 92.97 91.22 93.10 94.11 93.15 6.02 7.75 10.05 8.36 15.08 10.56 9.78 7.60 14.45 12.27 DoDEA2 70 99.40 96.83 Trial Urban (TUDA) Districts and Other Jurisdictions Albuquerque 30 100.00 100.00 Atlanta 30 100.00 100.00 Austin 30 100.00 100.00 Baltimore 60 100.00 100.00 City Boston 40 100.00 100.00 Charlotte 40 100.00 100.00 Chicago 100 100.00 100.00 Cleveland 90 100.00 100.00 Dallas 40 100.00 100.00 Detroit 50 100.00 100.00 Fresno 20 100.00 100.00 Hillsborough 50 100.00 100.00 Houston 50 100.00 100.00 Jefferson 40 100.00 100.00 County, KY Los Angeles 70 100.00 100.00 Miami 80 100.00 100.00 Milwaukee 60 100.00 100.00 New York 90 99.00 97.58 City 2,600 3.84 94.13 7.11 1,400 1,700 1,600 1,300 2.04 1.02 3.35 16.39 93.46 92.20 88.54 89.73 11.79 10.98 18.36 5.14 1,800 1,500 2,300 1,500 1,600 1,100 1,500 1,600 2,400 1,600 3.41 1.68 1.60 3.52 3.51 5.74 3.10 1.94 3.80 4.30 93.05 92.20 94.72 91.90 93.98 91.37 93.27 91.85 93.58 94.71 18.94 9.90 16.76 27.75 15.20 12.53 5.86 19.74 12.29 9.49 2,300 2,400 1,500 2,400 2.70 2.88 4.06 1.46 94.30 94.21 93.15 91.17 9.97 18.45 25.08 26.00 School type and jurisdiction Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Wyoming 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity schools. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Reading Assessment. Appendices A-C NAEP 2019-2020 63 School type and jurisdiction Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 50 30 40 100.00 100.00 100.00 100.00 100.00 100.00 400 69.63 130 270 87.18 53.51 Philadelphia San Diego District of Columbia (TUDA) National private Catholic Non-Catholic private Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 1,400 1,300 1,100 3.79 2.58 2.53 91.35 93.78 90.18 20.91 10.58 22.13 60.45 3,500 0.30 95.45 6.32 84.76 48.11 1,900 1,600 0.21 0.39 96.07 94.67 4.96 7.56 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity schools. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Reading Assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/participation_exclusion_and_accommodation_rates_for_grade_8_reading_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 64 NAEP Technical Documentation Website NAEP Technical Documentation Participation, Exclusion, and Accommodation Rates for Grade 12 Mathematics for the 2013 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 12 mathematics assessment. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 12 mathematics assessment, by school type and geographic r School type and geographic region All National all1 Northeast all Midwest all South all West all National public Arkansas Connecticut Florida Idaho Illinois Iowa Massachusetts Michigan New Hampshire New Jersey South Dakota Tennessee West Virginia Remaining jurisdictions2 National private Catholic Non-Catholic private Number of schools in original sample 2,200 2,200 School participation rates (percent) before substitution (weighted by base weight and enrollment) 89.51 89.51 School participation rates (percent) before substitution (weighted by base weight only) 82.66 82.66 Number of students sampled 62,200 62,200 510 650 710 330 2,030 100 110 120 100 130 120 110 140 80 110 140 130 90 89.05 87.14 89.42 92.21 92.95 100.00 98.93 99.05 100.00 90.38 100.00 99.04 100.00 100.00 98.14 99.74 100.00 100.00 81.63 83.20 85.99 77.24 93.31 100.00 99.45 99.30 100.00 93.98 100.00 99.45 100.00 100.00 98.57 99.07 100.00 100.00 570 160 40 120 91.16 53.34 68.06 38.52 90.91 55.43 79.95 50.25 egion: 2013 Weighted percentage of students excluded 2.16 2.16 Weighted student participation rates (percent) after makeups 84.33 84.33 Weighted percentage of students accommodated 8.65 8.65 16,200 16,600 20,300 9,100 60,400 2,900 3,200 3,300 3,000 3,300 3,300 3,200 4,000 4,100 3,300 3,100 4,100 3,300 2.29 1.65 2.31 2.32 2.31 2.78 1.76 3.21 1.65 1.85 1.13 2.21 1.90 1.61 1.89 1.51 2.51 2.00 81.79 83.87 86.52 83.37 84.17 92.09 81.22 77.25 89.17 85.16 83.05 81.71 86.94 76.64 84.10 87.48 88.15 83.68 11.95 8.61 7.98 7.15 8.77 8.61 8.71 12.67 6.72 9.79 10.78 11.13 8.78 11.22 14.28 5.78 7.84 7.01 16,200 1,800 1,000 800 2.26 0.63 0.83 0.42 84.41 86.51 85.53 87.96 10.55 7.32 5.46 9.28 1 Includes national public, national private, Bureau of Indian Education, and Department of Defense Education Activity schools located in the United States. 2 Includes national public schools not part of the state assessment. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Mathematics Assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/participation_exclusion_and_accommodation_rates_for_grade_12_mathematics_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 65 NAEP Technical Documentation Website NAEP Technical DocumentationParticipation, Exclusion, and Accommodation Rates for Grade 12 Reading for the 2013 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 12 reading assessment. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 12 r School type and geographic region All National all1 Northeast all Midwest all South all West all National public Arkansas Connecticut Florida Idaho Illinois Iowa Massachusetts Michigan New Hampshire New Jersey South Dakota Tennessee West Virginia Remaining jurisdictions2 National private Catholic Non-Catholic eading assessment, by school type and geographic r Number of schools in original sample 2,200 2,200 School participation rates (percent) before substitution (weighted by base weight and enrollment) 89.51 89.51 School participation rates (percent) before substitution (weighted by base weight only) 82.66 82.66 Number of students sampled 62,300 62,300 510 650 710 330 2,030 100 110 120 100 130 120 110 140 80 110 140 130 90 89.05 87.14 89.42 92.21 92.95 100.00 98.93 99.05 100.00 90.38 100.00 99.04 100.00 100.00 98.14 99.74 100.00 100.00 81.63 83.20 85.99 77.24 93.31 100.00 99.45 99.30 100.00 93.98 100.00 99.45 100.00 100.00 98.57 99.07 100.00 100.00 570 160 40 120 91.16 53.34 68.06 38.52 90.91 55.43 79.95 50.25 egion: 2013 Weighted percentage of students excluded 2.41 2.41 Weighted student participation rates (percent) after makeups 83.89 83.89 Weighted percentage of students accommodated 8.55 8.55 16,500 16,700 20,000 9,000 60,400 3,000 3,400 3,300 3,200 3,400 3,500 3,200 3,900 4,300 3,300 3,300 3,900 3,400 2.16 2.05 2.87 2.24 2.56 2.56 2.34 3.55 1.66 2.29 1.51 1.87 4.01 2.55 1.80 1.60 2.88 2.37 80.91 84.05 85.51 83.58 83.77 90.21 79.77 77.34 88.68 83.72 84.26 79.84 87.21 76.91 84.67 86.17 88.82 84.28 12.89 8.75 7.18 7.14 8.73 8.24 8.70 12.14 6.42 9.92 10.62 11.31 6.17 10.25 14.78 5.16 7.13 6.89 15,200 1,900 1,100 800 2.77 0.84 0.92 0.75 83.98 85.52 84.67 86.75 10.05 6.67 4.01 9.41 1 Includes national public, national private, Bureau of Indian Education, and Department of Defense Education Activity schools located in the United States. 2 Includes national public schools not part of the state assessment. NOTE: Numbers of schools are rounded to nearest ten, and numbers of students are rounded to nearest hundred. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Reading Assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/participation_exclusion_and_accommodation_rates_for_grade_12_reading_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 66 NAEP Technical Documentation Website NAEP Technical Documentation Nonresponse Bias Analyses for the 2013 Assessment NCES statistical standards call for a nonresponse bias analysis to be conducted for a sample with a response rate below 85 percent at any stage of sampling. Weighted school response rates for the 2013 assessment indicated a need for school nonresponse bias analyses for private school samples in grades 4, 8, and 12 (operational subjects). Student nonresponse bias analyses were necessary for the grade 12 public school student sample overall and in specific states, for both reading and mathematics: Connecticut, Florida, Iowa, Massachusetts, New Hampshire, and West Virginia. Additionally, a student nonresponse bias analysis was required for the grade 12 public school student sample in Illinois based on the weighted response rate for reading, while such an analysis was required for grade 12 public school student sample in New Jersey based on the weighted response rate for mathematics. Thus, three separate school-level analyses and nine separate student-level analyses were conducted. The procedures and results from these analyses are summarized briefly below. The analyses conducted consider only certain characteristics of schools and students. They do not directly consider the effects of the nonresponse on student achievement, the primary focus of NAEP. Thus, these analyses cannot be conclusive of either the existence or absence of nonresponse bias for student achievement. For more details, please see the NAEP 2013 NRBA report (657.56 KB). Each school-level analysis was conducted in three parts. The first part of the analysis looked for potential nonresponse bias that was introduced through school nonresponse. The second part of the analysis examined the remaining potential for nonresponse bias after accounting for the mitigating effects of substitution. The third part of the analysis examined the remaining potential for nonresponse bias after accounting for the mitigating effects of both school substitution and school-level nonresponse weight adjustments. The characteristics examined were Census region, reporting subgroup (private school type), urban-centric locale, size of school (categorical), and race/ethnicity percentages (mean). Based on the school characteristics available, for the private school samples at grade 4, there does not appear to be evidence of substantial potential bias resulting from school substitution or school nonresponse. However, the analyses suggest that a potential for nonresponse bias remains for the grade 8 and 12 private school samples. For grade 8, this result is evidently related to the fact that, among nonCatholic schools, larger schools were less likely to respond. Thus, when making adjustments to address the underrepresentation of non-Catholic schools among the respondents, the result is to over represent smaller schools at the expense of larger ones. The limited school sample sizes involved means that it is not possible to make adjustments that account fully for all school characteristics. For grade 12, the analyses suggested potential bias for percentage Asian and percentage Two or more races. Please see the full report for more details. Each student-level analysis was conducted in two parts. The first part of the analysis examined the potential for nonresponse bias that was introduced through student nonresponse. The second part of the analysis examined the potential for bias after accounting for the effects of nonresponse weight adjustments. The characteristics examined were gender, race/ethnicity, relative age, National School Lunch Program eligibility, student disability (SD) status, and English language learner (ELL) status. Based on the student characteristics available, there does not appear to be evidence of substantial potential bias resulting from student nonresponse. Please see the full report for more details. http://nces.ed.gov/nationsreportcard/tdw/weighting/2013/nonresponse_bias_analyses_for_the_2013_assessment.aspx Appendices A-C NAEP 2019-2020 67 NATIONAL CENTER FOR EDUCATION STATISTICS NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS National Assessment of Education Progress (NAEP) 2019 and 2020 Appendix B2 NAEP 2012 Long Term Trend (LTT) Weighting Procedures Design OMB# 1850-0928 v.15 February 2019 Appendices A-C NAEP 2019-2020 68 NAEP Technical Documentation Website NAEP Technical Documentation Weighting Procedures for the 2012 Long-Term Trend (LTT) Assessment NAEP assessments use complex sample designs to Computation of Full-Sample Weights create student samples that generate population and subpopulation estimates with reasonably high Computation of Replicate Weights for precision. Student sampling weights ensure valid Variance Estimation inferences from the student samples to their respective populations. In the 2012 long term trend Quality Control on Weighting (LTT) assessments, weights were developed for Procedures students sampled at ages 9, 13, and 17 for assessments in mathematics and reading. Each student was assigned a weight to be used for making inferences about students in the target population. This weight is known as the final full-sample student weight, and it contains five major components: the student base weight, school nonresponse adjustments, student nonresponse adjustments, school weight trimming adjustments, and student weight trimming adjustments. The student base weight is the inverse of the overall probability of selecting a student and assigning that student to a particular assessment. The sample design that determines the base weights is discussed in the NAEP 2012 LTT sample design section. The base weight is adjusted for two sources of nonparticipation: school level and student level. These weighting adjustments seek to reduce the potential for bias from such nonparticipation by increasing the weights of students from schools similar to those schools not participating, and increasing the weights of participating students similar to those students from within participating schools who did not attend the assessment session (or makeup session) as scheduled. Furthermore, the final weights reflect the trimming of extremely large weights at both the school and student level. These weighting adjustments seek to reduce variances of survey estimates. Appendices A-C NAEP 2019-2020 69 In addition to the final full-sample weight, a set of replicate weights was provided for each student. These replicate weights are used to calculate the variances of survey estimates using the jackknife repeated replication method. The methods used to derive these weights were aimed at reflecting the features of the sample design, so that when the jackknife variance estimation procedure is implemented, approximate unbiased estimates of sampling variance are obtained. In addition, the various weighting procedures were repeated on each set of replicate weights to appropriately reflect the impact of the weighting adjustments on the sampling variance of a survey estimate. Quality control checks were implemented throughout the weighting process to ensure the accuracy of the full-sample and replicate weights. See Quality Control for Weighting Procedures for the various checks implemented and main findings of interest. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt.aspx NAEP Technical Documentation Website NAEP Technical Documentation Computation of Full-Sample Weights for the 2012 LTT Assessment The full-sample or final student weight is the sampling weight used to derive NAEP Computation of Base Weights student estimates of population and subpopulation characteristics for a School and Student Nonresponse specified age (9, 13, or 17) and assessment Weight Adjustments subject (mathematics or reading). The fullsample student weight reflects the number of School and Student Weight students that the sampled student represents Trimming Adjustments in the population for purposes of estimation. The summation of the final student weights over a particular student group provides an estimate of the total number of students in that group within the population. The full-sample weight, which is used to produce survey estimates, is distinct from a replicate weight that is used to estimate variances of survey estimates. The full- Appendices A-C NAEP 2019-2020 70 sample weight is assigned to participating students and reflects the student base weight after the application of the various weighting adjustments. The full-sample weight for student k from school s in stratum j (FSTUWGTjsk) can be expressed as follows: where STU_BWTjsk is the student base weight; SCH_NRAFjs is the school-level nonresponse adjustment factor; STU_NRAFjsk is the student-level nonresponse adjustment factor; SCH_TRIMjs is the school-level weight trimming adjustment factor; and STU_TRIMjsk is the student-level weight trimming adjustment factor. School sampling strata for a given assessment varied by school type. See public school strata and private school strata for descriptions of the public and private school stratum definitions. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_comp_full_samp.aspx NAEP Technical Documentation Website NAEP Technical Documentation Computation of Base Weights for the 2012 LTT Assessment Every sampled school and student received a base weight equal to the reciprocal of its probability of selection. Computation of a school base weight varies by Appendices A-C NAEP 2019-2020 School Base Weights Student Base Weights 71 the type of sampled school (original or substitute); and the sampling frame (new school frame or not). Computation of a student base weight reflects the student's overall probability of selection accounting for school and student sampling; assignment to session type at the school- and student-level; and the student's assignment to the mathematics or reading assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_base.aspx Appendices A-C NAEP 2019-2020 72 NAEP Technical Documentation Website NAEP Technical Documentation School Base Weights for the 2012 LTT Assessment The school base weight for a sampled school is equal to the inverse of its overall probability of selection. The overall selection probability of a sampled school differs by type of sampled school (original or substitute); and sampling frame (new school frame or not). The overall probability of selection of an originally selected school reflects two components: Substitute public schools for the 2012 LTT assessments Substitute private schools for the 2012 LTT assessments the probability of selection of the primary sampling unit (PSU), and the probability of selection of the school within the selected PSU from either the NAEP public school frame or the private school frame. The overall selection probability of a school from the new school frame is the product of two quantities: the probability of selection of the school's district into the new-school district sample, and the probability of selection of the school into the new school sample. Substitute schools are preassigned to original schools and take their place if the original schools refuse to participate. For weighting purposes, they are treated as if they were the original schools that they replaced and are assigned the school base weight of the original schools. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_base_wghts_school.aspx NAEP Technical Documentation Website Appendices A-C NAEP 2019-2020 73 NAEP Technical Documentation Substitute Public Schools for the 2012 Long-Term Trend (LTT) Assessment Substitute schools were preselected for the public school samples by sorting the school frame file according to the actual order used in the sampling process (the implicit stratification). For operational reasons, the original selection order was embedded within the sampled primary sampling unit (PSU) and state. Each sampled school had each of its nearest neighbors within the same sampling stratum on the school frame file identified as a potential substitute. When age-eligible enrollment was used as the last sort ordering variable, the nearest neighbors had age enrollment values very close to that of the sampled school. This was done to facilitate the selection of about the same number of students within the substitute as would have been selected from the original sampled school. Schools were disqualified as potential substitutes if they were already selected in any of the original public school samples or assigned as a substitute for another public school (earlier in the sort ordering). Schools assigned as substitutes for age 17 schools were disqualified as potential substitutes for age 9 and 13 schools, and schools assigned as substitutes for age 13 schools were disqualified as potential substitutes for age 9 schools. If both nearest neighbors were still eligible to be substitutes, the one with a closer age-eligible enrollment was chosen. If both nearest neighbors were equally distant from the sampled school in their age enrollment (an uncommon occurrence), one of the two was randomly selected. Of the approximately 1,100 original sampled public schools for the ages 9, 13, and 17 assessments, about 30 schools had a substitute activated because the original eligible school did not participate. Ultimately, about 20 of the activated substitute public schools participated in an assessment. http://nces.ed.gov/nationsreportcard/tdw/sample_design/2012/2012_ltt_samp_pub_subs.aspx Appendices A-C NAEP 2019-2020 74 NAEP Technical Documentation Website NAEP Technical Documentation Substitute Private Schools for the 2012 Long-Term Trend (LTT) Assessment Substitutes were preselected for the private school samples by sorting the school frame file according to the actual order used in the sampling process (the implicit stratification). For operational reasons, the original selection order was embedded within the sampled primary sampling unit (PSU) and state. Each sampled school had each of its nearest neighbors within the same sampling stratum on the school frame file identified as a potential substitute. Since agespecific enrollment was used as the last sort ordering variable, the nearest neighbors had agespecific enrollment values very close to that of the sampled school. This was done to facilitate the selection of about the same number of students within the substitute as would have been selected from the original sampled school. Schools were disqualified as potential substitutes if they were already selected in any of the original private school samples or assigned as a substitute for another private school (earlier in the sort ordering). Schools assigned as substitutes for age seventeen schools were disqualified as potential substitutes for age nine and age thirteen schools, and schools assigned as substitutes for age thirteen schools were disqualified as potential substitutes for age nine schools. If both nearest neighbors were still eligible to be substitutes, the one with a closer age-specific enrollment was chosen. If both nearest neighbors were equally distant from the sampled school in their age-specific enrollment (an uncommon occurrence), one of the two was randomly selected. Of the 360 original sampled private schools for the long-term trend (LTT) assessment, 107 schools had substitutes activated when the original eligible schools did not participate. Ultimately, 43 of the activated substitute private schools participated. http://nces.ed.gov/nationsreportcard/tdw/sample_design/2012/2012_ltt_samp_priv_subs.aspx NAEP Technical Documentation Website Appendices A-C NAEP 2019-2020 75 NAEP Technical Documentation Student Base Weights for the 2012 LTT Assessment Every sampled student received a student base weight, whether or not the student participated in the assessment. The student base weight is the reciprocal of the probability that the student was sampled to participate in the assessment for a specified subject. The student base weight for student k from school s in stratum j (STU_BWTjsk) is the product of seven weighting components and can be expressed as follows: where SCH_BWTjs is the school base weight; SCHSESWTjs is the school-level session assignment weight that reflects the conditional probability, given the school, that the particular session type was assigned to the school; WINSCHWTjs is the within-school student weight that reflects the conditional probability, given the school, that the student was selected for the NAEP assessment; STUSESWTjsk is the student-level session assignment weight that reflects the conditional probability, given the particular session type was assigned to the school, that the student was assigned to that session type; SUBJFACjsk is the subject spiral adjustment factor that reflects the conditional probability, given the student was assigned to a particular session type, that the student was assigned the specified subject; SUBADJjs is the substitution adjustment factor to account for the difference in enrollment size between the substitute and original school; and YRRND_AFjs is the year-round adjustment factor to account for students in yearround schools on scheduled break at the time of the NAEP assessment and thus not available for sample. The within-school student weight (WINSCHWTjs) is the inverse of the student sampling rate in the school. The subject spiral adjustment factor (SUBJFACjsk) adjusts the student weight to account for the spiral pattern used in distributing mathematics or reading booklets to the students. The subject factor varies by sample age, subject, and school type (public/private). It is equal to the inverse of the booklet proportions (mathematics or reading) in the overall spiral for a specific sample. For cooperating substitutes of nonresponding sampled original schools, the substitution adjustment factor (SUBADJjs) is equal to the ratio of the estimated age-specific enrollment for the originally sampled school to the estimated age-specific enrollment for the substitute school. The student sample from the substitute school then "represents" the set of age-eligible students from the originally sampled school. Appendices A-C NAEP 2019-2020 76 The year-round adjustment factor (YRRND_AFjs) adjusts the student weight for students in yearround schools who do not attend school during the time of the assessment. This situation typically arises in overcrowded schools. School administrators in year-round schools randomly assign students to portions of the year in which they attend school and portions of the year in which they do not attend. At the time of assessment, a certain percentage of students (designated as OFFjs) do not attend school and thus cannot be assessed. The YRRND_AFjs for a school is calculated as 1/(1-OFFjs/100). http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_base_stud.aspx NAEP Technical Documentation Website NAEP Technical Documentation School and Student Nonresponse Weight Adjustments for the 2012 LTT Assessment Nonresponse is unavoidable in any voluntary survey of a School Nonresponse Weight human population. Nonresponse leads to the loss of sample Adjustment data that must be compensated for in the weights of the responding sample members. This differs from ineligibility, Student Nonresponse Weight for which no adjustments are necessary. The purpose of the Adjustment nonresponse adjustments is to reduce the mean square error of survey estimates. While the nonresponse adjustment reduces the bias from the loss of sample, it also increases variability among the survey weights leading to increased variances. However, it is presumed that the reduction in bias more than compensates for the increase in the variance, thereby reducing the mean square error and thus improving the accuracy of survey estimates. Nonresponse adjustments are made in the NAEP surveys at both the school and the student levels: the responding (original and substitute) schools receive a weighting adjustment to compensate for nonresponding schools, and responding students receive a weighting adjustment to compensate for nonresponding students. The paradigm used for nonresponse adjustment in NAEP is the quasi-randomization approach (Oh and Scheuren 1983). In this approach, school response cells are based on characteristics of schools known to be related to both response propensity and achievement level, such as the locale type (e.g., large principal city of a metropolitan area) of the school. Likewise, student response cells are based on characteristics of the schools containing the students and student characteristics, which are known to be related to both response propensity and achievement level, such as student race/ethnicity, gender, and age. Appendices A-C NAEP 2019-2020 77 Under this approach, sample members are assigned to mutually exclusive and exhaustive response cells based on predetermined characteristics. A nonresponse adjustment factor is calculated for each cell as the ratio of the sum of adjusted base weights for all eligible units to the sum of adjusted base weights for all responding units. The nonresponse adjustment factor is then applied to the adjusted base weight of each responding unit. In this way, the weights of responding units in the cell are "weighted up" to represent the full set of responding and nonresponding units in the response cell. The quasi-randomization paradigm views nonresponse as another stage of sampling. Within each nonresponse cell, the paradigm assumes that the responding sample units are a simple random sample from the total set of all sample units. If this model is valid, then the use of the quasirandomization weighting adjustment will eliminate any nonresponse bias. Even if this model is not valid, the weighting adjustments will eliminate bias if the achievement scores are homogeneous within the response cells (i.e., bias is eliminated if there is homogeneity either in response propensity or in achievement levels). See, for example, chapter 4 of Little and Rubin (1987). http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_nonresp.aspx NAEP Technical Documentation Website NAEP Technical Documentation School Nonresponse Weight Adjustments for the 2012 LTT Assessment The school nonresponse adjustment procedure inflates the weights of participating Development of Initial School Nonresponse Cells schools to account for eligible nonparticipating schools for which no Development of Final School Nonresponse substitute schools participated. The Cells adjustments are computed within nonresponse cells and are based on the assumption that the participating and School Nonresponse Adjustment Factor nonparticipating schools within the same cell Calculation are more similar to one another than to schools from different cells. Exactly how nonresponse cells were defined varied for public and private schools. Appendices A-C NAEP 2019-2020 78 http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_nonresp_schl.aspx NAEP Technical Documentation Website NAEP Technical Documentation Development of Initial School Nonresponse Cells for the 2012 LTT Assessment The cells for nonresponse adjustments are generally functions of the school sampling strata for the individual samples. For NAEP 2012 LTT, school sampling strata were the same for each age and subject sample, but differed by school type (public or private). Assessment subjects that are administered together by way of spiraling have the same school samples and stratification schemes. Subjects that are not spiraled with any other subjects have their own separate school sample. In NAEP 2012 LTT, the mathematics and reading assessments were spiraled together. The description of the initial nonresponse cells for the NAEP 2012 LTT samples is given below. Public School Samples For public school samples, initial weighting cells were formed within each age sample using the following nesting cell structure: census region, collapsed urbanicity (collapsed urban-centric locale) stratum, and race/ethnicity classification. Private School Samples For private school samples, initial weighting cells were formed within each age sample using the following nesting cell structure: affiliation (Catholic or non-Catholic), census region, and collapsed urbanicity (collapsed urban-centric locale) stratum. Appendices A-C NAEP 2019-2020 79 http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_nonresp_schl_initial.aspx NAEP Technical Documentation Website NAEP Technical Documentation Development of Final School Nonresponse Cells for the 2012 LTT Assessment Limits were placed on the magnitude of cell sizes and adjustment factors to prevent unstable nonresponse adjustments and unacceptably large nonresponse factors. All initial weighting cells with fewer than six cooperating schools or adjustment factors greater than 3.0 for the full sample weight were collapsed with suitable adjacent cells. Simultaneously, all initial weighting cells for any replicate with fewer than four cooperating schools or adjustment factors greater than the maximum of 3.0 (or two times the full sample nonresponse adjustment factor) were collapsed with suitable adjacent cells. Initial weighting cells were generally collapsed in reverse order of the cell structure; that is, starting at the bottom of the nesting structure and working up toward the top level of the nesting structure. Public School Samples For the public school samples, race/ethnicity classification cells within a collapsed urbanicity (collapsed urban-centric locale) stratum and census region were collapsed first. If further collapsing was required after all levels of race/ethnicity cells were collapsed, collapsedurbanicity strata within census region were combined next. Cells were never collapsed across census region. Private School Samples For the private school samples, collapsed-urbanicity strata within a census region and affiliation type were collapsed first. If further collapsing was required, census region cells within an affiliation type were collapsed. Cells were never collapsed across affiliation. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_nonresp_schl_final.aspx Appendices A-C NAEP 2019-2020 80 NAEP Technical Documentation Website NAEP Technical Documentation School Nonresponse Adjustment Factor Calculation for the 2012 LTT Assessment In each final school nonresponse adjustment cell c, the school nonresponse adjustment factor SCH_NRAFc was computed as follows: where Sc is the set of all eligible sampled schools (cooperating original and substitute schools and refusing original schools with noncooperating or no assigned substitute) in cell c, Rc is the set of all cooperating schools within Sc, SCH_BWTs is the school base weight, SCH_TRIMs is the school-level weight trimming factor, SCHSESWTs is the school-level session assignment weight, and Xs is the estimated age-specific enrollment corresponding to the original sampled school. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_nonresp_schl_factor.aspx NAEP Technical Documentation Website Appendices A-C NAEP 2019-2020 81 NAEP Technical Documentation Website NAEP Technical DocumentationStudent Nonresponse Adjustment Factor Calculation for the 2012 LTT Assessment In each final student nonresponse adjustment cell c for a given sample, the student nonresponse adjustment factor STU_NRAF c was computed as follows: where Sc is the set of all eligible sampled students in cell c for a given sample, Rc is the set of all assessed students within Sc, STU_BWT k is the student base weight for a given student k, SCH_TRIMk is the school-level weight trimming factor for the school associated with student k, SCH_NRAF k is the school-level nonresponse adjustment factor for the school associated with student k, and SUBJFACk is the subject factor for a given student k. The student weight used in the calculation above is the adjusted student base weight, without regard to subject, adjusted for school weight trimming and school nonresponse. Nonresponse adjustment procedures are not applied to excluded students because they are not required to complete an assessment. In effect, excluded students were placed in a separate nonresponse cell by themselves and all received an adjustment factor of 1. While excluded students are not included in the analysis of the NAEP scores, weights are provided for excluded students in order to estimate the size of this group and its population characteristics. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_nonresp_stud_factor.aspx Appendices A-C NAEP 2019-2020 82 NAEP Technical Documentation School and Student Weight Trimming Adjustments for the 2012 LTT Assessment Weight trimming is an adjustment procedure that involves detecting Trimming of School and reducing extremely large weights. "Extremely large weights" Base Weights generally refer to large sampling weights that were not anticipated in the design of the sample. Unusually large weights are likely to Trimming of Student produce large sampling variances for statistics of interest, especially Weights when the large weights are associated with sample cases reflective of rare or atypical characteristics. To reduce the impact of these large weights on variances, weight reduction methods are typically employed. The goal of weight reduction methods is to reduce the mean square error of survey estimates. While the trimming of large weights reduces variances, it also introduces some bias. However, it is presumed that the reduction in the variances more than compensates for the increase in the bias, thereby reducing the mean square error and thus improving the accuracy of survey estimates (Potter 1988). NAEP employs weight trimming at both the school and student levels. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_trimming_adjustments.aspx NAEP Technical Documentation Website NAEP Technical Documentation Trimming of School Base Weights for the 2012 LTT Assessment Large school weights can occur for schools selected from the NAEP new-school sampling frame and for private schools. New schools that are eligible for weight trimming are schools with a disproportionately large student enrollment in a particular grade from a school district that was selected with a low probability of selection. The school base weights for such schools may be large relative to what they would have been if they had been selected as part of the original sample. To detect extremely large weights among new schools, a comparison was made between a new school's school base weight and its ideal weight (i.e., the weight that would have resulted had the Appendices A-C NAEP 2019-2020 83 school been selected from the original school sampling frame). If the school base weight was more than three times the ideal weight, a trimming factor was calculated for that school that scaled the base weight back to three times the ideal weight. The calculation of the school-level trimming factor for a new school s is expressed in the following formula: where EXP_WTs is the ideal base weight the school would have received if it had been on the NAEP public school sampling frame, and SCH_BWTs is the actual school base weight the school received as a sampled school from the new school frame. No new schools in any of the NAEP 2012 LLT samples had their weights trimmed. Private schools eligible for weight trimming were Private School Universe Survey (PSS) nonrespondents who were found subsequently to have either larger enrollments than assumed at the time of sampling, or an atypical probability of selection given their affiliation, the latter being unknown at the time of sampling. For private school s, the formula for computing the schoollevel weight trimming factor SCH_TRIMs is identical to that used for new schools. For private schools, EXP_WTs is the ideal base weight the school would have received if it had been on the NAEP private school sampling frame with accurate enrollment and known affiliation, and SCH_BWTs is the actual school base weight the school received as a sampled private school. No private schools in any of the NAEP 2012 LTT samples had their weights trimmed. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_base_schtrim.aspx NAEP Technical Documentation Website Appendices A-C NAEP 2019-2020 84 NAEP Technical Documentation Trimming of Student Weights for the 2012 LTT Assessment Large student weights generally come from compounding nonresponse adjustments at the school and student levels with artificially low first-stage selection probabilities, which can result from inaccurate enrollment data on the school frame used to define the school size measure. Even though measures are in place to limit the number and size of excessively large weights—such as the implementation of adjustment factor size constraints in both the school and student nonresponse procedures and the use of the school trimming procedure—large student weights can still occur. The student weight trimming procedure uses a multiple median rule to detect excessively large student weights. Any student weight within a given trimming group greater than a specified multiple of the median weight value of the given trimming group has its weight scaled back to that threshold. Trimming groups were defined by age, subject, region, and Black/Hispanic strata (age 17 only) for public schools, and affiliation (Catholic/non-Catholic) for private schools. The procedure computes the median of the nonresponse-adjusted student weights in the trimming group g for a given grade and subject sample. Any student k with a weight more than M times the median (where M = 3.5 for public and private schools) received a trimming factor calculated as follows: where M is the trimming multiple, MEDIANg is the median of nonresponse-adjusted student weights in trimming group g,and STUWGTgk is the weight after student nonresponse adjustment for student k in trimming group g. In the NAEP 2012 LTT assessments, relatively few students had weights considered excessively large. Out of the approximately 53,500 students included in the combined 2012 LTT assessment samples, only 22 students had their weights trimmed. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_studtrim.aspx Appendices A-C NAEP 2019-2020 85 NAEP Technical Documentation Website NAEP Technical Documentation Computation of Replicate Weights for Variance Estimation for the 2012 LTT Assessment In addition to the full-sample weight, a Defining Replicate Strata and Forming set of 62 replicate weights was provided Replicates for each student. These replicate weights are used in calculating Computing School-Level Replicate Base the sampling variance of estimates Weights obtained from the data, using the jackknife repeated replication Computing Student-Level Replicate Base method. The method of deriving these Weights weights was aimed at reflecting the Replicate Variance Estimation features of the sample design appropriately for each sample, so that when the jackknife variance estimation procedure is implemented, approximate unbiased estimates of sampling variance are obtained. This section gives the specifics for generating the replicate weights for the 2012 LTT assessment samples. The theory that underlies the jackknife variance estimators used in NAEP studies is discussed in the section Replicate Variance Estimation. For each sample, replicates were formed in two steps. First, each school was assigned to one or more of 62 replicate strata. In the next step, a random subset of schools (or, in some cases, students within schools) in each replicate stratum was excluded. The remaining subset and all schools in the other replicate strata then constituted one of the 62 replicates. A replicate weight was calculated for each of the 62 replicates using weighting procedures similar to those used for the full-sample weight. Each replicate base weight contains an additional component, known as a replicate factor, to account for the subsetting of the sample to form the replicate. By repeating the various Appendices A-C NAEP 2019-2020 86 weighting procedures on each set of replicate base weights, the impact of these procedures on the sampling variance of an estimate is appropriately reflected in the variance estimate. Each of the 62 replicate weights for student k in school s and stratum j can be expressed as follows: where STU_BWTjsk(r) is the student base weight for replicate r; SCH_NRAFjs(r) is the school-level nonresponse adjustment factor for replicate r; STU_NRAFjsk(r) is the student-level nonresponse adjustment factor for replicate r; SCH_TRIMjs is the school-level weight trimming adjustment factor; and STU_TRIMjsk is the student-level weight trimming adjustment factor. Specific school and student nonresponse adjustment factors were calculated separately for each replicate, thus the use of the index (r), and applied to the replicate student base weights. Computing separate nonresponse adjustment factors for each replicate allows resulting variances from the use of the final student replicate weights to reflect components of variance due to these various weight adjustments. School and student weight trimming adjustments were not replicated, that is, not calculated separately for each replicate. Instead, each replicate used the school and student trimming adjustment factors derived for the full sample. Statistical theory for replicating trimming adjustments under the jackknife approach has not been developed in the literature. Due to the absence of a statistical framework, and since relatively few school and student weights in NAEP require trimming, the weight trimming adjustments were not replicated. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_rep_var_est.aspx Appendices A-C NAEP 2019-2020 87 NAEP Technical Documentation Website NAEP Technical DocumentatioDefining Replicate Strata and Forming Replicates for the 2012 LTT Assessment In the NAEP 2012 LTT assessment, replicates were formed separately for each sample indicated by age (9, 13, 17), and school type (public, private). The first step in forming replicates was to assign each first-stage sampling unit in a primary stratum to a replicate stratum. In 2012, the formation of replicate strata varied by noncertainty and certainty primary sampling units (PSUs). For noncertainty PSUs, the first-stage units were PSUs, and the primary stratum was the combination of region and metropolitan status (MSA or non-MSA). For certainty PSUs, the firststage units were schools, and the primary stratum was school type (public or private). For noncertainty PSUs, where only one PSU was selected per PSU stratum, replicate strata were formed by pairing sampled PSUs with similar stratum characteristics within the same primary stratum (region by metropolitan status). This was accomplished by first sorting the 38 sampled PSUs by PSU stratum number and then grouping adjacent PSUs into 19 pairs. The values for a PSU stratum number reflect region and metropolitan status, as well as socioeconomic characteristics such as percent Black and percent children below poverty (those eligible for free/reduced-price school lunch). The formation of these 19 replicate strata in this manner models a design of selecting two PSUs with probability proportional to size with replacement from each of 19 strata. For certainty PSUs, the first stage of sampling is at the school level, and the formation of replicate strata must reflect the sampling of schools within the certainty PSUs. Replicate strata were formed by sorting the sampled schools in the 29 certainty PSUs by their order of selection within a primary stratum (school type) so that the sort order reflected the implicit stratification (region, locality type, race/ethnicity classification, and student enrollment for public schools; and region, private school type, and student enrollment size for private schools) and systematic sampling features of the sample design. The first-stage units were then paired off into 43 preliminary replicate strata. Within each primary stratum with an even number of first-stage units, all of the preliminary replicate strata were pairs, and within primary strata with an odd number of first-stage units, one of the replicate strata was a triplet (the last one), and all others were pairs. If there were more than 43 preliminary replicate strata within a primary stratum, the preliminary replicate strata were grouped to form 43 replicate strata. This grouping effectively maximized the distance in the sort order between grouped preliminary replicate strata. The first 43 preliminary replicate strata, for example, were assigned to 43 different final replicate strata in order (1 through 43), with the next 43 preliminary replicate strata assigned to final replicate strata 1 through 43, so that, for example, preliminary replicate stratum 1, preliminary replicate stratum Appendices A-C NAEP 2019-2020 88 44, preliminary replicate stratum 87 (if there were that many), etc., were all assigned to the first final replicate stratum. The final replicate strata for the schools in the certainty PSUs were 1 through 43. Within each pair of preliminary replicate stratum, the first first-stage unit was assigned as the first variance unit and the second first-stage unit as the second variance unit. Within each triplet preliminary replicate stratum, the three schools were assigned variance units 1 through 3. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_repwts_strata.aspx NAEP Technical Documentation Website NAEP Technical Documentation Defining Replicate Strata and Forming Replicates for the 2012 LTT Assessment In the NAEP 2012 LTT assessment, replicates were formed separately for each sample indicated by age (9, 13, 17), and school type (public, private). The first step in forming replicates was to assign each first-stage sampling unit in a primary stratum to a replicate stratum. In 2012, the formation of replicate strata varied by noncertainty and certainty primary sampling units (PSUs). For noncertainty PSUs, the first-stage units were PSUs, and the primary stratum was the combination of region and metropolitan status (MSA or non-MSA). For certainty PSUs, the firststage units were schools, and the primary stratum was school type (public or private). For noncertainty PSUs, where only one PSU was selected per PSU stratum, replicate strata were formed by pairing sampled PSUs with similar stratum characteristics within the same primary stratum (region by metropolitan status). This was accomplished by first sorting the 38 sampled PSUs by PSU stratum number and then grouping adjacent PSUs into 19 pairs. The values for a PSU stratum number reflect region and metropolitan status, as well as socioeconomic characteristics such as percent Black and percent children below poverty (those eligible for free/reduced-price school lunch). The formation of these 19 replicate strata in this manner models a design of selecting two PSUs with probability proportional to size with replacement from each of 19 strata. For certainty PSUs, the first stage of sampling is at the school level, and the formation of replicate strata must reflect the sampling of schools within the certainty PSUs. Replicate strata were formed by sorting the sampled schools in the 29 certainty PSUs by their order of selection within a primary stratum (school type) so that the sort order reflected the Appendices A-C NAEP 2019-2020 89 implicit stratification (region, locality type, race/ethnicity classification, and student enrollment for public schools; and region, private school type, and student enrollment size for private schools) and systematic sampling features of the sample design. The first-stage units were then paired off into 43 preliminary replicate strata. Within each primary stratum with an even number of first-stage units, all of the preliminary replicate strata were pairs, and within primary strata with an odd number of first-stage units, one of the replicate strata was a triplet (the last one), and all others were pairs. If there were more than 43 preliminary replicate strata within a primary stratum, the preliminary replicate strata were grouped to form 43 replicate strata. This grouping effectively maximized the distance in the sort order between grouped preliminary replicate strata. The first 43 preliminary replicate strata, for example, were assigned to 43 different final replicate strata in order (1 through 43), with the next 43 preliminary replicate strata assigned to final replicate strata 1 through 43, so that, for example, preliminary replicate stratum 1, preliminary replicate stratum 44, preliminary replicate stratum 87 (if there were that many), etc., were all assigned to the first final replicate stratum. The final replicate strata for the schools in the certainty PSUs were 1 through 43. Within each pair of preliminary replicate stratum, the first first-stage unit was assigned as the first variance unit and the second first-stage unit as the second variance unit. Within each triplet preliminary replicate stratum, the three schools were assigned variance units 1 through 3. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_repwts_strata.aspx NAEP Technical Documentation Website NAEP Technical Documentation Computing School-Level Replicate Base Weights for the 2012 LTT Assessment For the NAEP 2012 LTT assessment, school-level replicate base weights for school s in primary stratum j (SCH_BWTjs(r), r = 1,..., 62) were calculated as follows: Appendices A-C NAEP 2019-2020 90 where SCH_BWTjs is the school base weight for school s in primary stratum j, Rjr is the set of schools within the r-th replicate stratum for primary stratum j, and Ujs is the variance unit (1 or 2) for school s in primary stratum j. For schools in replicate strata comprising three variance units, two sets of school-level replicate base weights were computed (see replicate variance estimation for details): one for the first replicate r1 and another for the second replicate r2. The two sets of school-level replicate base weights SCH_BWTjs(r1), r1 = 1,..., 62 and SCH_BWTjs(r2), r2 = 1,..., 62 were calculated as described below. where SCH_BWTjs is the school base weight for school s in primary stratum j, Rjr1 is the set of schools within the r1-th replicate stratum for primary stratum j, Rjr2 is the set of schools within the r2-th replicate stratum for primary stratum j, and Ujs is the variance unit (1, 2, or 3) for school s in primary stratum j. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_repwts_schl.aspx Appendices A-C NAEP 2019-2020 91 NAEP Technical Documentation Website NAEP Technical Documentation Computing Student-Level Replicate Base Weights for the 2012 LTT Assessment For the 2012 LTT assessment, the calculation of the student-level replicate base weights for student k from school s in stratum j for each of the 62 replicates, STU_BWTjsk(r), where r = 1 to 62, were calculated as follows: where SCH_BWTjs(r) is the replicate school base weight; SCHSESWTjs is the school-level session assignment weight used in the full-sample weight; WINSCHWTjs is the within-school student sampling weight used in the full-sample weight; STUSESWTjsk is the student-level session assignment weight used in the full-sample weight; SUBJFACjs is the subject factor used in the full-sample weight; SUBADJjs is the substitute adjustment factor used in the full-sample weight; and YRRND_AFjs is the year-round adjustment factor used in the full-sample weight. These components are described on the Student Base Weights page. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_repwts_stud.aspx NAEP Technical Documentation Website NAEP Technical Documentation Replicate Variance Estimation for the 2012 Assessment Appendices A-C NAEP 2019-2020 92 Variances for NAEP assessment estimates are computed using the paired jackknife replicate variance procedure. This technique is applicable for common statistics, such as means and ratios, as well as for more complex statistics such as Item Response Theory (IRT) scores. In general, the paired jackknife replicate variance procedure involves pairing clusters of firststage sampling units to form H variance strata (h = 1, 2, 3, ...,H) with two units per stratum. The first replicate is formed by deleting one unit at random from the first variance stratum, inflating the weight of the remaining unit to weight up to the variance stratum total, and using all other units from the other (H - 1) strata. This procedure is carried out for each variance stratum resulting in H replicates, each of which provides an estimate of the population total. The jackknife estimate of the variance for any given statistic is given by the following formula: where represents the full sample estimate of the given statistic, and represents the corresponding estimate for replicate h. Each replicate undergoes the same weighting procedure as the full sample so that the jackknife variance estimator reflects the contributions to or reductions in variance resulting from the various weighting adjustments. The NAEP jackknife variance estimator is based on 62 variance strata resulting in a set of 62 replicate weights assigned to each school and student. The basic idea of the paired jackknife variance estimator is to create the replicate weights so that use of the jackknife procedure results in an unbiased variance estimator for simple totals and means, which is also reasonably efficient (i.e., has a low variance as a variance estimator). The jackknife variance estimator will then produce a consistent (but not fully unbiased) estimate of variance for (sufficiently smooth) nonlinear functions of total and mean estimates such as ratios, regression coefficients, and so forth (Shao and Tu, 1995). The development below shows why the NAEP jackknife variance estimator returns an unbiased variance estimator for totals and means, which is the cornerstone to the asymptotic results for nonlinear estimators. See for example Rust (1985). This paper also discusses why this variance estimator is generally efficient (i.e., more reliable than alternative approaches requiring similar computational resources). The development is done for an estimate of a mean based on a simplified sample design that closely approximates the sample design for first-stage units used in the NAEP studies. The sample design is a stratified random sample with H strata with population weights Wh, stratum Appendices A-C NAEP 2019-2020 93 sample sizes nh, and stratum sample means unbiased variance estimator . The population estimator and standard are: with The paired jackknife replicate variance estimator assigns one replicate h=1,…, H to each stratum, so that the number of replicates equals H. In NAEP, the replicates correspond generally to pairs and triplets (with the latter only being used if there are an odd number of sample units within a particular primary stratum generating replicate strata). For pairs, the process of generating replicates can be viewed as taking a simple random sample (J) of size nh/2 within the replicate stratum, and assigning an increased weight to the sampled elements, and a decreased weight to the unsampled elements. In certain applications, the increased weight is double the full sample weight, while the decreased weight is in fact equal to zero. In this with , the latter being the sample simplified case, this assignment reduces to replacing mean of the sampled nh/2 units. Then the replicate estimator corresponding to stratum ris The r-th term in the sum of squares for is thus: In stratified random sampling, when a sample of size nr/2 is drawn without replacement from a population of size nr,, the sampling variance is See for example Cochran (1977), Theorem 5.3, using nr, as the “population size,” nr/2 as the “sample size,” and sr2 as the “population variance” in the given formula. Thus, Appendices A-C NAEP 2019-2020 94 Taking the expectation over all of these stratified samples of size nr/2, it is found that In this sense, the jackknife variance estimator “gives back” the sample variance estimator for means and totals as desired under the theory. In cases where, rather than doubling the weight of one half of one variance stratum and assigning a zero weight to the other, the weight of one unit is multiplied by a replicate factor of (1+δ), while the other is multiplied by (1- δ), the result is that In this way, by setting δ equal to the square root of the finite population correction factor, the jackknife variance estimator is able to incorporate a finite population correction factor into the variance estimator. In practice, variance strata are also grouped to make sure that the number of replicates is not too large (the total number of variance strata is usually 62 for NAEP). The randomization from the original sample distribution guarantees that the sum of squares contributed by each replicate will be close to the target expected value. For triples, the replicate factors are perturbed to something other than 1.0 for two different replicate factors, rather than just one as in the case of pairs. Again in the simple case where replicate factors that are less than 1 are all set to 0, with the replicate weight factors calculated as follows. For unit i in variance stratum r where weight wi is the full sample base weight. Furthermore, for r' = r + 31 (mod 62): Appendices A-C NAEP 2019-2020 95 And for all other values r*, other than r and r´,wi(r*) = 1. In the case of stratified random sampling, this formula reduces to replacing replicate r and with for replicate r'. with for is the sample mean from a “2/3” sample of is the sample mean from 2nr/3 units from the nr sample units in the replicate stratum, and another overlapping “2/3” sample of 2nr/3 units from the nr sample units in the replicate stratum. The r-th and r´-th replicates can be written as: From these formulas, expressions for the r-th and r´-th components of the jackknife variance estimator are obtained (ignoring other sums of squares from other grouped components attached to those replicates): These sums of squares have expectations as follows, using the general formula for sampling variances: Appendices A-C NAEP 2019-2020 96 Thus, as desired again. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_var_est_appdx.aspx NAEP Technical Documentation Website NAEP Technical Documentation Quality Control on Weighting Procedures for the 2012 LTT Assessment Given the complexity of the weighting procedures utilized in NAEP, a range of quality control (QC) checks was conducted throughout the weighting process to identify potential problems with collected student-level demographic data or with specific weighting procedures. The QC processes included Main QC Findings of Interest Participation, Exclusion, and Accommodation Rates Nonresponse Bias Analysis checks performed within each step of the weighting process; checks performed across adjacent steps of the weighting process; review of response, exclusion, and accommodation rates; checking demographic data of individual schools; comparisons with 2008 demographic data; and nonresponse bias analyses. Appendices A-C NAEP 2019-2020 97 To validate the weighting process, extensive tabulations of various school and student characteristics at different stages of the process were conducted. The school-level characteristics included in the tabulations were enrollment by race/ethnicity and urban-centric locale. At the student level, the tabulations included race/ethnicity, gender, categorized grade, students with disability (SD) status, English language learners (ELL) status, and participation status in National School Lunch Program (NSLP). http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_qc_procedures.aspx NAEP Technical Documentation Website NAEP Technical Documentation Participation, Exclusion and Accommodation Rates for the 2012 LTT Assessment Final participation, exclusion, and accommodation rates were presented in quality control tables for each age and subject by reporting group. School-level participation rates were calculated as they had been calculated for previous assessments and according to National Center for Education Statistics (NCES) standards. Age 9 Mathematics Age 9 Reading Age 13 Mathematics Age 13 Reading Age 17 Mathematics School-level participation rates were below 85 percent for Age 17 Reading private schools at all three ages. Student-level participation rates were all above 85 percent. As required by NCES standards, nonresponse bias analyses were conducted on each reporting group falling below the 85 percent participation threshold. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_ltt_part_exclusion_acc_rates.as px Appendices A-C NAEP 2019-2020 98 NAEP TECHNICAL DOCUMENTATION Participation, Exclusion, and Accommodation Rates for Age 9 Mathematics for the 2012 LTT Assessment The following table displays the school-level participation rates and student-level participation, exclusion, and accommodation rates for the age 9 long-term trend (LTT) mathematics assessment. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sampled schools only and do not reflect any effect of substitution. The rates weighted by the school base weight and enrollment show the approximate proportion of the student population in the domain that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates for age 17 long-term trend mathematics assessment, by geographic region and school type: 2012 Number of schools in original sample School participation rates (percent) before substitution (weighted by school base weight and enrollment) School participation rates (percent) before substitution (weighted by school base weight only) 482 83.82 Northeast all 81 Midwest all of students sampled Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 80.26 10,900 1.74 88.06 9.57 92.27 74.44 2,000 2.55 85.59 13.29 97 90.74 90.45 2,100 1.46 88.15 10.59 South all 184 82.17 78.53 4,100 1.49 89.96 8.06 West all 120 72.76 75.82 2,600 1.72 87.18 7.91 National public 389 85.58 87.57 10,000 1.86 88.22 9.61 National private 93 62.51 60.45 833 0.13 85.87 9.11 Catholic 16 88.18 86.99 378 0.25 86.80 5.97 Non-Catholic 77 40.30 50.18 455 0.00 84.42 12.30 Geographic region and school type National all Number NOTE: National all includes national public, national private, Bureau of Indian Education (BIE), and Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS) that are located in the United States. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Mathematics Long-Term Trend Assessment. Appendices A-C NAEP 2019-2020 99 NAEP TECHNICAL DOCUMENTATION Participation, Exclusion, and Accommodation Rates for Age 9 Reading for the 2012 LTT Assessment The following table displays the school-level participation rates and student-level participation, exclusion, and accommodation rates for the age 9 long-term trend (LTT) reading assessment. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sampled schools only and do not reflect any effect of substitution. The rates weighted by the school base weight and enrollment show the approximate proportion of the student population in the domain that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates for age 9 long-term trend reading assessment, by geographic region and school type: 2012 School participation rates (percent) before substitution (weighted by school base weight and enrollment) School participation rates (percent) before substitution (weighted by school base weight only) 484 86.64 81.54 83 93.39 77.87 Midwest all 100 90.82 South all 186 West all Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 9,800 1.68 94.94 10.46 1,500 1.54 94.55 13.30 86.94 1,800 1.50 95.10 12.64 84.18 76.81 4,200 2.31 94.99 10.36 115 82.22 84.85 2,300 0.96 95.00 6.71 National public 347 89.03 89.93 8,900 1.79 95.03 11.15 National private 137 61.16 58.60 918 0.44 93.80 2.18 32 95.06 92.80 392 0.00 97.52 2.04 105 37.71 44.77 526 0.77 89.86 2.29 Geographic region and school type National all Number of schools in original sample Northeast all Catholic Non-Catholic Number of students sampled NOTE: National all includes national public, national private, Bureau of Indian Education (BIE), and Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS) that are located in the United States. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Reading Long-Term Trend Assessment. Appendices A-C NAEP 2019-2020 100 NAEP TECHNICAL DOCUMENTATION Participation, Exclusion, and Accommodation Rates for Age 13 Mathematics for the 2012 LTT Assessment The following table displays the school-level participation rates and student-level participation, exclusion, and accommodation rates for the age 13 long-term trend (LTT) mathematics assessment. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sampled schools only and do not reflect any effect of substitution. The rates weighted by the school base weight and enrollment show the approximate proportion of the student population in the domain that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates for age 13 long-term trend mathematics assessment, by geographic region and school type: 2012 and school type Number of schools in original sample School participation rates (percent) before substitution (weighted by school base weight and enrollment) School participation rates (percent) before substitution (weighted by school base weight only) Number of students sampled National all 505 87.87 80.75 85 94.87 66.98 Midwest all 106 90.38 91.73 South all 189 87.69 West all 125 81.27 National public 375 National private 130 Catholic Non-Catholic Geographic region Northeast all Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 10,000 1.17 93.03 10.61 1,600 0.61 91.14 14.78 1,900 1.12 94.70 10.96 78.36 4,100 1.56 92.26 10.07 80.68 2,400 1.00 93.90 8.21 89.94 89.99 9,000 1.27 92.85 11.04 68.63 62.72 995 0.16 95.10 6.03 37 91.61 91.70 489 0.34 95.43 3.22 93 49.13 50.95 506 0.00 94.70 8.49 NOTE: National all includes national public, national private, Bureau of Indian Education (BIE), and Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS) that are located in the United States. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Mathematics Long-Term Trend Assessment. Appendices A-C NAEP 2019-2020 101 NAEP TECHNICAL DOCUMENTATION Participation, Exclusion, and Accommodation Rates for Age 13 Reading for the 2012 LTT Assessment The following table displays the school-level participation rates and student-level participation, exclusion, and accommodation rates for the age 13 long-term trend (LTT) reading assessment. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sampled schools only and do not reflect any effect of substitution. The rates weighted by the school base weight and enrollment show the approximate proportion of the student population in the domain that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates for age 13 long-term trend reading assessment, by geographic region and school type: 2012 Geographic region and school type National all Northeast all Number of schools in original sample School participation rates (percent) before substitution (weighted by school base weight and enrollment) School participation rates (percent) before substitution (weighted by school base weight only) 505 87.87 80.75 85 94.87 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 10,000 1.89 93.19 10.14 66.98 1,600 1.60 92.23 14.57 Number of students sampled Midwest all 106 90.38 91.73 1,900 1.43 94.97 11.48 South all 189 87.69 78.36 4,200 2.42 92.45 8.84 West all 125 81.27 80.68 2,400 1.74 93.21 7.71 National public 375 89.94 89.99 9,000 2.03 93.13 10.69 National private 130 68.63 62.72 986 0.38 93.94 4.16 Catholic 37 91.61 91.70 484 0.21 96.42 2.01 Non-Catholic 93 49.13 50.95 502 0.53 91.05 6.16 NOTE: National all includes national public, national private, Bureau of Indian Education (BIE), and Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS) that are located in the United States. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Reading Long-Term Trend Assessment. Appendices A-C NAEP 2019-2020 102 NAEP TECHNICAL DOCUMENTATION Participation, Exclusion, and Accommodation Rates for Age 17 Mathematics for the 2012 LTT Assessment The following table displays the school-level participation rates and student-level participation, exclusion, and accommodation rates for the age 17 long-term trend (LTT) mathematics assessment. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sampled schools only and do not reflect any effect of substitution. The rates weighted by the school base weight and enrollment show the approximate proportion of the student population in the domain that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates for age 13 long-term trend reading assessment, by geographic region and school type: 2012 Number of schools in original sample School participation rates (percent) before substitution (weighted by school base weight and enrollment) School participation rates (percent) before substitution (weighted by school base weight only) of students sampled 505 87.87 80.75 85 94.87 66.98 Midwest all 106 90.38 91.73 South all 189 87.69 West all 125 81.27 National public 375 National private 130 Catholic Non-Catholic Geographic region and school type National all Northeast all Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 10,000 1.89 93.19 10.14 1,600 1.60 92.23 14.57 1,900 1.43 94.97 11.48 78.36 4,200 2.42 92.45 8.84 80.68 2,400 1.74 93.21 7.71 89.94 89.99 9,000 2.03 93.13 10.69 68.63 62.72 986 0.38 93.94 4.16 37 91.61 91.70 484 0.21 96.42 2.01 93 49.13 50.95 502 0.53 91.05 6.16 Number NOTE: National all includes national public, national private, Bureau of Indian Education (BIE), and Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS) that are located in the United States. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Reading Long-Term Trend Assessment. Appendices A-C NAEP 2019-2020 103 NAEP TECHNICAL DOCUMENTATION Participation, Exclusion, and Accommodation Rates for Age 17 Reading for the 2012 LTT Assessment The following table displays the school-level participation rates and student-level participation, exclusion, and accommodation rates for the age 17 long-term trend (LTT) reading assessment. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sampled schools only and do not reflect any effect of substitution. The rates weighted by the school base weight and enrollment show the approximate proportion of the student population in the domain that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates for age 17 long-term trend reading assessment, by geographic region and school type: 2012 Geographic region and school type National all Northeast all Midwest all Number of schools in original sample School participation rates (percent) before substitution (weighted by school base weight and enrollment) School participation rates (percent) before substitution (weighted by school base weight only) 482 83.82 80.26 81 92.27 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 11,300 1.96 88.29 8.92 74.44 2,000 2.68 84.55 13.83 Number of students sampled 97 90.74 90.45 2,200 1.39 89.18 10.13 South all 184 82.17 78.53 4,300 2.29 90.17 6.94 West all 120 72.76 75.82 2,700 1.43 87.90 6.96 National public 389 85.58 87.57 10,400 2.10 88.34 8.90 National private 93 62.51 60.45 858 0.13 87.64 9.18 Catholic 16 88.18 86.99 362 0.28 88.10 7.27 Non-Catholic 77 40.30 50.18 496 0.00 87.01 10.84 NOTE: National all includes national public, national private, Bureau of Indian Education (BIE), and Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS) that are located in the United States. Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Reading Long-Term Trend Assessment. Appendices A-C NAEP 2019-2020 104 NAEP TECHNICAL DOCUMENTATION NAEP Technical Documentation Nonresponse Bias Analysis for the 2012 LTT Assessment NCES Statistical standards call for a nonresponse bias analysis to be conducted for a sample with a response rate below 85 percent at any stage of sampling. Weighted school response rates for the 2012 assessment indicate a need for school nonresponse bias analyses for private school samples for ages 9, 13, and 17. No student nonresponse bias analyses were necessary since the student-level participation rates for all groups were above the 85 percent participation threshold. The school-level analyses were conducted separately at each age. Thus, three separate school- level analyses were conducted. The procedures and results from these analyses are summarized briefly below. The analyses conducted consider only certain characteristics of schools and students. They do not directly consider the effects of the nonresponse on student achievement, the primary focus of NAEP. Thus, these analyses cannot be conclusive of either the existence or absence of nonresponse bias for student achievement. For more details, please see the NAEP 2012 LTT NRBA report (337KB). Each school-level analysis was conducted in three parts. The first part of the analysis looked for potential nonresponse bias that was introduced through school nonresponse. The second part of the analysis examined the remaining potential for nonresponse bias after accounting for the mitigating effects of substitution. The third part of the analysis examined the remaining potential for nonresponse bias after accounting for the mitigating effects of both school substitution and school-level nonresponse weight adjustments. The characteristics examined were census region, reporting subgroup (private school type), urban-centric locale, size of school (categorical), size of school (continuous), and race/ethnicity enrollment percentages. Based on the school characteristics available, for the private school samples at ages 13 and 17, there does not appear to be evidence of substantial potential bias resulting from school substitution or school nonresponse. However, the analyses suggest that a potential for nonresponse bias remains for the age 9 private school samples for school percentage race/ethnicity characteristics. Please see the full report for more details. http://nces.ed.gov/nationsreportcard/tdw/weighting/2012/2012_weighting_nonresponse_bias_analysis.asp x Appendices A-C NAEP 2019-2020 105 NATIONAL CENTER FOR EDUCATION STATISTICS NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS National Assessment of Educational Progress (NAEP) 2019 and 2020 Appendix C1 2019 Sampling Memo OMB# 1850-0928 v.15 September 2018 No changes since v.10 Appendices A-C NAEP 2019-2020 106 Date: February 28, 2018 Memo: 20191.1A/1.1B/1.1D/1.1E To: William Ward, NCES Ed Kulick, ETS David Freund, ETS Chris Averett Kavemuii Murangi Erin Wiley Amy Dresher, ETS Dwight Brock Cathy White, Pearson Saira Brenner, Fulcrum Dianne Walsh Lauren Byrne Lisa Rodriguez Rick Rogers Rob Dymowski William Wall David Hubble Yiting Dai Jing Kang Sabrina Zhang Leslie Wallace Natalia Weil Greg Binzer From: Amy Lin, John Burke, and Lloyd Hicks Reviewer: Keith Rust Subject: I. Sample Design for 2019 NAEP - DRAFT Introduction For 2019, the NAEP assessment involves the following components: A. National assessments in reading, mathematics, and science at grades 4, 8, and 12; B. State-by-state and Trial Urban District Assessment (TUDA) assessments in reading and mathematics for public schools at grades 4 and 8; C. An assessment of mathematics in Puerto Rico at grades 4 and 8; D. Pilot tests in reading, mathematics, and vocabulary at grades 4 and 8. Appendices A-C NAEP 2019-2020 107 Below is a summary list of the features of the 2019 sample design. 1. The alpha samples for grades 4 and 8 public, and the delta samples for private schools at grades 4 and 8, will be used for the operational assessments in reading and mathematics. 2. The beta public school samples and the epsilon private school samples at grades 4 and 8 will be used for the national science assessments and the various pilot tests. The beta and epsilon samples at grade 12 will be used for the operational reading, mathematics, and science assessments. 3. As in recent NAEP studies, each Trial Urban District Assessment (TUDA) sample will form part of the corresponding state sample, and each state sample will form part of the national sample. There are twenty-seven Trial Urban District Assessment (TUDA) participants. These are the same districts that participated in 2017. 4. Schools in the alpha and delta samples will be assessed using DBA with tablets. Schools in the beta and epsilon samples will receive a mixture of DBA assessments, using tablets, and pencil and paper (PBA) assessments. 5. All BIE schools and students will be included in the operational samples at grades 4 and 8. This is because, after a hiatus in 2017, the National Indian Education Study (NIES) is resuming. Having all BIE students in sample is designed to provide detailed national results for American Indian and Alaskan Native (AIAN) students in reading and mathematics, as part of the NIES. 6. There will be no samples in territories other than for Puerto Rico at grades 4 and 8. 7. As in 2017, the Department of Defense Schools are expected to be reported as a single jurisdiction (DoDEA). 8. At grade 12, there will be no state-level samples. 9. Oversampling of private schools at grades 4 and 8 will be done at the same level as 2017. Response rates permitting, this will allow separate reporting for reading and mathematics for Catholic and non-Catholic schools at grades 4 and 8, but no further breakdowns by private school type. 10. The sample sizes of assessed students for these various components are shown in Table 1 (which also shows the approximate numbers of participating schools). 11. In the beta samples, there will be moderate oversampling of schools with moderate to high proportions of Black, Hispanic, and American Indian and Alaska Native students. Appendices A-C NAEP 2019-2020 108 Table 1. Target sample sizes of assessed students, and expected number of participating schools, for 2019 NAEP Spiral Spiral Indic. Grade 4 Nat’l/state reading (DBA) Nat’l/state math (DBA) Puerto Rico (DBA) Total - alpha Total- delta Typical max. no. students/school Average assessed students/school Total schools - alpha, delta Science (DBA) Science (PBA) Math Pilot Reading Pilot Vocabulary initial-Pilot Total - beta Total - epsilon Typical max. no. students/school Average assessed students/school Total schools - beta, epsilon Total number of students grade 4 Total number of schools grade 4 Appendices A-C NAEP 2019-2020 DS DS DP Jurisdictions States (incl. DC, Urban DoDEA) districts 52 52 1 27 27 Students Public Private school school students students 176,000 144,000 3,000 323,000 50 40 8,075 DA PA DA DA DA 17,100 8,100 10,350 4,050 1,980 41,580 3,700 3,000 0 6,700 50 25 268 1,900 900 1,150 450 220 62 50 832 4,620 62 25 185 364,580 8,907 11,320 453 Total 179,000 147,000 3,000 323,000 6,700 8,343 19,000 9,000 11,500 4,500 2,200 41,580 4,620 1,017 375,900 9,360 109 Table 1. Target sample sizes of assessed students, and expected number of participating schools, for 2019 NAEP (Continued) Spiral Spiral Indic. Grade 8 Nat’l/state reading (DBA) Nat’l/state math (DBA) Puerto Rico (DBA) Total - alpha Total- delta Typical max. no. students/school Average assessed students/school Total schools - alpha, delta Science (DBA) Science (PBA) Math Pilot Reading pilot Vocabulary initial-Pilot Total – beta Total – epsilon Typical max. no. students/school Average assessed students/school Total schools - beta, epsilon Total number of students grade 8 Total number of schools grade 8 Appendices A-C NAEP 2019-2020 DS, DT DS, DT DP Jurisdictions States (incl. DC, Urban DoDEA) districts 52 52 1 27 27 Students Public Private school school students students 176,000 144,000 3,000 323,000 50 47 6,870 DA PA DA DA DA 17,100 9,000 10,350 4,050 1,980 42,480 3,700 3,000 0 6,700 50 25 268 1,900 1,000 1,150 450 220 63 52 817 4,720 63 25 189 365,480 7,687 11,420 457 Total 179,000 147,000 3,000 323,000 6,700 7,138 19,000 10,000 11,500 4,500 2,200 42,480 4,720 1,006 376,900 8,144 110 Table 1. Target sample sizes of assessed students, and expected number of participating schools, for 2019 NAEP (Continued) Spiral Jurisdictions States Spiral (incl. DC, Urban Indic. DoDEA) districts Grade 12 Reading (DBA) Reading (PBA) Math (DBA) Math (PBA) Science (DBA) Science (PBA) Total - beta Total- epsilon Typical max. no. students/school Average assessed students/school Total schools – beta, epsilon DA PA DA PA DA PA Total number of students grade 12 Total number of schools grade 12 GRAND TOTAL STUDENTS GRAND TOTAL SCHOOLS II. Students Public Private School school students students 13,500 11,700 12,600 12,600 17,100 9,900 77,400 1,500 1,300 1,400 1,400 1,900 1,100 Total 15,000 13,000 14,000 14,000 19,000 11,000 77,400 8,600 68 50 1,548 8,600 68 40 215 77,400 1,548 8,600 215 86,000 1,763 807,460 18,142 31,340 1,125 838,800 19,267 1,763 Assessment Types The assessment spiral types are shown in Table 2. Four different spirals will be used at grades 4 and 8, and two at grade 12. Session IDs contain six characters, traditionally. The first two characters identify the assessment “type” (subjects and type of spiral in a general way). Grade is contained in the second pair of characters, and the session sequential number (within schools) in the last two characters. For example, session DS0401 denotes the first grade 4 reading and mathematics operational DBA assessment in a given school. Appendices A-C NAEP 2019-2020 111 Table 2. ID NAEP 2019 assessment types and IDs Type Subjects Grades Schools Comments All schools in the alpha (except Puerto Rico) and delta samples. DS Operational DBA Reading, math (22:27) 4, 8 Public, Private DA Operational, and pilot DBA Science, reading, math, vocabulary (190:45:115:22) 4, 8 Public, Private All schools in the beta and epsilon samples. PA Operational Science 4, 8 DA Operational PA Operational DP Operational Public, Private Public, Private Public, Private Public All schools in the beta and epsilon samples. All schools in the beta and epsilon samples. All schools in the beta and epsilon samples. Puerto Rico alpha samples III. Reading, math, science, (15:14:19) Reading, math, science (13:14:11) Mathematics 12 12 4, 8 Sample Types and Sizes In similar fashion to past years (but somewhat different), we will identify four different types of school samples: Alpha, Beta, Delta, and Epsilon. These distinguish sets of schools that will be conducting distinct portions of the assessment. 1. Alpha Samples at Grades 4 and 8 These are public school samples for grades 4 and 8. They will be used for the operational state-bystate assessments in reading and mathematics, and contribute to the national samples for these subjects as well. There will be alpha samples for each state, DC, DoDEA, BIE, and Puerto Rico. The details of the target student sample sizes for the alpha samples are as follows: A. At each grade, the target student sample size is 5,700 per state. The goal in each state (before considering the contribution of TUDA districts) is to roughly assess 2,700 student for math and 2,200 students for reading. The DS session type will be used. B. There will be samples for twenty-seven TUDA districts. For the six large TUDA districts (New York, Los Angeles, Chicago, Miami-Dade, Clark Co., and Houston) the assessed student target sample sizes are three-quarters the size of a state sample (3,675). The target sample size after considering attrition is 4,275. Appendices A-C NAEP 2019-2020 112 C. For the remaining 21 TUDA districts, the assessed student target sample sizes are half the size of a state sample (2,450). The target sample size after inflation to account for attrition is 2,850. D. Note that, above, there is a conflict between sample size requirements at the state level, and the TUDA district level. This will be resolved as in previous years: the districts will have the target samples indicated in B and C, and reflected in Table 3. For the states that contain one or more of these districts, the target sample size indicated in A (and shown in Table 3) will be used to determine a school sampling rate for the state, which will be applied to the balance of the state outside the TUDA district(s). Thus the target student sample sizes, shown in Table 3, for states that contain a TUDA district, are only ‘design targets’, and are smaller than the final total sample size for the state, but larger than the sample for the balance of the state, exclusive of its TUDA districts. In the case of the District of Columbia, the state sample size requirement is that all schools and students be included. This renders moot any requirements for the DC TUDA sample, which by default consists of all schools operated by the DCPS district (but excludes charter schools in DC, even though those are all included in the state sample, as these are not operated by DCPS). E. In Puerto Rico, the target sample size is 4,000 per grade (grades 4 and 8), with the goal of assessing 3,000 students. Under normal circumstances this target would be set at 3,500, but because of the rapid and substantial shifts in the school population in Puerto Rico, this has been increased to provide some insurance against attrition due to closed schools and declining enrollments. As in past state-by-state assessments, schools with fewer than 20 students in the grade in question will be sampled at a moderately lower rate than other schools (at least half, and often higher, depending upon the size of the school). This is in implicit recognition of the greater cost and burden associated with surveying these schools. As mentioned above, the NAEP 2019 design includes an oversample of high proportion American Indian schools in certain states (as part of the NIES design). These schools will be sampled at higher rates than the other schools. The NIES oversample will take place in Arizona, Minnesota, North Carolina, Oregon, Utah, Washington, and Wisconsin. Schools with relatively large percentages of American Indian students will be separately stratified, as explained below, and oversampled by factors ranging from 3 to 6 based on state and grade. Table 3 below shows the thresholds used to define the NIES oversampling strata along with their corresponding oversampling factors. Appendices A-C NAEP 2019-2020 113 Table 3. Percent American Indian thresholds and oversampling factors for the NIES school oversample by state and grade State Arizona Minnesota North Carolina Oregon Utah Washington Wisconsin Grade 4 Percent American Oversampling Indian thresholds factor 50 4 10 5 10 6 10 6 5 6 10 6 10 6 Grade 8 Percent American Oversampling Indian thresholds factor 50 3 10 5 10 6 10 6 5 6 10 6 10 6 Table 4 shows the target student sample sizes, and the approximate counts of schools to be selected in the alpha samples, along with the school and student frame counts, by state and TUDA districts for grades 4 and 8. The table also identifies the jurisdictions where we take all schools and where we take all students. Note that the additional sample that will result from NIES oversampling is not included in this table. Table 5 consolidates the target student (and resulting school) sample size numbers, to show the total target sample sizes in each state, combining the TUDA targets with those for the balance of the state. Appendices A-C NAEP 2019-2020 114 Table 4. Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for the 2019 state-by-state and TUDA district assessments (Alpha samples) Grade 4 Jurisdiction Alabama Alaska Arizona Arkansas Bureau of Indian Education California Colorado Connecticut Delaware District of Columbia DoDEA Schools Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Appendices A-C NAEP 2019-2020 Schools in frame 709 352 1,193 480 137 5,979 1,054 602 119 119 110 2,225 1,248 205 381 2,205 1,050 638 704 721 760 320 903 958 1,711 956 423 1,166 Schools in sample 120 185 123 121 137 119 123 121 99 119 95 118 115 118 128 124 119 128 132 120 121 147 119 120 123 126 118 129 Students in frame 57,548 9,361 86,472 36,937 3,357 471,633 67,814 39,544 10,393 5,536 7,547 212,520 133,243 15,494 22,864 149,235 78,837 37,147 37,202 52,221 55,735 13,444 67,399 70,968 111,240 65,262 38,316 69,574 Grade 8 Overall target student sample size 5,700 5,700 5,700 5,700 3,357 5,700 5,700 5,700 5,700 5,536 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 ** ** Schools in frame 456 270 793 303 113 2,933 567 339 61 69 65 1,219 562 83 209 1,561 489 368 393 417 488 202 373 485 1,083 712 287 709 Schools in sample 118 131 122 114 113 120 121 118 61 69 65 119 115 62 100 123 116 118 125 121 120 112 117 116 123 128 112 127 Students in frame 55,820 9,019 83,469 36,503 2,936 455,487 65,088 40,679 10,105 4,520 5,629 202,235 129,475 13,314 22,319 151,830 79,653 35,691 36,033 50,755 51,981 13,473 61,983 71,662 114,211 63,732 36,486 67,833 Overall target student sample size 5,700 5,700 5,700 5,700 2,936 5,700 5,700 5,700 5,700 4,520 5,629 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 ** * ** ** 115 Table 4. Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for the 2019 state-by-state and TUDA district assessments (Alpha samples) (Continued) Grade 4 Jurisdiction Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Puerto Rico Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Wyoming Appendices A-C NAEP 2019-2020 Schools in frame 392 532 394 270 1,371 444 2,471 1,457 261 1,740 869 746 1,607 931 164 643 312 995 4,431 621 216 1,109 1,231 417 1,099 192 Schools in sample 174 146 119 135 120 128 118 119 166 121 132 128 118 169 111 118 163 120 118 118 216 117 122 138 128 137 Students in frame 11,534 23,315 35,875 13,734 99,697 26,208 201,226 118,118 8,471 129,087 50,988 43,589 130,442 31,308 10,777 57,878 10,517 77,202 399,283 50,010 6,204 97,550 81,904 20,578 61,686 7,639 Grade 8 Overall target student sample size 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 4,000 5,700 5,700 5,700 5,700 5,700 5,700 6,204 5,700 5,700 5,700 5,700 5,700 ** Schools in frame 271 294 171 142 765 232 1,498 728 184 1,093 583 428 888 398 60 306 246 584 2,251 256 121 379 609 190 649 89 Schools in sample 136 114 91 89 118 110 117 117 142 119 127 124 116 161 60 115 135 119 119 113 121 114 122 110 123 89 Students in frame 10,811 22,561 34,346 14,078 99,117 25,079 196,197 117,176 7,789 131,562 48,784 42,824 131,525 30,211 10,720 54,617 9,657 73,441 383,849 47,320 5,999 95,187 79,084 20,464 61,152 7,042 Overall target student sample size 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 5,700 4,000 5,700 5,700 5,700 5,700 5,700 5,700 5,999 5,700 5,700 5,700 5,700 5,700 * ** * 116 Table 4. Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for the 2019 state-by-state and TUDA district assessments (Alpha samples) (Continued) Grade 4 Jurisdiction Albuquerque Atlanta Austin Baltimore City Boston Charlotte Chicago Clark County, NV Cleveland Dallas Denver Detroit Duval County, FL Fresno Fort Worth Guilford County, NC Hillsborough County, FL Houston Jefferson County, KY Los Angeles Miami Milwaukee New York City Philadelphia San Diego Shelby County, TN District of Columbia PS Schools in frame 95 55 80 128 72 105 433 226 71 151 102 65 119 68 85 74 176 174 100 496 285 111 788 148 120 120 76 Schools in sample 57 55 56 64 57 57 93 87 71 58 59 55 58 55 57 56 58 86 59 87 88 65 88 58 59 59 76 Students in frame 7,412 4,285 6,867 6,716 4,086 11,696 27,360 25,311 2,754 13,325 7,108 3,889 10,313 5,788 7,073 5,492 16,522 17,729 7,718 45,361 26,690 5,668 73,248 11,227 9,125 9,250 3,584 Grade 8 Overall target student sample size 2,850 2,850 2,850 2,850 2,850 2,850 4,275 4,275 2,754 2,850 2,850 2,850 2,850 2,850 2,850 2,850 2,850 4,275 2,850 4,275 4,275 2,850 4,275 2,850 2,850 2,850 3,584 * ** ** Schools in frame 40 23 22 96 43 46 434 80 70 41 60 49 50 19 32 29 87 61 43 122 177 83 524 112 38 61 32 Schools in sample 40 23 22 62 43 35 93 58 70 41 47 49 35 19 32 29 50 49 29 75 82 56 88 54 38 44 32 Students in frame 6,691 3,554 5,427 5,504 3,667 11,007 27,895 24,676 2,685 10,873 6,060 2,963 8,873 5,147 5,977 5,339 15,096 13,063 7,306 36,142 26,957 4,977 66,513 8,849 7,433 8,277 2,394 Overall target student sample size 2,850 3,554 2,850 2,850 3,667 2,850 4,275 4,275 2,685 2,850 2,850 2,963 2,850 2,850 2,850 2,850 2,850 4,275 2,850 4,275 4,275 2,850 4,275 2,850 2,850 2,850 2,394 * ** * ** ** * ** * * * * ** Counts for states do not reflect the oversampling for their constituent TUDA districts, nor the impact of oversampling for NIES. Appendices A-C NAEP 2019-2020 117 Target student sample sizes reflect sample sizes prior to attrition due to exclusion, ineligibility, and nonresponse. * identifies jurisdictions where all schools (but not all students) for the given grade are included in the NAEP sample. ** identifies jurisdictions where all students for the given grade are included in the NAEP sample. Appendices A-C NAEP 2019-2020 118 Table 5. Total sample sizes, combining state and TUDA samples Grade 4 Jurisdiction Alabama Alaska Arizona Arkansas Bureau Of Indian Education California Colorado Connecticut Delaware District Of Columbia DoDEA Schools Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Appendices A-C NAEP 2019-2020 Schools in frame 709 352 1,193 480 137 5,979 1,054 602 119 119 110 2,225 1,248 205 381 2,205 1,050 638 704 721 760 320 903 958 1,711 956 423 1,166 392 Schools in sample 120 184 123 121 137 305 169 121 99 119 95 293 166 118 128 194 119 128 132 162 121 147 169 170 174 126 118 129 174 Students in frame 57,548 9,361 86,472 36,937 3,357 471,633 67,814 39,544 10,393 5,536 7,547 212,520 133,243 15,494 22,864 149,235 78,837 37,147 37,202 52,221 55,735 13,444 67,399 70,968 111,240 65,262 38,316 69,574 11,534 Grade 8 Overall target student sample size 5,700 5,700 5,700 5,700 3,357 14,945 7,950 5,700 5,700 5,536 5,700 14,238 8,367 5,700 5,700 8,927 5,700 5,700 5,700 7,709 5,700 5,700 7,983 8,222 8,350 5,700 5,700 5,700 5,700 ** ** Schools in frame 456 270 793 303 113 2,933 567 339 61 69 65 1,219 562 83 209 1,561 489 368 393 417 488 202 373 485 1,083 712 287 709 271 Schools in sample 117 131 123 114 113 240 157 118 61 69 65 256 135 61 100 194 116 118 125 133 120 112 167 153 168 128 112 127 136 Students in frame 55,820 9,019 83,469 36,503 2,936 455,487 65,088 40,679 10,105 4,520 5,629 202,235 129,475 13,314 22,319 151,830 79,653 35,691 36,033 50,755 51,981 13,473 61,983 71,662 114,211 63,732 36,486 67,833 10,811 Overall target student sample size 5,700 5,700 5,700 5,700 2,936 15,064 8,018 5,700 5,700 4,520 5,629 14,238 9,098 5,700 5,700 8,924 5,700 5,700 5,700 7,730 5,700 5,700 8,044 9,076 8,515 5,700 5,700 5,700 5,700 ** * ** ** 119 Table 5. Total sample sizes, combining state and TUDA samples (Continued) Grade 4 Jurisdiction Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Puerto Rico Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Wyoming Total Schools in frame 532 394 270 1,371 444 2,471 1,457 261 1,740 869 746 1,607 931 164 643 312 995 4,431 621 216 1,109 1,231 417 1,099 192 52,343 Schools in sample 145 124 135 120 152 164 215 166 189 132 128 166 169 111 118 163 165 361 118 216 117 122 138 181 137 8,314 Students in frame 23,315 35,875 13,734 99,697 26,208 201,226 118,118 8,471 129,087 50,988 43,589 130,442 31,308 10,777 57,878 10,517 77,202 399,283 50,010 6,204 97,550 81,904 20,578 61,686 7,639 3,831,663 Grade 8 Overall target student sample size 5,700 5,945 5,700 5,700 6,923 7,899 10,570 5,700 8,332 5,700 5,700 8,059 4,000 5,700 5,700 5,700 7,866 17,881 5,700 6,204 5,700 5,700 5,700 8,024 5,700 369,705 ** Schools in frame 294 171 142 765 232 1,498 728 184 1,093 583 428 888 398 60 306 246 584 2,251 256 121 379 609 190 649 89 29,024 Schools in sample 114 91 89 118 123 165 165 142 186 127 124 162 161 60 115 135 149 251 113 121 114 122 110 168 89 7,082 Students in frame 22,561 34,346 14,078 99,117 25,079 196,197 117,176 7,789 131,562 48,784 42,824 131,525 30,211 10,720 54,617 9,657 73,441 383,849 47,320 5,999 95,187 79,084 20,464 61,152 7,042 3,732,513 Overall target student sample size 5,700 5,874 5,700 5,700 7,021 8,042 10,604 5,700 8,269 5,700 5,700 8,167 4,000 5,700 5,700 5,700 7,907 17,999 5,700 5,999 5,700 5,700 5,700 8,085 5,700 370,482 * ** * Sample sizes for each state do reflect the samples in the TUDA districts within the state, but do not reflect the impact of NIES oversampling. * identifies jurisdictions where all schools (but not all students) for the given grade are included in the NAEP sample. ** identifies jurisdictions where all students for the given grade are included in the NAEP sample. Appendices A-C NAEP 2019-2020 120 Stratification Each state and grade will be stratified separately, but using a common approach in all cases. TUDA districts will be separated from the balance of their state, and each part stratified separately. The first level of stratification will be based on urban-centered type of location. This variable has 12 levels (some of which may not be present in a given state or TUDA district), and these will be collapsed so that each of the resulting location categories contains at least 9 percent of the student population (12 percent for large TUDA districts and 18 percent for small TUDA districts). In those states with school oversampling for NIES, the schools to be oversampled will be placed in a separate stratum, apart from the location strata used for other schools. Within each of the resulting location categories (with the exception of the NIES oversampling strata), schools will be assigned a minority enrollment status. This is based on the two race/ethnic groups that are the second and third most prevalent within the location category. If these groups are both low in percentage terms, no minority classification will be used. Otherwise three (or occasionally four) equal-sized groups (generally high, medium, and low minority) will be formed based on the distribution across schools of the two minority groups. Within the resulting location and minority group classes (of which there are likely to be from three to fifteen, depending upon the jurisdiction), and the NIES oversampling stratum in states where this is applicable, schools will be sorted by a measure derived from school level results from the most recent available state achievement tests at the relevant grade. In general, mathematics test results will be used, but where these are not available, reading results will be used. In the few states that do not have math or reading tests at grades 4 and 8 (or where we are unable to match the results to the NAEP school frame), instead of achievement data, schools will be sorted using a measure of socioeconomic status. This is the median household income of the 5-digit ZIP Code area where the school is located, based on the 2016 ACS (5-year) data. For BIE and DoDEA schools neither achievement data nor income data are available, and so grade enrollment is used in these cases. Once the schools are sorted by location class, minority enrollment class, and achievement data (or household income), a systematic sample of schools will be selected using a random start. Schools will be sampled with probability proportional to size. The exact details of this process are described in the individual sampling specification memos. Appendices A-C NAEP 2019-2020 121 2. Beta Sample The beta sample comprises the national public school samples at grades 4, 8, and 12. At grades 4 and 8 the beta samples will be used for the national science assessments (PBA and DBA) and for pilot tests of reading, math, and vocabulary (DBA-only). At grade 12 the beta sample will be used for the operational reading, mathematics, and science assessments (PBA and DBA). Each of these samples will be nationally representative, selected to have minimal overlap with the alpha sample schools at the same grade. The number of students targeted per school will be 62 at grade 4, 63 at grade 8, and 68 at grade 12. In order to increase the likelihood that the results for American Indian/Alaskan Native (AIAN) students can be reported for the operational samples, we will oversample high-AIAN public schools. That is, a public school with more than 5 AIAN students and greater than 5 percent AIAN enrollment will be given four times the chance of selection of a public school of the same size with a lower AIAN percentage. For all other schools, whenever there are more than 10 Black or Hispanic students enrolled and the combined Black and Hispanic enrollment exceeds 15 percent, the school will be given twice the chance of selection of a public school of the same size with a lower percentage of these two groups. This approach is effective in increasing the sample sizes of AIAN, Black, and Hispanic students without inducing undesirably large design effects on the sample, either overall, or for particular subgroups. Stratification The Beta samples will have an implicit stratification, using a hierarchy of stratifiers and a serpentine sort. The highest level of the hierarchy is Census division (9 implicit strata). The next stratifier in the hierarchy is type of location, which has twelve categories. Many of the type of location strata nested within Census divisions will be collapsed with neighboring type of location cells (this will occur if the expected school sample size within the cell is less than 4.0). These geographic strata will be subdivided into three substrata: 1) schools being oversampled for AIAN, 2) schools being oversampled for Blacks and Hispanics, and 3) low-minority schools not being oversampled. If the expected sample size in an oversampled substratum is less than 8.0, it will be left as is. If the expected sample size is greater than 8.0, then it will be subdivided into up to four substrata (two for expected sample size up to 12.0, three for expected sample size up to 16.0, and four for expected sample size greater than 16.0). For the oversampling strata, the subdivision will be by percentage AIAN or percentage Black and Hispanic, as appropriate. For the low-minority sampling strata, the subdivision will be by state or groups of contiguous states. Within these substrata, the schools are to Appendices A-C NAEP 2019-2020 122 be sorted by school type (public, BIE, DoDEA) and median household income from the 2016 5year ACS (using a serpentine sort within the school type substrata). 3. Delta Samples These are the private school samples at grades 4 and 8 for conducting the operational assessments in reading and mathematics. The sample sizes are large enough to report results by Catholic and nonCatholic at grades 4 and 8. Approximately half the sample at each grade will be from Catholic schools. The number of students targeted per school will be 50 at each grade. Stratification The private schools are to be explicitly stratified by private school type (Catholic/Other). Within each private school type, stratification will be by Census region (4 categories), type of location (12 categories), race/ethnicity composition, and enrollment size. In general, where there are few or no schools in a given stratum, categories will be collapsed together, always preserving the private school type. 4. Epsilon Sample With regard to subjects and grades assessed, this sample is analogous to the beta sample, but for private schools. However, in contrast to the beta sample, there will be no oversampling of high minority schools. The same stratification variables will be used as for the delta samples. The epsilon sample schools will have minimum overlap with the delta sample schools which, given the respective sample sizes, means that no schools will be selected for both the delta and epsilon samples at the same grade. The number of students targeted per school will be 62 at grade 4, 63 at grade 8, and 68 at grade 12. Appendices A-C NAEP 2019-2020 123 IV. New Schools To compensate for the fact that files used to create the NAEP school sampling frames are at least two years out of date at the time of frame construction, we will supplement the Alpha, Beta, Delta, and Epsilon samples with new school samples at each grade. The new school samples will be drawn using a two-stage design. At the first stage, a minimum of ten school districts (in states with at least ten districts) will be selected from each state for public schools, and ten Catholic dioceses will be selected nationally for the private schools. The sampled districts and dioceses will be asked to review lists of their respective schools and identify new schools. Frames of new schools will be constructed from these updates, and new schools will be drawn with probability proportional to size using the same sample rates as their corresponding original school samples. The school sample sizes in the above tables do not reflect new school samples. V. Substitute Samples Substitute samples will be selected for each of the Beta, Delta and Epsilon samples. The substitute school for each original will be the next “available” school on the sorted sampling frame, with the following exceptions: A. Schools selected for any NAEP samples will not be used as substitutes. B. Private schools whose school affiliation is unknown will not be used as substitutes. Also, unknown affiliated private schools in the original samples will not get substitutes. C. A school can be a substitute for one and only one sample. (If a school is selected as a substitute school for grade 8, for example, it cannot be used as a substitute for grade 4.) D. A public school substitute will always be in the same state as its original school. E. A catholic school substitute will always be a Catholic school, and the same for non-Catholic schools. Appendices A-C NAEP 2019-2020 124 VI. Contingency Samples The districts that are taking part in the TUDA program are volunteers. Thus it is possible that at some point over the next few months, a given district might choose to opt out of the TUDA program for 2019. However, it is not acceptable for all schools in such a district to decline NAEP, as then the state estimates will be adversely affected. Thus to deal with this possibility, in each TUDA district, subsamples of the alpha sample schools will be identified as contingency samples. In the event that the district withdraws from the TUDA program prior to the selection of the student sample, all alpha sampled schools from that district will be dropped from the sample, with the exception of those selected in the contingency sample. The contingency sample will provide a proportional representation of the district, within the aggregate state sample. Student sampling in those schools will then proceed in the same way as for the other schools within the same state. VII. Student Sampling Students within the sampled schools will be selected with equal probability. The student sampling parameters vary by sample type (Alpha, Beta, Delta, and Epsilon) and grade, as described below. Alpha Sample, Grades 4 and 8 Schools (Except Puerto Rico) A. All students, up to 52, will be selected. B. If the school has more than 52 students, a systematic sample of 50 students will be selected. In some schools, the school may be assigned more than one ‘hit’ in sampling. In these schools we will select a sample of size 50 times the number of hits, taking all students if this target is greater than or equal to 50/52 of the total enrollment. Alpha Sample, Puerto Rico Grades 4 and 8 A. All students, up to 26, will be selected. B. If the school has more than 26 students, a systematic sample of 25 students will be selected. Delta Samples, Grades 4 and 8 A. All students, up to 52, will be selected. B. If the school has more than 52 students, a systematic sample of 50 students will be selected. Appendices A-C NAEP 2019-2020 125 Beta and Epsilon Samples, Grades 4, 8, and 12 A. At grade 4 all students will be selected, up to 70. If the school has more than 70 students, 62 will be selected. Of these students, 50 will be assigned to DBA and the rest to PBA. In schools with fewer than 21 students, all will be assigned to DBA or all to PBA. In schools with 32 to 37 students, 25 will be assigned to DBA and the rest to PBA. In all other schools, 25/31 of the students will be assigned to DBA with the rest to PBA. B. At grade 8 all students will be selected, up to 70. If the school has more than 70 students, 63 will be selected. Of these students, 50 will be assigned to DBA and the rest to PBA. In schools with fewer than 21 students, all will be assigned to DBA or all to PBA. In schools with 31 to 37 students, 25 will be assigned to DBA and the rest to PBA. In all other schools, 50/63 of the students will be assigned to DBA with the rest to PBA. C. At grade 12 all students will be selected, up to 75. If the school has more than 75 students, 68 will be selected. Of these students, 38 will be assigned to DBA and the rest to PBA. In schools with fewer than 20 students, all will be assigned to DBA or all to PBA. In schools with 32 to 36 students, 19 will be assigned to DBA and the rest to PBA. In all other schools, 19/34 of the students will be assigned to DBA with the rest to PBA. VIII. Weighting Requirements The Operational Reading and Mathematics Assessments, Grade 4 and 8 The sample weights will reflect probabilities of selection, school and student nonresponse, any trimming, and the random assignment to the particular subject. There will be separate replication schemes by grade and public/private. Weights will also be derived for the Puerto Rico KaSA assessment at grades 4 and 8. The Operational Reading and Mathematics Assessments, Grade 12, and Science Assessment, Grades 4, 8, and 12 The exact weighting requirements for these samples have yet to be determined. One possibility is that three sets of weights will be required – for DBA alone, PBA alone, and DBA/PBA combined. The sample weights will reflect probabilities of selection, school and student nonresponse, any trimming, and the random assignment to the particular subject. There will be a separate replication scheme by grade and public/private. Appendices A-C NAEP 2019-2020 126 Pilot Assessments in Reading, Mathematics, and Vocabulary, at Grades 4 and 8 As is standard practice, only preliminary weights will be provided for these assessments. The sample weights will reflect probabilities of selection, and the random assignment to the particular subject (necessary because these assessments are spiraled in with other assessment components). Appendices A-C NAEP 2019-2020 127 NATIONAL CENTER FOR EDUCATION STATISTICS NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS National Assessment of Educational Progress (NAEP) 2019 and 2020 Appendix C2 2020 Sampling Memo OMB# 1850-0928 v.15 March 2019 Appendices A-C NAEP 2019-2020 128 Date: February 26, 2019 To: William Ward, NCES Ed Kulick, ETS David Freund, ETS Amy Dresher, ETS Cathy White, Pearson William Wall Rob Dymowski Chris Averett Kavemuii Murangi John Burke Saira Brenner, Fulcrum Greg Binzer Lauren Byrne Lisa Rodriguez Rick Rogers Dwight Brock Joel Wakesberg Jing Kang Veronique Lieber Shaohua Dong Memo: From: Dave Hubble Reviewers: Keith Rust, Leslie Wallace Subject: Sample Design for 2020 NAEP - Overview I. 2020-m01v01psu/s Introduction For 2020, the sample design involves only one component: Long-Term Trend (LTT) Paper-based Assessment (PBA). 1. LTT Reading Operational assessments at ages 9, 13, and 17; 2. LTT Mathematics Operational assessments at ages 9, 13, and 17; There will be no pilot assessments in 2020 LTT PBA. The target sample sizes of assessed students for LTT are shown in Table 1 (which also shows the estimated numbers of sampled schools before attrition). Unlike most years, the NAEP 2020 LTT assessment components will take place at various seasons throughout the school year. With that in mind, the last column was added to provide the season in which the assessment will be fielded. Appendices A-C NAEP 2019-2020 129 Table 1. 2020 NAEP Sample Sizes (Public and Private) and Season Fielded Session Age 9 LTT Math (O) LTT Reading (O) Subtotal Schools LT09 Age 13 LTT Math (O) LTT Reading (O) Subtotal Schools LT13 Age 17 LTT Math (O) LTT Reading (O) Subtotal Schools LT17 GRAND TOTAL Schools Public school students Private school students Total students Season Fielded 7,200 7,200 14,400 430 800 800 1,600 220 8,000 8,000 16,000 650 Winter 7,200 7,200 14,400 440 800 800 1,600 190 8,000 8,000 16,000 630 Fall 7,200 7,200 14,400 490 800 800 1,600 130 8,000 8,000 16,000 620 Spring 43,200 1,360 4,800 540 48,000 1,900 (O) = Operational II. Assessment Types For 2020 NAEP, there is only one type of assessment. While the detailed target counts of LTT assessed students are provided in Table 1, a summary of major points follows. The LTT spiral at ages 9, 13, and 17. This paper-based assessment (PBA) will be conducted in LTT PSUs. The spiral includes Math and Reading operational samples. The LTT session type has a target of 16,000 assessed students each at age 9, age 13, and age 17. Note, 10% of the assessed students are allocated to private schools. This roughly represents a proportional sample, as about 10% of the population attends private schools. III. Primary Sampling Units Selection and Overlap Control As the LTT assessments are national, with a total sample size of assessed students of about 48,000, for reasons of operational efficiency in conducting the assessments a sample of Primary Sampling Units (PSUs) was selected, and all sampled schools were drawn from within the sampled PSUs. The PSUs were created from aggregates of counties. Data on counties were obtained from the 2010 Census, and the definitions of Metropolitan Statistical Areas (MeSAs) used were the December 2009 Office of Management and Budget (OMB) definitions. Each Metropolitan Statistical Area (MeSA) constitutes a PSU, except that MeSAs that cross state boundaries were split into their individual regional components. Non-metropolitan PSUs were formed by aggregating counties into geographic units of sufficient minimum size to provide enough schools to constitute a workload of about 1% of the total sample. These Appendices A-C NAEP 2019-2020 130 PSUs were made of contiguous counties where possible, and almost contiguous counties (separated by MeSA counties) otherwise. Each PSU falls within a single state. This process generated a frame of approximately 1,000 PSUs. The PSUs were stratified, using characteristics aggregated from county-level characteristics, found by analysis to be related to NAEP achievement in past assessments. A sample of 105 PSUs was selected for the LTT samples. Twenty-nine large MeSAs were selected with certainty, and the remaining sample was a stratified probability proportional to size (PPS) sample, where the size measure was a function of the number of children as given in the most recent population estimates prepared by the U.S. Census Bureau. IV. Stratification and Oversampling As in the recent past, the plan is to draw separate public and private school samples. This approach has proven to be useful, in that, selecting the samples separately has three advantages: 1) it permits the timing of sample selection to vary between public and private schools, should this prove necessary; 2) it allows us to readily assume different response and eligibility rates for public schools and private schools; and 3) it makes it easier to use different sort variables for public schools and private schools. It also allows for the possibility of a late change of mind concerning the sample sizes that differ between public and private schools. Explicit stratification will take place at the PSU level. For schools within PSUs, stratification gains are achieved by sorting the school file prior to systematic selection. As in past national samples, the expectation is that, within the set of certainty MeSA PSUs within a census region, PSU will not necessarily be the highest level sort variable. Thus, type of location will be used as the primary sort variable. Consider for example the large MeSAs in the Midwest region. The design is aimed primarily at getting the correct balance of city, suburban, town, and rural schools crossed by city size and distance from urbanized areas, as a priority over getting exactly a proportional representation from each MeSA (Chicago, Detroit, Minneapolis), although of course it should be possible to get a high degree of control over both of these characteristics. The sort of the schools will use other variables beyond the type of location variable, such as a race/ethnicity percentage variable. The exact set of variables used in sorting the schools prior to sampling will be specified in the particular sampling specification memos. In addition, we will implement oversampling of certain public schools. In order to increase the likelihood that the results for American Indian/Alaskan Native (AIAN) students can be reported for the operational samples, we will oversample high-AIAN public schools for LTT for ages 9, 13, and 17. That is, a public school with 5 percent or more AIAN enrollment will be given four times the chance of selection of a public school of the same size with a lower AIAN percentage. Recent research into oversampling schemes that could benefit AIAN students indicates that this approach should be effective in increasing the sample sizes of AIAN students, without inducing undesirably large design effects on the sample, either overall or for particular subgroups. In addition, high minority public schools for LTT that are not oversampled for AIAN enrollment will be oversampled for Black and Hispanic enrollment. That is, a public school with 15 percent or more Black and Hispanic combined enrollment will be given twice the chance of selection of a public school of the same size with a lower percentage of these two groups. This approach is effective in increasing the sample sizes of Black and Hispanic students, without inducing undesirably large design effects on the sample, either overall or for particular subgroups. Beyond this, we will also implement the oversampling of AIAN, Black, and Hispanic students at the student level in schools not being oversampled at the school level. The preliminary 2017/18 CCD and the updated 2017/18 PSS school files were approved for use by NCES. They serve as the basis for the public and private school frames for the 2020 NAEP. Appendices A-C NAEP 2019-2020 131 V. New Schools To compensate for the fact that files used to create the NAEP school sampling frames are two years out of date at the time of assessment, we will supplement the samples in the LTT PSUs with a sample of new public schools for each age sample. . The new school samples will be drawn using a three-stage design. The first stage is the selection of the LTT sample PSUs, as discussed above. At the second stage, a national sample of school districts will be selected from the LTT sample PSUs. The sampled districts will be asked to review lists of their respective schools and identify new schools. Frames of new schools will be constructed from these updates, and, at the third stage, new schools will be drawn with probability proportional to size using the same sampling rates as their corresponding original school samples. Note that the student and school sample sizes in Table 1 do not reflect these new school samples. However, some schools from the original sample will prove to be closed or otherwise ineligible, and the new school procedure essentially compensates for the sample losses from these sources, as well as ensuring full coverage of the population. VI. Within PSU Overlap Control with Other Samples As LTT is the only NAEP sample in 2020 and there are no other NCES-related operational samples (e.g., PIRLS, PISA, etc.) in 2020 there will be no need for LTT within PSU sampling overlap control. Selection of 2020 Field Trial schools for PIRLS and PISA will avoid NAEP LTT sample schools. VII. Substitute Samples A portion of the eligible 2020 LTT sample schools will choose to not participate in the assessment. In order to maintain sample yields, substitute school samples will be selected for each of the 2020 LTT samples. Within the 2020 LTT samples, the order for selecting substitute schools will be from “oldest” to “youngest”. That is, age 17, 13, and then 9. This ordering of samples by age is necessary since no school can be selected as a substitute more than once and there are fewer schools available to serve as substitutes at the higher ages. This will be done separately for both public and private schools. The general steps for selecting substitutes are to put the substitute frames in their original sampling sort order, and take the 'nearest neighbor' of each original sampled school, excluding schools selected for any of the NAEP 2020 LTT samples, schools already selected to serve as a substitute school, and schools which cross PSU or state boundaries, as potential substitutes. The nearest neighbor is the school adjacent (immediately preceding or succeeding) the original school in the sorted frame with the closer estimated age enrollment value. If estimated age enrollment of both potential substitute schools differs from the original school by the exact same amount, the selection procedure will randomly choose one of the schools. If neither the preceding or succeeding school is eligible to be a substitute, then the sampled school is not assigned a substitute. In addition, sampled private schools whose school affiliation is unknown will not get substitutes nor can such private schools not in sample serve as substitute schools. Also, new schools will not get substitute schools nor serve as substitutes. Appendices A-C NAEP 2019-2020 132 VIII. Student Sampling Students within the sampled schools will be selected with equal probability, except in public schools where oversampling of AIAN, Black and Hispanic students will take place. In addition to this, student sample sizes for LTT within each school are determined as the combined result of several factors: 1. We wish to take all students in relatively small schools. 2. We do not wish to have a sample that is too clustered for any one assessment subject. 3. We do not wish to have many physical sessions that contain only a very small number of students, as this is inefficient. 4. We do not wish to overburden the schools with unduly large student samples. The plans for LTT below reflect the design that results from considering each of these factors and balancing them. LTT Private Schools and Oversampled Public Schools In all private schools and public schools that are oversampled (as described in Section IV), the target sample size is 50 assessed students for each age. We will select all students of a certain age, up to 50. In schools with more than 50 such students we will select 50. There will be only one session type. LTT Non-Oversampled Public Schools In public schools not oversampled at the school level (i.e., under 5% AIAN and under 15% Black and Hispanic students), we will select 50 students plus an oversample of up to 5 additional AIAN, Black, and Hispanic students. The maximum number of sample students will be 55 in these schools. IX. Weighting Requirements The LTT operational samples currently require a single set of weights for each subject (LTT Math and LTT Reading at ages 9, 13, and 17), applied to reflect probabilities of selection, school and student nonresponse, any trimming, and the random assignment to the particular subject. There will be a separate replication scheme by age and public/private. LTT Preliminary weights will be developed as required by the DAR contractor. Appendices A-C NAEP 2019-2020 133
| File Type | application/pdf |
| File Title | Appendix A (Statute Authorizing NAEP) |
| Author | joconnell |
| File Modified | 2019-06-21 |
| File Created | 2018-02-01 |