Document
Appendices A-C Supplemental Documents
ICR 201705-1850-002 · OMB 1850-0928 · Object 73723102.
⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0928 can be found here:
Document [pdf]
Download: pdf | txt
NATIONAL CENTER FOR EDUCATION STATISTICS NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS National Assessment of Educational Progress (NAEP) 2018 and 2019 Appendix A-C Appendix A: External Advisory Committees Appendix B: NAEP 2011 Weighting Procedures Appendix C: 2018 Sampling Memo OMB# 1850-0928 v.5 April 2017 NATIONAL CENTER FOR EDUCATION STATISTICS NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS National Assessment of Educational Progress (NAEP) 2018 and 2019 Appendix A External Advisory Committees OMB# 1850-0928 v.5 March 2017 Table of Contents Appendix A-1: NAEP Design and Analysis Committee......................................................................... 4 Appendix A-2: NAEP Validity Studies Panel......................................................................................... 4 Appendix A-3: NAEP Quality Assurance Technical Panel .................................................................... 5 Appendix A-4: NAEP National Indian Education Study Technical Review Panel ................................ 5 Appendix A-5: NAEP Civics Standing Committee ................................................................................ 6 Appendix A-6: NAEP Economics Standing Committee ......................................................................... 6 Appendix A-7: Geography Standing Committee .................................................................................... 6 Appendix A-8: NAEP Mathematics Standing Committee ...................................................................... 7 Appendix A-9: NAEP Reading Standing Committee ............................................................................. 7 Appendix A-10: NAEP Science Standing Committee ............................................................................ 8 Appendix A-11: NAEP Survey Questionnaires Standing Committee ..................................................... 8 Appendix A-12: NAEP Technology and Engineering Literacy Standing Committee ............................. 9 Appendix A-13: NAEP U.S. History Standing Committee ..................................................................... 9 Appendix A-14: NAEP Writing Standing Committee ...........................................................................10 Appendix A-15: NAEP Principals’ Panel Standing Committee ............................................................10 Appendix A-16: NAEP Mathematics Translation Review Committee .................................................. 11 Appendix A-17: NAEP Science Translation Review Committee .......................................................... 11 Appendix A-18: NAEP Grade 8 Social Sciences Translation Review Committee................................. 11 Page 2 Appendix A-1: NAEP Design and Analysis Committee Name Betsy Becker Peter Behuniak Lloyd Bond Steve Elliott Derek Briggs Steve Elliott Ben Hansen Matthew Johnson Brian Junker David Kaplan Kenneth Koedinger Yan Li Sophia Rabe-Hesketh Michael Rodriguez S. Lynne Stokes Chun Wang Affiliation Florida State University University of Connecticut University of North Carolina, Greensboro (Emeritus)/ Carnegie Foundation (retired) University of Colorado University of Colorado Arizona State University University of Michigan Columbia University Carnegie Mellon University University of Wisconsin-Madison Carnegie Mellon University University of Maryland University of California, Berkeley University of Minnesota Southern Methodist University University of Minnesota Appendix A-2: NAEP Validity Studies Panel Name Affiliation Peter Behuniak University of Connecticut George Bohrnstedt American Institutes for Research, Washington, DC Jim Chromy RTI International (Emeritus Fellow), Raleigh, NC Phil Daro Strategic Education Research (SERP) Institute, Berkeley, CA University of Richard Duran California David Grissmer University of Virginia Larry Hedges Northwestern University Gerunda Hughes Howard University Ina Mullis Boston College Scott Norton Council of Chief State School Officers, Washington, DC Jim Pellegrino University of Illinois at Chicago/Learning Sciences Research Institute Gary Phillips American Institutes for Research, Washington, DC Lorrie Shepard University of Colorado at Boulder David Thissen The University of North Carolina at Chapel Hill Gerald Tindal University of Oregon Sheila Valencia Universty of Washington NAEP 2018-2019 OMB Clearance: Appendix A Page 3 Appendix A-3: NAEP Quality Assurance Technical Panel Name Affiliation Jamal Abedi University of California, Davis Chuck Cowan Analytic Focus LLC, San Antonio, TX Gail Goldberg Gail Goldberg Consulting, Ellicott City, MD Brian Gong National Center for the Improvement of Educational Assessment, Dover, NH Jim Pellegrino University of Illinois at Chicago/Learning Sciences Research Institute Mark Reckase Michigan State University Michael (Mike) Russell Boston College Phoebe Winter Consultant, Chesterfield, VA Richard Wolfe University of Toronto (Emeritus), Ontario, Canada Appendix A-4: NAEP National Indian Education Study Technical Review Panel Name Affiliation Doreen E. Brown ASD Education Center, Anchorage, AK Robert B.Cook Native American Initiative/Teach for America, Summerset, SD Steve Andrew Culpepper University of Illinois at Urbana-Champaign Susan C. Faircloth University of North Carolina Wilmington Jeremy MacDonald Rocky Boy Elementary, Box, Elder, MT Rebecca Izzo-Manymules Southwest Indian Polytechnic Institute, Albuquerque, NM Jeannette Muskett Miller Tohatchi High School, Tohatchi, NM Debora Norris Salt River Pima-Maicopa Indian Community Martin Reinhardt Northern Michigan University Tarajean Yazzie-Mintz Wakanyeja ECE Initative/American Indian College Fund, Denver, CO NAEP 2018-2019 OMB Clearance: Appendix A Page 4 Appendix A-5: NAEP Civics Standing Committee Name Affiliation Patricia Avery University of Minnesota Christopher Elnicki Cherry Creek School District, Greenwood Village, CO Fay Gore North Carolina Public Schools, Raleigh, NC Barry Leshinsky Challenger Middle School, Huntsville, AL Peter Levine CIRCLE (Center for Information & Research on Civic Learning and Engagement), Medford, MA Clarissa Peterson DePauw University Terri Richmond Golden Valley High School, Bakersville, CA Jackie Viana Miami-Dade County Schools, Miami, FL Appendix A-6: NAEP Economics Standing Committee Name Affiliation Kris Bertelsen Little Rock Branch-Federal Reserve Bank of St. Louis, Stephen Buckles Steven L. Cobb Little Rock, AR Vanderbilt University University of North Texas Jaime Festa-Daigle Lake Havasu High School, Lake Havasu City, AZ Julie Heath University of Memphis Richard MacDonald St. Cloud State University Andrea Morgan Oregon Department of Education, Salem, OR Kevin Smith Renaissance High School, Detroit, MI William Walstad University of Nebraska–Lincoln Appendix A-7: Geography Standing Committee Name Affiliation Sarah Bednarz Texas A&M University Osa Brand National Council for Geographic Education, Washington, DC Seth Dixon Rhode Island College Charlie Fitzpatrick ESRI Schools, Arlington, VA Ruth Luevanos Pacoima Middle School, Pacoima, CA Joe Stoltman Western Michigan University Kelly Swanson Johnson Senior High, St. Paul, MN NAEP 2018-2019 OMB Clearance: Appendix A Page 5 Appendix A-8: NAEP Mathematics Standing Committee Name Affiliation Jennifer Alvarez Sultana Elementary School, Ontario, CA Daniel Chazan University of Maryland, College Park Carl Cowen Indiana University–Purdue University Julie Guthrie Texas Education Agency Kathleen Heid Pennsylvania State University Mark Howell Gonzaga College High School, Washington, DC Russ Keglovits Nevada Department of Education, Carson City, NV Carolyn Maher Rutgers University Michele Mailhot Maine Department of Education, Augusta, ME Brian Nelson Curtis Corner Middle School, Wakefield, RI Matthew Owens Spring Valley High School, Columbia, SC Carole Philip Alice Deal Middle School, Washington, DC Melisa M. Ramos Trinidad Educación Bilingüe Luis Muñoz Iglesias, Cidra, PR Ann Trescott Stella Maris Academy, La Jolla, CA Appendix A-9: NAEP Reading Standing Committee Name Affiliation Marilyn Adams Brown University Peter Afflerbach University of Maryland Patricia Alexander University of Maryland Margretta Browne Richard Montgomery High School, Silver Spring, MD Julie Coiro University of Rhode Island Bridget Dalton University of Colorado Boulder Valerie Harrison Claflin University Karen Malone Fort Wingate High School, Fort Wingate, NM Pamela Mason Harvard Graduate School of Education Margaret McKeown University of Pittsburgh P. David Pearson University of California, Berkeley Jenny Thomson University of Sheffield, Sheffield, UK Monica Verra-Tirado Florida Department of Education, Tallahassee, FL Victoria Young Texas Education Agency, Austin, TX Zynia Zepeda Crane Elementary School District, Yuma, AZ NAEP 2018-2019 OMB Clearance: Appendix A Page 6 Appendix A-10: NAEP Science Standing Committee Name Alicia Cristina Alonzo Affiliation Michigan State University George Deboer Crystal Edwards American Association for the Advancement of Science, Washington, DC Millersville University Lawrence Township Public Schools, Lawrenceville, NJ Ibari Igwe Shrewd Learning, Elkridge, MD Michele Lombard Kenmore Middle School, Arlington, VA Emily Miller Consultant, WI Blessing Mupanduki Department of Defense, Washington, DC Amy Pearlmutter Littlebrook Elementary School, Princeton, NJ Brian Reiser Northwestern University, Evanston, IL Michal Robinson Alabama Department of Education, Montgomery, AL Gloria Schmidt Darby Junior High School, Fort Smith, AR Steve Semken Arizona State University, Tempe, AZ Roberta Tanner Board of Science Education, Longmont, CO David White Lamoille North Supervisory Union School District, Hyde Park, VT Alex Decaria Appendix A-11: NAEP Survey Questionnaires Standing Committee Name Affiliation Angela Duckworth University of Pennsylvania Hunter Gehlbach Harvard University Camille Farrington University of Chicago, Chicago, IL Gerunda Hughes Howard University David Kaplan University of Wisconsin-Madison Henry Levin Teachers College, Columbia University Stanley Presser University of Maryland Augustina Reyes University of Houston, Houston, TX Leslie Rutkowski Indiana University Bloomington Jonathon Stout Lock Haven University Roger Tourangeau Westat, Rockville, MD Akane Zusho Fordham University NAEP 2018-2019 OMB Clearance: Appendix A Page 7 Appendix A-12: NAEP Technology and Engineering Literacy Standing Committee Name Affiliation Keith Barton Indiana University Bloomington John Behrens Pearson eLEADS Center, Mishawaka, IN Brooke Bourdelat-Parks Biological Sciences Curriculum Study, Colorado Springs, CO Barbara Bratzel Shady Hill School, Cambridge, MA Lewis Chappelear James Monroe High School, North Hills, CA Britte Haugan Cheng SRI International, Menlo Park, CA Meredith Davis North Carolina State University Chris Dede Harvard Graduate School of Education Richard Duran University of California, Santa Barbara Maurice Frazier Oscar Smith High School, Chesapeake, VA Camilla Gagliolo Arlington Public Schools, Arlington, VA Christopher Hoadley New York University Eric Klopfer Massachusetts Institute of Technology Beth McGrath Stevens Institute of Technology Greg Pearson National Academy of Engineering, Washington, DC John Poggio University of Kansas Erin Reilly University of Southern California Troy Sadler University of Missouri Science Education Center, Columbia, MO Kimberly Scott Arizona State University Teh-Yuan Wan New York State Education Department, Albany, NY Appendix A-13: NAEP U.S. History Standing Committee Name Affiliation Keith Barton Indiana University Bloomington Michael Bunitsky Frederick County Public Schools, Frederick, MD Teresa Herrera Shenandoah Middle School, Miami, FL Cosby Hunt Center for Inspired Teaching, Washington, DC Helen Ligh Macy Intermediate School, Monterey, CA Amanda Prichard Green Mountain High School, Lakewood, CO Kim Rasmussen Auburn Washburn Unified School District, Topeka, KS Diana Turk New York University, New York, NY NAEP 2018-2019 OMB Clearance: Appendix A Page 8 Appendix A-14: NAEP Writing Standing Committee Name Margretta Browne Affiliation Montgomery County Public Schools, Silver Spring, MD Robert Crongeyer Elyse Eidman-Aadahl Robla School, Sacramento, CA National Writing Project, Berkeley, CA Nikki Elliot-Schuman Smarter Balanced Assessment Consortium Charles MacArthur University of Delaware, Newark, DE Michael McCloskey Norma Mota-Altman Johns Hopkins University, Baltimore, MD San Gabriel High School, Alhambra, CA Sandra Murphy University of California, Davis, Walnut Creek, CA Drew Sterner Tamanend Middle School, Warrington, PA Sherry Swain National Writing Project, Berkeley, CA Victoria Young Texas Education Agency, Austin, TX Appendix A-15: NAEP Principals’ Panel Standing Committee Name Affiliation David Atherton Clear Creek Middle School, Gresham, OR Ardith Bates Gladden Middle School, Chatsworth, GA Williams Carozza Harold Martin Elementary School, Hopkinton, NH Diane Cooper St. Joseph’s Academy, Clayton, MO Brenda Creel Alta Vista Elementary School, Cheyenne, WY Rita Graves Pin Oak Middle School, Bellaire, TX Don Hoover Lincoln Junior High School, Springdale, AR Stephen Jackson (Formerly with) Paul Laurence Dunbar High School, Washington, DC Anthony Lockhart Lake Shore Middle School, Belle Glade, FL Susan Martin Berrendo Middle School, Roswell, NM Lillie McMillan Porter Elementary School, San Diego, CA Jason Mix Howard Lake–Waverly–Winsted High School, Howard Lake, MN NAEP 2018-2019 OMB Clearance: Appendix A Page 9 Appendix A-16: NAEP Mathematics Translation Review Committee Name Affiliation Gilberto Cuevas Texas State University, San Marcos, TX Néstor Díaz Coral Gables Senior High School, Coral Gables, FL David Feliciano P.S./M.S 29, The Melrose School, Bronx, NY Yvonne Fuentes Author and Spanish Linguist, Carrollton, GA Flor Yanira Gurrola Valenzuela Washington Middle School, Albuquerque, NM Luz N. Rosario Cristóbal Puerto Rico Department of Education, Hato Rey, PR Melisa M. Ramos Trinidad Educación Bilingüe Luis Muñoz Iglesias, Cidra, PR Sonia Suazo Escuela Salvador Brau Elemental, Cayey, PR Enid Valle Kalamazoo College, Kalamazoo, MI Appendix A-17: NAEP Science Translation Review Committee Name Affiliation Néstor Díaz Coral Gables Senior High School, Coral Gables, FL Yvonne Fuentes Author and Spanish Linguist, Carrollton, GA Myrna Rasmussen Austin Independent School District, Austin, TX Enid Valle Kalamazoo College, Kalamazoo, MI Appendix A-18: NAEP Grade 8 Social Science Translation Review Committee Name Affiliation Yvonne Fuentes Author and Spanish Linguist, Carrollton, GA Jose Antonio Paulino Middle School Teacher, Nathan Strauss Preparatory School, NY, NY Dagoberto Eli Ramierz Bilingual Education Expert, Palmhurst, TX Enid Valle Kalamazoo College, Kalamazoo, MI NAEP 2018-2019 OMB Clearance: Appendix A Page 10 NATIONAL CENTER FOR EDUCATION STATISTICS NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS National Assessment of Educational Progress (NAEP) 2018 and 2019 Appendix B NAEP 2011 Weighting Procedures OMB# 1850-0928 v.5 March 2017 1 of 60 NAEP Technical Documentation Website Weighting Procedures for the 2011 Assessment NAEP assessments use complex sample designs to create student samples that generate population and subpopulation estimates with reasonably high precision. Student sampling weights ensure valid inferences from the student samples to their respective populations. In 2011, weights were developed for students sampled at grades 4, 8, and 12 for assessments in mathematics, reading, science, and a writing computer-based assessment (WCBA). Each student was assigned a weight to be used for making inferences about students in the target population. This weight is known as the final full-sample student weight, which contains the following major components: the student base weight; school nonresponse adjustments; student nonresponse adjustments; school weight trimming adjustments; student weight trimming adjustments; and student raking adjustment. Computation of Full-Sample Weights Computation of Base Weights School and Student Nonresponse Adjustments School and Student Weight Trimming Adjustments Student Raking Adjustment Computation of Replicate Weights for Variance Estimation Quality Control on Weighting Procedures The student base weight is the inverse of the overall probability of selecting a student and assigning that student to a particular assessment. The sample design that determines the base weights is discussed in the NAEP 2011 sample design section. The student base weight is adjusted for two sources of nonparticipation: school level and student level. These weighting adjustments seek to reduce the potential for bias from such nonparticipation by increasing the weights of students from participating schools similar to those schools not participating; and increasing the weights of participating students similar to those students from within participating schools who did not attend the assessment session (or makeup session) as scheduled. Furthermore, the final weights reflect the trimming of extremely large weights at both the school and student level. These weighting adjustments seek to reduce variances of survey estimates. Starting in 2009, an additional weighting adjustment was implemented in the state samples so that estimates for key student-level characteristics were in agreement across assessments in reading, mathematics, and science. This procedure was implemented using a raking procedure. In addition to the final full-sample weight, a set of replicate weights was provided for each student. These replicate weights are used to calculate the variances of survey estimates using the jackknife repeated replication method. The methods used to derive these weights were aimed at reflecting the features of the sample design, so that when the jackknife variance estimation procedure is implemented, approximately unbiased estimates of sampling variance are obtained. In addition, the various weighting procedures were repeated on each set of replicate weights to appropriately reflect the impact of the weighting adjustments on the sampling variance of a survey estimate. In 2011, a finite population correction (fpc) factor was used in computing variance estimates for the reading, mathematics, and science assessments. See Computation of Replicate Weights for Variance Estimation for details. Quality control checks were carried out throughout the weighting process to ensure the accuracy of the full-sample and replicate weights. See Quality Control for Weighting Procedures for the various checks implemented and main findings of interest. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 1 7/22/2016 2:46 PM 2 of 60 Computation of Full-Sample Weights for the 2011 Assessment The full-sample or final student weight is the sampling weight used to derive NAEP student estimates of population and subpopulation characteristics for a specified grade (4, 8, or 12) and assessment subject (reading, mathematics, science, and writing [WCBA]). The full-sample student weight reflects the number of students that the sampled student represents in the population for purposes of estimation. The summation of the final student weights over a particular student group provides an estimate of the total number of students in that group within the population. Computation of Base Weights School and Student Nonresponse Weight Adjustments School and Student Weight Trimming Adjustments The full-sample weight, which is used to produce survey estimates, is distinct from a Student Raking Adjustment replicate weight that is used to estimate variances of survey estimates. The full-sample weight is assigned to participating students and reflects the student base weight after the application of the various weighting adjustments. The full-sample weight for student k from school s in stratum j (FSTUWGTjsk) can be expressed as follows: where STU_BWTjsk is the student base weight; SCH_NRAFjs is the school-level nonresponse adjustment factor; STU_NRAFjsk is the student-level nonresponse adjustment factor; SCH_TRIMjs is the school-level weight trimming adjustment factor; STU_TRIMjsk is the student-level weight trimming adjustment factor; and STU_RAKEjsk is the student-level raking adjustment factor. School sampling strata for a given assessment varies by school type and grade. See the links below for descriptions of the school strata for the various assessments. Reading, mathematics, and science at grades 4 and 8 for public schools Reading, mathematics, and science at grades 4 and 8 for private schools WCBA at grades 8 and 12 for public schools WCBA at grades 8 and 12 for private schools http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_comp_full_samp_weights.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 2 7/22/2016 2:46 PM 3 of 60 Computation of Base Weights for the 2011 Assessment Every sampled school and student received a base weight equal to the reciprocal of its probability of selection. Computation of a school base weight varies by School Base Weights Student Base Weights type of sampled school (original or substitute); sampling frame (new school frame or not); and assessment subject (writing [WCBA] samples involved the selection of geographic units known as primary sampling units [PSUs], but reading, mathematics, and science did not). Computation of a student base weight reflects the student's overall probability of selection accounting for school and student sampling; assignment to session type at the school- and student-level; and the student's assignment to the reading, mathematics, science, or writing (WCBA) assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_base.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 3 7/22/2016 2:46 PM 4 of 60 School Base Weights for the 2011 Assessment The school base weight for a sampled school is equal to the inverse of its overall probability of selection. The overall selection probability of a sampled school differs by type of sampled school (original or substitute); sampling frame (new school frame or not); and assessment subject (writing [WCBA] samples involved the selection of geographic units known as primary sampling units [PSUs], but reading, mathematics, and science did not). The overall selection probability of an originally selected school in a reading, mathematics, or science sample is equal to its probability of selection from the NAEP public/private school frame. The overall probability of selection of an originally selected school in the WCBA sample reflects two components: Substitute schools for the 2011 private school national assessment Substitute public schools for the 2011 computer-based assessment Substitute private schools for the 2011 computer-based assessment the probability of selection of the PSU; and the probability of selection of the school within the selected PSU from the NAEP public/private school frame. The overall selection probability of a school from the new school frame in a reading, mathematics, science, or WCBA sample is the product of two quantities: the probability of selection of the school's district into the new-school district sample; and the probability of selection of the school into the new school sample. Substitute schools are preassigned to original schools and take the place of original schools if they refuse to participate. For weighting purposes, they are treated as if they were the original schools that they replaced, so substitute schools are assigned the school base weight of the original schools. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_base_school.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 4 7/22/2016 2:46 PM 5 of 60 Student Base Weights for the 2011 Assessment Every sampled student received a student base weight, whether or not the student participated in the assessment. The student base weight is the reciprocal of the probability that the student was sampled to participate in the assessment for a specified subject. The student base weight for student k from school s in stratum j (STU_BWTjsk) is the product of seven weighting components and can be expressed as follows: where SCH_BWTjs is the school base weight; SCHSESWTjs is the school-level session assignment weight that reflects the conditional probability, given the school, that the particular session type was assigned to the school; WINSCHWTjs is the within-school student weight that reflects the conditional probability, given the school, that the student was selected for the NAEP assessment; STUSESWTjsk is the student-level session assignment weight that reflects the conditional probability, given that the particular session type was assigned to the school, that the student was assigned to the session type; SUBJFACjsk is the subject spiral adjustment factor that reflects the conditional probability, given that the student was assigned to a particular session type, that the student was assigned the specified subject; SUBADJjs is the substitution adjustment factor to account for the difference in enrollment size between the substitute and original school; and YRRND_AFjs is the year-round adjustment factor to account for students in year-round schools on scheduled break at the time of the NAEP assessment and thus not available to be included in the sample. The within-school student weight (WINSCHWTjs) is the inverse of the student sampling rate in the school. The subject spiral adjustment factor (SUBJFACjsk) adjusts the student weight to account for the spiral pattern used in distributing mathematics, reading, science, or writing (WCBA) booklets to the students. The subject factor varies by grade, subject, school type (public/private), jurisdiction (states participating in the science assessment/states declining to participate in the science assessment), and it is equal to the inverse of the booklet proportions (mathematics, reading, science, or WCBA) in the overall spiral for a specific sample. For cooperating substitutes of nonresponding original sampled schools, the substitution adjustment factor (SUBADJjs) is equal to the ratio of the estimated grade enrollment for the original sampled school to the estimated grade enrollment for the substitute school. The student sample from the substitute school then "represents" the set of grade-eligible students from the original sampled school. The year-round adjustment factor (YRRND_AFjs) adjusts the student weight for students in year-round schools who do not attend school during the time of the assessment. This situation typically arises in overcrowded schools. School administrators in year-round schools randomly assign students to portions of the year in which they attend school and portions of the year in which they do not attend. At the time of assessment, a certain percentage of students (designated as OFFjs) do not attend school and thus cannot be assessed. The YRRND_AFjs for a school is calculated as 1/(1-OFFjs/100). http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_base_stud.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 5 7/22/2016 2:46 PM 6 of 60 School and Student Nonresponse Weight Adjustments for the 2011 Assessment Nonresponse is unavoidable in any voluntary survey of a human population. Nonresponse leads to School Nonresponse Weight the loss of sample data that must be compensated for in the weights of the responding sample Adjustment members. This differs from ineligibility, for which no adjustments are necessary. The purpose of the nonresponse adjustments is to reduce the mean square error of survey estimates. While the Student Nonresponse Weight nonresponse adjustment reduces the bias from the loss of sample, it also increases variability Adjustment among the survey weights leading to increased variances of the sample estimates. However, it is presumed that the reduction in bias more than compensates for the increase in the variance, thereby reducing the mean square error and thus improving the accuracy of survey estimates. Nonresponse adjustments are made in the NAEP surveys at both the school and the student levels: the responding (original and substitute) schools receive a weighting adjustment to compensate for nonresponding schools, and responding students receive a weighting adjustment to compensate for nonresponding students. The paradigm used for nonresponse adjustment in NAEP is the quasi-randomization approach (Oh and Scheuren 1983). In this approach, school response cells are based on characteristics of schools known to be related to both response propensity and achievement level, such as the locale type (e.g., large principal city of a metropolitan area) of the school. Likewise, student response cells are based on characteristics of the schools containing the students and student characteristics, which are known to be related to both response propensity and achievement level, such as student race/ethnicity, gender, and age. Under this approach, sample members are assigned to mutually exclusive and exhaustive response cells based on predetermined characteristics. A nonresponse adjustment factor is calculated for each cell as the ratio of the sum of adjusted base weights for all eligible units to the sum of adjusted base weights for all responding units. The nonresponse adjustment factor is then applied to the base weight of each responding unit. In this way, the weights of responding units in the cell are "weighted up" to represent the full set of responding and nonresponding units in the response cell. The quasi-randomization paradigm views nonresponse as another stage of sampling. Within each nonresponse cell, the paradigm assumes that the responding sample units are a simple random sample from the total set of all sample units. If this model is valid, then the use of the quasi-randomization weighting adjustment will eliminate any nonresponse bias. Even if this model is not valid, the weighting adjustments will eliminate bias if the achievement scores are homogeneous within the response cells (i.e., bias is eliminated if there is homogeneity either in response propensity or in achievement levels). See, for example, chapter 4 of Little and Rubin (1987). http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 6 7/22/2016 2:46 PM 7 of 60 School Nonresponse Weight Adjustment for the 2011 Assessment The school nonresponse adjustment procedure inflates the weights of cooperating schools to account for eligible noncooperating schools for which no substitute schools participated. The adjustments are computed within nonresponse cells and are based on the assumption that the cooperating and noncooperating schools within the same cell are more similar to each other than to schools from different cells. School nonresponse adjustments were carried out separately by sample; that is, by grade (4, 8, 12); school type (public, private); and assessment subject (reading, mathematics, science, writing [WCBA]). Development of Initial School Nonresponse Cells Development of Final School Nonresponse Cells School Nonresponse Adjustment Factor Calculation http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_schl.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 7 7/22/2016 2:46 PM 8 of 60 Development of Final School Nonresponse Cells for the 2011 Assessment Limits were placed on the magnitude of cell sizes and adjustment factors to prevent unstable nonresponse adjustments and unacceptably large nonresponse factors. All initial weighting cells with fewer than six cooperating schools or adjustment factors greater than 3.0 for the full sample weight were collapsed with suitable adjacent cells. Simultaneously, all initial weighting cells for any replicate with fewer than four cooperating schools or adjustment factors greater than the maximum of 3.0 or two times the full sample nonresponse adjustment factor were collapsed with suitable adjacent cells. Initial weighting cells were generally collapsed in reverse order of the cell structure; that is, starting at the bottom of the nesting structure and working up toward the top level of the nesting structure. Public School Samples for Reading, Mathematics, and Science at Grades 4 and 8 For the public school samples, cells with the most similar race/ethnicity classification within a given jurisdiction/Trial Urban District Assessment (TUDA) district and urbanicity (urban-centric locale) stratum were collapsed first. If further collapsing was required after all levels of race/ethnicity strata were collapsed, cells with the most similar urbanicity strata were combined next. Cells were never permitted to be collapsed across jurisdiction or TUDA district. Private School Samples for Reading, Mathematics, and Science at Grades 4 and 8 For the private school samples, cells with the most similar race/ethnicity classification within a given affiliation, census division, and urbanicity stratum were collapsed first. If further collapsing was required after all levels of race/ethnicity strata were collapsed, cells with the most similar urbanicity classification were combined. Any further collapsing occurred across census division strata but never across affiliation. Public School Samples for WCBA at Grades 8 and 12 For the public school samples, cells with similar high percentage Black/Hispanic status within a given census region and urbanicity stratum were collapsed first. If further collapsing was required, cells with the most similar urbanicity strata within a given census region were combined next. No further collapsing occurred after all levels of urbanicity strata were collapsed. That is, collapsing never occurred across census region. Private School Samples for WCBA at Grades 8 and 12 For the private school samples, if collapsing was necessary, all census region cells within a given affiliation were collapsed. However, collapsing never occurred across affiliation. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_schl_final.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 8 7/22/2016 2:46 PM 9 of 60 Development of Initial School Nonresponse Cells for the 2011 Assessment The cells for nonresponse adjustments are generally functions of the school sampling strata for the individual samples. School sampling strata usually differ by assessment subject, grade, and school type (public or private). Assessment subjects that are administered together by way of spiraling have the same school samples and stratification schemes. Subjects that are not spiraled with any other subjects have their own separate school sample. In NAEP 2011, the reading, mathematics, and science assessments were spiraled together, but writing (WCBA) was not spiraled with any other subject. The initial nonresponse cells for the various NAEP 2011 samples are described below. Public School Samples for Reading, Mathematics, and Science at Grades 4 and 8 For these samples, initial weighting cells were formed within each jurisdiction using the following nesting cell structure: Trial Urban District Assessment (TUDA) district vs. the balance of the state for states with TUDA districts; urbanicity (urban-centric locale) stratum; and race/ethnicity classification, or achievement level, or median income, or grade enrollment. In general, the nonresponse cell structure used minority stratum as the lowest level variable. However, where there was only one race/ethnicity category within a particular urbanicity stratum, categorized achievement or median income data were used instead. Private School Samples for Reading, Mathematics, and Science at Grades 4 and 8 The initial weighting cells for these samples were formed within each grade using the following nesting cell structure: affiliation; census division stratum; urbanicity stratum; and race/ethnicity classification. Public School Samples for WCBA at Grades 8 and 12 The initial weighting cells for these samples were formed within each subject and grade using the following nesting cell structure: census region; urban-centric locale; and indicator of high percentage of Black/Hispanic students within school. Private School Samples for WCBA at Grades 8 and 12 The initial weighting cells for these samples were formed within each subject and grade using the following nesting cell structure: affiliation; and census region. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_schl_initial.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 9 7/22/2016 2:46 PM 10 of 60 School Nonresponse Adjustment Factor Calculation for the 2011 Assessment In each final school nonresponse adjustment cell c, the school nonresponse adjustment factor SCH_NRAFc was computed as follows: where Sc is the set of all eligible sampled schools (cooperating original and substitute schools and refusing original schools with noncooperating or no assigned substitute) in cell c, Rc is the set of all cooperating schools within Sc, SCH_BWTs is the school base weight, SCH_TRIMs is the school-level weight trimming factor, SCHSESWTs is the school-level session assignment weight, and Xs is the estimated grade enrollment corresponding to the original sampled school. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_schl_factor.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 10 7/22/2016 2:46 PM 11 of 60 Student Nonresponse Weight Adjustment for the 2011 Assessment The student nonresponse adjustment procedure inflates the weights of assessed students to account for eligible sampled students who did not participate in the assessment. These inflation factors offset the loss of data associated with absent students. The adjustments are computed within nonresponse cells and are based on the assumption that the assessed and absent students within the same cell are more similar to one another than to students from different cells. Like its counterpart at the school level, the student nonresponse adjustment is intended to reduce the mean square error and thus improve the accuracy of NAEP assessment estimates. Also, like its counterpart at the school level, student nonresponse adjustments were carried out separately by sample; that is, by Development of Initial Student Nonresponse Cells Development of Final Student Nonresponse Cells Student Nonresponse Adjustment Factor Calculation grade (4, 8, 12); school type (public, private); and assessment subject (reading, mathematics, science, writing [WCBA]). http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_stud.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 11 7/22/2016 2:46 PM 12 of 60 Development of Final Student Nonresponse Cells for the 2011 Assessment Similar to the school nonresponse adjustment, cell and adjustment factor size constraints are in place to prevent unstable nonresponse adjustments or unacceptably large adjustment factors. All initial weighting cells with either fewer than 20 participating students or adjustment factors greater than 2.0 for the full sample weight were collapsed with suitable adjacent cells. Simultaneously, all initial weighting cells for any replicate with either fewer than 15 participating students or an adjustment factor greater than the maximum of 2.0 or 1.5 times the full sample nonresponse adjustment factor were collapsed with suitable adjacent cells. Initial weighting cells were generally collapsed in reverse order of the cell structure; that is, starting at the bottom of the nesting structure and working up toward the top level of the nesting structure. Race/ethnicity cells within SD/ELL group, school nonresponse cell, age, and gender classes were collapsed first. If further collapsing was required after collapsing all race/ethnicity classes, cells were next combined across gender, then age, and finally school nonresponse cells. Cells are never collapsed across SD/ELL for any sample. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_stud_final.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 12 7/22/2016 2:46 PM 13 of 60 Development of Initial Student Nonresponse Cells for the 2011 Assessment Initial student nonresponse cells are generally created within each sample as defined by grade, school type (public, private), and assessment subject. However, when subjects are administered together by way of spiraling, the initial student nonresponse cells are created across the subjects in the same spiral. The rationale behind this decision is that spiraled subjects are in the same schools and the likelihood of whether an eligible student participates in an assessment is more related to its school than the subject of the assessment booklet. In NAEP 2011, the reading, mathematics, and science assessments were spiraled together, but writing (WCBA) was not spiraled with any other subject. The initial student nonresponse cells for the various NAEP 2011 samples are described below. Nonresponse adjustment procedures are not applied to excluded students because they are not required to complete an assessment. Public School Samples for Reading, Mathematics, and Science at Grades 4 and 8 The initial student nonresponse cells for these samples were defined within grade, jurisdiction, and Trial Urban District Assessment (TUDA) district using the following nesting cell structure: students with disabilities (SD)/English language learners (ELL) by subject; school nonresponse cell; age1 (classified into "older" student and "modal age or younger" student); gender; and race/ethnicity. The highest level variable in the cell structure separates students who were classified either as having disabilities (SD) or as English language learners (ELL) from those who are neither, since SD or ELL students tend to score lower on assessment tests than non-SD/non-ELL students. In addition, the students in the SD or ELL groups are further broken down by subject, since rules for excluding students from the assessment differ by subject. Non-SD and non-ELL students are not broken down by subject, since the exclusion rules do not apply to them. Private School Samples for Reading, Mathematics, and Science at Grades 4 and 8 The initial weighting cells for these private school samples were formed hierarchically within grade as follows: SD/ELL; school nonresponse cell; age1 (classified into "older" student and "modal age or younger" student); gender; and race/ethnicity. Although exclusion rules differ by subject, there were not enough SD or ELL private school students to break out by subject as was done for the public schools. Public School Samples for WCBA at Grades 8 and 12 The initial weighting cells for these samples were formed hierarchically within grade using the following cell structure: SD/ELL; school nonresponse cell; age1 (classified into "older" student and "modal age or younger" student); gender; and race/ethnicity. Private School Samples for WCBA at Grades 8 and 12 The initial weighting cells for these samples were formed hierarchically within grade using the following cell structure: SD/ELL; NAEP 2018-2019 OMB Clearance: Appendix B Page 13 7/22/2016 2:46 PM 14 of 60 school nonresponse cell; age1 (classified into "older" student and "modal age or younger" student); gender; and race/ethnicity. 1 Older students are those born before October 1, 2000, for grade 4; October 1, 1996, for grade 8; and October 1, 1992, for grade 12. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_stud_initial.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 14 7/22/2016 2:46 PM 15 of 60 Student Nonresponse Adjustment Factor Calculation for the 2011 Assessment In each final student nonresponse adjustment cell c for a given sample, the student nonresponse adjustment factor STU_NRAFc was computed as follows: where Sc is the set of all eligible sampled students in cell c for a given sample, Rc is the set of all assessed students within Sc, STU_BWTk is the student base weight for a given student k, SCH_TRIMk is the school-level weight trimming factor for the school associated with student k, SCH_NRAFk is the school-level nonresponse adjustment factor for the school associated with student k, and SUBJFACk is the subject factor for a given student k. The student weight used in the calculation above is the adjusted student base weight, without regard to subject, adjusted for school weight trimming and school nonresponse. Nonresponse adjustment procedures are not applied to excluded students because they are not required to complete an assessment. In effect, excluded students were placed in a separate nonresponse cell by themselves and all received an adjustment factor of 1. While excluded students are not included in the analysis of the NAEP scores, weights are provided for excluded students in order to estimate the size of this group and its population characteristics. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_stud_factor.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 15 7/22/2016 2:46 PM 16 of 60 School and Student Weight Trimming Adjustments for the 2011 Assessment Trimming of School Weight trimming is an adjustment procedure that involves detecting and reducing extremely large Base Weights weights. "Extremely large weights" generally refer to large sampling weights that were not anticipated in the design of the sample. Unusually large weights are likely to produce large sampling variances for Trimming of Student statistics of interest, especially when the large weights are associated with sample cases reflective of rare Weights or atypical characteristics. To reduce the impact of these large weights on variances, weight reduction methods are typically employed. The goal of employing weight reduction methods is to reduce the mean square error of survey estimates. While the trimming of large weights reduces variances, it also introduces some bias. However, it is presumed that the reduction in the variances more than compensates for the increase in the bias, thereby reducing the mean square error and thus improving the accuracy of survey estimates (Potter 1988). NAEP employs weight trimming at both the school and student levels. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_trimming_adj.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 16 7/22/2016 2:46 PM 17 of 60 Trimming of School Base Weights for the 2011 Assessment Large school weights can occur for schools selected from the NAEP new-school sampling frame and for private schools. New schools that are eligible for weight trimming are schools with a disproportionately large student enrollment in a particular grade from a school district that was selected with a low probability of selection. The school base weights for such schools may be large relative to what they would have been if they had been selected as part of the original sample. To detect extremely large weights among new schools, a comparison was made between a new school's school base weight and its ideal weight (i.e., the weight that would have resulted had the school been selected from the original school sampling frame). If the school base weight was more than three times the ideal weight, a trimming factor was calculated for that school that scaled the base weight back to three times the ideal weight. The calculation of the school-level trimming factor for a new school s is expressed in the following formula: where EXP_WTs is the ideal base weight the school would have received if it had been on the NAEP public school sampling frame, and SCH_BWTs is the actual school base weight the school received as a sampled school from the new school frame. Twenty-one (21) new schools had their weights trimmed: seven at grade 4, and fourteen at grade 8. Private schools eligible for weight trimming were Private School Universe Survey (PSS) nonrespondents who were found subsequently to have either larger enrollments than assumed at the time of sampling, or an atypical probability of selection given their affiliation, the latter being unknown at the time of sampling. For private school s, the formula for computing the school-level weight trimming factor SCH_TRIMs is identical to that used for new schools. For private schools, EXP_WTs is the ideal base weight the school would have received if it had been on the NAEP private school sampling frame with accurate enrollment and known affiliation, and SCH_BWTs is the actual school base weight the school received as a sampled private school. No private schools had their weights trimmed. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_base_schtrim.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 17 7/22/2016 2:46 PM 18 of 60 Trimming of Student Weights for the 2011 Assessment Large student weights generally come from compounding nonresponse adjustments at the school and student levels with artificially low school selection probabilities, which can result from inaccurate enrollment data on the school frame used to define the school size measure. Even though measures are in place to limit the number and size of excessively large weights—such as the implementation of adjustment factor size constraints in both the school and student nonresponse procedures and the use of the school trimming procedure—large student weights can occur due to compounding effects of the various weighting components. The student weight trimming procedure uses a multiple median rule to detect excessively large student weights. Any student weight within a given trimming group greater than a specified multiple of the median weight value of the given trimming group has its weight scaled back to that threshold. Student weight trimming was implemented separately by grade, school type (public or private), and subject. The multiples used were 3.5 for public school trimming groups and 4.5 for private school trimming groups. Trimming groups were defined by jurisdiction and Trial Urban District Assessment (TUDA) districts for the public school samples for reading and mathematics at grades 4 and 8, and science at grade 8; by the nation for the public school and private school samples for writing [WCBA] at grades 8 and 12; and by affiliation (Catholic, Conservative Christian, Lutheran, and Other private) for the private school samples for reading, mathematics, and science at grades 4 and 8. The procedure computes the median of the nonresponse-adjusted student weights in the trimming group g for a given grade and subject sample. Any student k with a weight more than M times the median received a trimming factor calculated as follows: where M is the trimming multiple, MEDIANg is the median of nonresponse-adjusted student weights in trimming group g, and STUWGTgk is the weight after student nonresponse adjustment for student k in trimming group g. In the 2011 assessment, relatively few students had weights considered excessively large. Out of the approximately 987,000 students included in the combined 2011 assessment samples, 710 students had their weights trimmed. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_studtrim.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 18 7/22/2016 2:46 PM 19 of 60 Student Weight Raking Adjustment for the 2011 Assessment Weighted estimates of population totals for student-level subgroups for a given Development of Final Raking Dimensions grade will vary across subjects even though the student samples for each subject generally come from the same schools. These differences are the result of Raking Adjustment Control Totals sampling error associated with the random assignment of subjects to students through a process known as spiraling. For state assessments in particular, any Raking Adjustment Factor Calculation difference in demographic estimates between subjects, no matter how small, may raise concerns about data quality. To remove these random differences and potential data quality concerns, a new step was added to the NAEP weighting procedure starting in 2009. The new step adjusts the student weights in such a way that the weighted sums of population totals for specific subgroups are the same across all subjects. The new weighting step was implemented using a raking procedure and applied only to state-level assessments. Raking is a weighting procedure based on the iterative proportional fitting process developed by Deming and Stephan (1940) and involves simultaneous ratio adjustments to two or more marginal distributions of population totals. Each set of marginal population totals is known as a dimension, and each population total in a dimension is referred to as a control total. Raking is carried out in a sequence of adjustments. Sampling weights are adjusted to one marginal distribution and then to the second marginal distribution, and so on. One cycle of sequential adjustments to the marginal distributions is called an iteration. The procedure is repeated until convergence is achieved. The criterion for convergence can be specified either as the maximum number of iterations or an absolute difference (or relative absolute difference) from the marginal population totals. More discussion on raking can be found in Oh and Scheuren (1987). For NAEP 2011, the student raking adjustment was carried out separately in each state for the reading and mathematics public school samples at grades 4 and 8, and science public school samples at grade 8. The dimensions used in the raking process were National School Lunch Program (NSLP) eligibility, race/ethnicity, SD/ELL status, and gender. The control totals for these dimensions were obtained from the NAEP student sample weights of the reading, mathematics, and science samples combined. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_stud_raking.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 19 7/22/2016 2:46 PM 20 of 60 Development of Final Raking Dimensions for the 2011 Assessment The raking procedure involved four dimensions. The variables used to define the dimensions are listed below along with the categories making up the initial raking cells for each dimension. National School Lunch Program (NSLP) eligibility 1. Eligible for free or reduced-price lunch 2. Otherwise Race/Ethnicity 1. White, not Hispanic 2. Black, not Hispanic 3. Hispanic 4. Asian 5. American Indian/Alaska Native 6. Native Hawaiian/Pacific Islander 7. Two or More Races SD/ELL status 1. SD, but not ELL 2. ELL, but not SD 3. SD and ELL 4. Neither SD nor ELL Gender 1. Male 2. Female In states containing districts that participated in TUDA at grades 4 and 8, the initial cells were created separately for each TUDA district and the balance of the state. Similar to the procedure used for school and student nonresponse adjustments, limits were placed on the magnitude of the cell sizes and adjustment factors to prevent unstable raking adjustments that could have resulted in unacceptably large or small adjustment factors. Levels of a dimension were combined whenever there were fewer than 30 assessed or excluded students (20 for any of the replicates) in a category, if the smallest adjustment was less than 0.5, or if the largest adjustment was greater than 2 for the full sample or for any replicate. If collapsing was necessary for the race/ethnicity dimension, the following groups were combined first: American Indian/Alaska Native with Two or More Races; Asian with the Pacific Islander; and Black, not Hispanic with Hispanic. If further collapsing was necessary, the five categories American Indian/Alaska Native; Two or More Races; Asian; Native Hawaiian/Pacific Islander; and White, not Hispanic were combined. In some instances, all seven categories had to be collapsed. If collapsing was necessary for the SD/ELL dimension, the SD/not ELL and SD/ELL categories were combined first, followed by ELL/not SD if further collapsing was necessary. In some instances, all four categories had to be collapsed. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_stud_raking_final_dim.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 20 7/22/2016 2:46 PM 21 of 60 Raking Adjustment Control Totals for the 2011 Assessment The control totals used in the raking procedure for NAEP 2011 grades 4, 8, and 12 were estimates of the student population derived from the set of assessed and excluded students pooled across subjects. The control totals for category c within dimension d were computed as follows: where Rc(d) is the set of all assessed students in category c of dimension d, Ec(d) is the set of all excluded students in category c of dimension d, STU_BWTk is the student base weight for a given student k, SCH_TRIMk is the school-level weight trimming factor for the school associated with student k, SCH_NRAFk is the school-level nonresponse adjustment factor for the school associated with student k, STU_NRAFk is the student-level nonresponse adjustment factor for student k, and SUBJFACk is the subject factor for student k. The student weight used in the calculation of the control totals above is the adjusted student base weight, without regard to subject, adjusted for school weight trimming, school nonresponse, and student nonresponse. Control totals were computed for the full sample and for each replicate independently. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_stud_raking_ctrl_tots.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 21 7/22/2016 2:46 PM 22 of 60 Raking Adjustment Factor Calculation for the 2011 Assessment For assessed and excluded students in a given subject, the raking adjustment factor STU_RAKEk was computed as follows: First, the weight for student k was intialized as follows: where STU_BWTk is the student base weight for a given student k, SCH_TRIMk is the school-level weight trimming factor for the school associated with student k, SCH_NRAFk is the school-level nonresponse adjustment factor for the school associated with student k, STU_NRAFk is the student-level nonresponse adjustment factor for student k, and SUBJFACk is the subject factor for student k. Then, the sequence of weights for the first iteration was calculated as follows for student k in category c of dimension d: For dimension 1: For dimension 2: For dimension 3: For dimension 4: where Rc(d) is the set of all assessed students in category c of dimension d, Ec(d) is the set of all excluded students in category c of dimension d, and Totalc(d) is the control total for category c of dimension d. The process is said to converge if the maximum difference between the sum of adjusted weights and the control totals is 1.0 for each NAEP 2018-2019 OMB Clearance: Appendix B Page 22 7/22/2016 2:46 PM 23 of 60 category in each dimension. If after the sequence of adjustments the maximum difference was greater than 1.0, the process continues to the next iteration, cycling back to the first dimension with the initial weight for student k equalling STUSAWTkadj(4) from the previous iteration. The process continued until convergence was reached. Once the process converged, the adjustment factor was computed as follows: where STUSAWTk is the weight for student k after convergence. The process was done independently for the full sample and for each replicate. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_stud_raking_factor_cal.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 24 7/22/2016 2:46 PM 24 of 60 Computation of Replicate Weights for the 2011 Assessment In addition to the full-sample weight, a set of 62 replicate weights was provided for each student. These replicate weights are used in calculating the sampling variance of estimates obtained from the data, using the jackknife repeated replication method. The method of deriving these weights was aimed at reflecting the features of the sample design appropriately for each sample, so that when the jackknife variance estimation procedure is implemented, approximately unbiased estimates of sampling variance are obtained. This section gives the specifics for generating the replicate weights for the 2011 assessment samples. The theory that underlies the jackknife variance estimators used in NAEP studies is discussed in the section Replicate Variance Estimation. Defining Variance Strata and Forming Replicates Computing School-Level Replicate Factors Computing Student-Level Replicate Factors Replicate Variance Estimation In general, the process of creating jackknife replicate weights takes place at both the school and student level. The precise implementation differs between those samples that involve the selection of Primary Sampling Units (PSUs) and those where the school is the first stage of sampling. The procedure for this second kind of sample also differed in 2011 from all previous NAEP assessments. The change that was implemented permitted the introduction of a finite population correction factor at the school sampling stage, developed by Rizzo and Rust (2011). In past assessments this adjustment factor has always been implicitly assumed equal to 1.0, resulting in some overestimation of the sampling variance. For each sample, the calculation of replicate weighting factors at the school level was conducted in a series of steps. First, each school was assigned to one of 62 variance estimation strata. Then, a random subset of schools in each variance estimation stratum was assigned a replicate factor of between 0 and 1. Next, the remaining subset of schools in the same variance stratum was assigned a complementary replicate factor greater than 1. All schools in the other variance estimation strata were assigned a replicate factor of exactly 1. This process was repeated for each of the 62 variance estimation stratum so that 62 distinct replicate factors were assigned to each school in the sample. This process was then repeated at the student level. Here, each individual sampled student was assigned to one of 62 variance estimation strata, and 62 replicate factors with values either between 0 and 1, greater than 1, or exactly equal to 1 were assigned to each student. For example, consider a single hypothetical student. For replicate 37, that student’s student replicate factor might be 0.8, while for the school to which the student belongs, for replicate 37, the school replicate factor might be 1.6. Of course, for a given student, for most replicates, either the student replicate factor, the school replicate factor, or (usually) both, is equal to 1.0. In the case of PSU-based samples, the replication procedure was only carried out at the school level. Conceptually, one can include this process under the framework of replication at both school and student levels but where the replicate factors at the student level are equal to 1.0 for every replicate for every student. A replicate weight was calculated for each student, for each of the 62 replicates, using weighting procedures similar to those used for the full-sample weight. Each replicate weight contains the school and student replicate factors described above. By repeating the various weighting procedures on each set of replicates, the impact of these procedures on the sampling variance of an estimate is appropriately reflected in the variance estimate. Each of the 62 replicate weights for student k in school s in stratum j can be expressed as follows: where STU_BWTjsk is the student base weight; SCH_REPFACjs(r) is the school-level replicate factor for replicate r; SCH_NRAFjs(r) is the school-level nonresponse adjustment factor for replicate r; STU_REPFACjsk(r) is the student-level replicate factor for replicate r; STU_NRAFjsk(r) is the student-level nonresponse adjustment factor for replicate r; SCH_TRIMjs is the school-level weight trimming adjustment factor; NAEP 2018-2019 OMB Clearance: Appendix B Page 25 7/22/2016 2:46 PM 25 of 60 STU_TRIMjsk is the student-level weight trimming adjustment factor; and STU_RAKEjsk(r) is the student-level raking adjustment factor for replicate r. Specific school and student nonresponse and student-level raking adjustment factors were calculated separately for each replicate, thus the use of the index (r), and applied to the replicate student base weights. Computing separate nonresponse and raking adjustment factors for each replicate allows resulting variances from the use of the final student replicate weights to reflect components of variance due to these various weight adjustments. School and student weight trimming adjustments were not replicated, that is, not calculated separately for each replicate. Instead, each replicate used the school and student trimming adjustment factors derived for the full sample. Statistical theory for replicating trimming adjustments under the jackknife approach has not been developed in the literature. Due to the absence of a statistical framework, and since relatively few school and student weights in NAEP require trimming, the weight trimming adjustments were not replicated. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_repwts.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 26 7/22/2016 2:46 PM 26 of 60 Computing School-Level Replicate Factors for the 2011 Assessment The calculation of school-level replicate factors for an assessment depended on the variance replication procedure, with or without finite population corrections for the first stage of sampling. Mathematics, Reading, and Science Assessments The replicate variance estimation approach for the mathematics, reading, and science assessments involved finite population corrections at the school level. The calculation of school-level replicate factors for these assessments depended upon whether or not a school was selected with certainty. For certainty schools, the school-level replicate factors for all replicates are set to unity – this is true regardless of whether or not the variance replication method uses finite population corrections – since certainty schools are not subject to sampling variability. Alternatively, one can view the finite population correction factor for such schools as being equal to zero. Thus, for each certainty school in a given assessment, the school-level replicate factor for each of the 62 replicates (r = 1, ..., 62) was assigned as follows: where SCH_REPFACjs(r) is the school-level replicate factor for school s in primary stratum j for the r-th replicate. For noncertainty schools, where variance strata were formed by grouping schools into pairs or triplets, school-level replicate factors were calculated for each of the 62 replicates based on this grouping. For schools in variance strata comprising pairs of schools, the school-level replicate factors, SCH_REPFACjs(r), r = 1,..., 62, were calculated as follows: where min(πj1, πj2) is the smallest school probability between the two schools comprising Rjr, Rjr is the set of schools within the r-th variance stratum for primary stratum j, and Ujs is the variance unit (1 or 2) for school s in primary stratum j. For noncertainty schools in variance strata comprising three schools, the school-level replicate factors, SCH_REPFACjs(r), r = 1,..., 62, were calculated as follows: For school s from primary stratum j, variance stratum r, while for r' = r + 31 (mod 62): NAEP 2018-2019 OMB Clearance: Appendix B Page 27 7/22/2016 2:46 PM 27 of 60 and for all other r* other than r and r': where min(πj1, πj2,πj3) is the smallest school probability among the three schools comprising Rjr, Rjr is the set of schools within the r-th variance stratum for primary stratum j, and Ujs is the variance unit (1, 2, or 3) for school s in primary stratum j. Computer-Based Writing Assessments The replicate variance estimation approach for the computer-based writing assessments did not involve school-level finite population corrections. As described in the section Defining Variance Strata and Forming Replicates, variance strata were defined by grouping first-stage units (schools or geographic PSUs) into pairs or triplets. The calculation of the school-level replicate factors for each of the 62 replicates was based on this grouping. For schools in variance strata comprising pairs of first-stage units, the school-level replicate factors, SCH_REPFACjs(r), r = 1,..., 62, were calculated as follows: where Rjr is the set of schools within the r-th variance stratum for primary stratum j, and Ujs is the variance unit (1 or 2) for school s in primary stratum j. For schools in variance strata comprising three first-stage units, school-level replicate factors for school were calculated as follows: For school s in primary stratum j, variance stratum r, while for r' = r + 31 (mod 62): NAEP 2018-2019 OMB Clearance: Appendix B Page 28 7/22/2016 2:46 PM 28 of 60 and for all other r* other than r and r': where Rjr is the set of schools within the r-th variance stratum for primary stratum j, and Ujsis the variance unit (1, 2, or 3) for school s in primary stratum j. In primary strata with fewer than 62 variance strata, the replicate weights for the “unused” variance strata (the remaining ones up to 62) for these schools were set equal to the school base weight (so that those replicates contribute nothing to the variance estimate). http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_repwts_schl.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 29 7/22/2016 2:46 PM 29 of 60 Computing Student-Level Replicate Factors for the 2011 Assessment The calculation of student-level replicate factors for an assessment depended on whether the variance replication procedure incorporated finite population corrections at the first stage of sampling. Mathematics, Reading, and Science Assessments For the mathematics, reading, and science assessments, which involved school-level finite population corrections, the student-level replication factors were calculated the same way regardless of whether or not the student was in a certainty school. For students in student-level variance strata comprising pairs of students, the student-level replicate factors, STU_REPFACjsk(r), r = 1,..., 62, were calculated as follows: where πs is the probability of selection for school s, Rjsr is the set of students within the r-th variance stratum for school s in primary stratum j, and Ujsk is the variance unit (1 or 2) for student k in school s in stratum j. For students in variance strata comprising three students, the student-level replicate factors STU_REPFACjsk(r), r = 1,..., 62, were calculated as follows: while for r' = r + 31 (mod 62): and for all other r* other than r and r': STU_REPFACjsk(r*) = 1 where πs is the probability of selection for school s, Rjsr is the set of students within the r-th replicate stratum for school s in stratum j, and Ujsk is the variance unit (1, 2, or 3) for student k in school s in stratum j. Note, for students in certainty schools, where πs = 1, the student replicate factors are 2 and 0 in the case of pairs, and 1.5, 1.5, and 0 in the case of triples. Page 30 NAEP 2018-2019 OMB Clearance: Appendix B 7/22/2016 2:46 PM 30 of 60 Computer-Based Writing Assessments For the computer-based writing assessments, which did not involve first-stage finite population corrections and there were no certainty schools, replication at the student level was not relevant. As a consequence, the replicate factors for all replicates for every student in the computer-based writing assessments were set to unity. That is, the student-level replicate factors, STU_REPFACjsk(r), r = 1,..., 62, were calculated as follows: STU_REPFACjsk(r) = 1. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_repwts_stud.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 31 7/22/2016 2:46 PM 31 of 60 Defining Variance Strata and Forming Replicates for the 2011 Assessment In the NAEP 2011 assessment, replicates were formed separately for each sample indicated by grade (4, 8, 12), school type (public, private), and assessment subject (mathematics, reading, science, computer-based writing). To reflect the school-level finite population corrections in the variance estimators for the two-stage samples used for mathematics, reading, and science assessments, variance strata were formed at both the school and student levels for these samples. For the computer-based writing assessments, which did not use school-level finite population corrections in the variance estimators, variance strata were formed only at the first-stage level. The first step in forming replicates was to create variance strata in each primary stratum. In 2011, the mathematics, reading, and science assessments required formation of variance strata at both the school and student levels. The computer-based writing assessments required formation of variance strata only at the first-stage level. The next step was to sort the appropriate sampling unit (school or student) in the order of its selection within the primary stratum and then pair off into preliminary variance strata. Sorting sample units by their order of sample selection reflects the implicit stratification and systematic sampling features of the sample design. Within each primary stratum with an even number of sampling units, all of the preliminary variance strata consisted of pairs of sampling units. However, within primary strata with an odd number of sampling units, all but one variance strata consisted of pairs of sampling units, while the last one consisted of three sampling units. If there were more than 62 preliminary variance strata within a primary stratum, the preliminary variance strata were grouped to form 62 variance strata. This grouping effectively maximized the distance in the sort order between grouped preliminary variance strata. The first 62 preliminary variance strata, for example, were assigned to 62 different final variance strata in order (1 through 62), with the next 62 preliminary variance strata assigned to final variance strata 1 through 62, so that, for example, preliminary variance stratum 1, preliminary variance stratum 63, preliminary variance stratum 125 (if in fact there were that many), etc., were all assigned to the first final variance stratum. If, on the other hand, there were fewer than 62 preliminary variance strata within a primary stratum, then the number of final variance strata was set equal to the number of preliminary variance strata. For example, consider a primary stratum with 111 sampled units sorted in their order of selection. The first two units were in the first preliminary variance stratum; the next two units were in the second preliminary variance stratum, and so on, resulting in 54 preliminary variance strata with two sample units each (doublets). The last three sample units were in the 55th preliminary variance stratum (triplet). Since there are no more than 62 preliminary variance strata, these were also the final variance strata. Within each preliminary variance stratum containing a pair of sampling units, one sampling unit was randomly assigned as the first variance unit and the other as the second variance unit. Within each preliminary variance stratum containing three sampling units, the three first-stage units were randomly assigned variance units 1 through 3. Reading, Mathematics, and Science Assessments As described above, the mathematics, reading, and science assessments required variance strata at both the school and student level. At the school-level for these samples, formation of variance strata did not pertain to certainty schools, since they are not subject to sampling variability, but only to noncertainty schools. The primary stratum for noncertainty schools was the highest school-level sampling stratum variable listed below, and the order of selection was defined by sort order on the school sampling frame. Trial Urban District Assessment (TUDA) districts, remainder of states (for states with TUDAs), or entire states for the public school samples at grades 4 and 8; and Private school affiliation (Catholic, Lutheran, Conservative Christian, and Other private) for the private school samples at grades 4 and 8. At the student-level, all students were assigned to variance strata. The primary stratum was school, and the order of selection was defined by session number and position on the administration schedule. Computer-Based Writing Assessments As described above, variance strata for the computer-based writing assessments were formed at the first-stage sampling level and so differed by certainty and noncertainty PSUs. For noncertainty PSUs, the first-stage sampling units were PSUs, and the primary stratum was the combination of region and metropolitan status (MSA or non-MSA). For certainty PSUs, the first-stage sampling units were schools, and the primary stratum was school type (public or private). NAEP 2018-2019 OMB Clearance: Appendix B Page 32 7/22/2016 2:46 PM 32 of 60 For noncertainty PSUs, where only one PSU was selected per PSU stratum, variance strata were formed by pairing sampled PSUs with similar stratum characteristics within the same primary stratum (region by metropolitan status). This was accomplished by first sorting the 38 sampled PSUs by PSU stratum number and then grouping adjacent PSUs into 19 pairs. The values for a PSU stratum number reflect region and metropolitan status, as well as demographic and socioeconomic characteristics such as percentage of Black youth and percentage of children whose family income is below the poverty threshold. The formation of these 19 variance strata in this manner models a design of selecting two PSUs with probability proportional to size with replacement from each of 19 strata. For certainty PSUs, the first stage of sampling is at the school level, and the formation of variance strata must reflect the sampling of schools within the certainty PSUs. Variance strata were formed by sorting the sampled schools in the 29 certainty PSUs by their order of selection within a primary stratum (school type) so that the sort order reflected the implicit stratification (region, locality type, minority status, and student enrollment for public schools; and region, private school type, and student enrollment size for private schools) and systematic sampling features of the sample design. The first-stage units were then paired off into 43 preliminary variance strata. Within each primary stratum with an even number of first-stage units, all of the preliminary variance strata were pairs, and within primary strata with an odd number of first-stage units, one of the variance strata was a triplet (the last one), and all others were pairs. If there were more than 43 preliminary variance strata within a primary stratum, the preliminary variance strata were grouped to form 43 variance strata. This grouping effectively maximized the distance in the sort order between grouped preliminary variance strata. The first 43 preliminary variance strata, for example, were assigned to 43 different final variance strata in order (1 through 43), with the next 43 preliminary variance strata assigned to final variance strata 1 through 43, so that, for example, preliminary variance stratum 1, preliminary variance stratum 44, preliminary variance stratum 87 (if there were that many), etc., were all assigned to the first final variance stratum. The final variance strata for the schools in the certainty PSUs were 1 through 43. Within each pair of preliminary variance strata, one first-stage unit, designated at random, was assigned as the first variance unit and the other first-stage unit as the second variance unit. Within each triplet preliminary variance stratum, the three schools were randomly assigned variance units 1 through 3. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_repwts_strata.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 33 7/22/2016 2:46 PM 33 of 60 Replicate Variance Estimation for the 2011 Assessment Variances for NAEP assessment estimates are computed using the paired jackknife replicate variance procedure. This technique is applicable for common statistics, such as means and ratios, and differences between these for different subgroups, as well as for more complex statistics such as linear or logistic regression coefficients. In general, the paired jackknife replicate variance procedure involves initially pairing clusters of first-stage sampling units to form H variance strata (h = 1, 2, 3, ...,H) with two units per stratum. The first replicate is formed by assigning, to one unit at random from the first variance stratum, a replicate weighting factor of less than 1.0, while assigning the remaining unit a complementary replicate factor greater than 1.0, and assigning all other units from the other (H - 1) strata a replicate factor of 1.0. This procedure is carried out for each variance stratum resulting in H replicates, each of which provides an estimate of the population total. In general, this process is repeated for subsequent levels of sampling. In practice, this is not practicable for a design with three or more stages of sampling, and the marginal improvement in precision of the variance estimates would be negligible in all such cases in the NAEP setting. Thus in NAEP, when a two-stage design is used – sampling schools and then students – beginning in 2011 replication is carried out at both stages. (See Rizzo and Rust (2011) for a description of the methodology.) When a three-stage design is used, involving the selection of geographic Primary Sampling Units (PSUs), then schools, and then students, the replication procedure is only carried out at the first stage of sampling (the PSU stage for noncertainty PSUs, and the school stage within certainty PSUs). In this situation, the school and student variance components are correctly estimated, and the overstatement of the between-PSU variance component is relatively very small. The jackknife estimate of the variance for any given statistic is given by the following formula: where represents the full sample estimate of the given statistic, and represents the corresponding estimate for replicate h. Each replicate undergoes the same weighting procedure as the full sample so that the jackknife variance estimator reflects the contributions to or reductions in variance resulting from the various weighting adjustments. The NAEP jackknife variance estimator is based on 62 variance strata resulting in a set of 62 replicate weights assigned to each school and student. The basic idea of the paired jackknife variance estimator is to create the replicate weights so that use of the jackknife procedure results in an unbiased variance estimator for simple totals and means, which is also reasonably efficient (i.e., has a low variance as a variance estimator). The jackknife variance estimator will then produce a consistent (but not fully unbiased) estimate of variance for (sufficiently smooth) nonlinear functions of total and mean estimates such as ratios, regression coefficients, and so forth (Shao and Tu, 1995). The development below shows why the NAEP jackknife variance estimator returns an unbiased variance estimator for totals and means, which is the cornerstone to the asymptotic results for nonlinear estimators. See for example Rust (1985). This paper also discusses why this variance estimator is generally efficient (i.e., more reliable than alternative approaches requiring similar computational resources). The development is done for an estimate of a mean based on a simplified sample design that closely approximates the sample design for first-stage units used in the NAEP studies. The sample design is a stratified random sample with H strata with population weights Wh, stratum sample sizes nh, and stratum sample means . The population estimator and standard unbiased variance estimator are: NAEP 2018-2019 OMB Clearance: Appendix B Page 34 7/22/2016 2:46 PM 34 of 60 with The paired jackknife replicate variance estimator assigns one replicate h=1,…, H to each stratum, so that the number of replicates equals H. In NAEP, the replicates correspond generally to pairs and triplets (with the latter only being used if there are an odd number of sample units within a particular primary stratum generating replicate strata). For pairs, the process of generating replicates can be viewed as taking a simple random sample (J) of size nh/2 within the replicate stratum, and assigning an increased weight to the sampled elements, and a decreased weight to the unsampled elements. In certain applications, the increased weight is double the full sample weight, while the decreased weight is in fact equal to zero. In this simplified case, this assignment reduces to replacing with , the latter being the sample mean of the sampled nh/2 units. Then the replicate estimator corresponding to stratum r is The r-th term in the sum of squares for is thus: In stratified random sampling, when a sample of size nr/2 is drawn without replacement from a population of size nr,, the sampling variance is See for example Cochran (1977), Theorem 5.3, using nr, as the “population size,” nr/2 as the “sample size,” and sr2 as the “population variance” in the given formula. Thus, Taking the expectation over all of these stratified samples of size nr/2, it is found that In this sense, the jackknife variance estimator “gives back” the sample variance estimator for means and totals as desired under the theory. In cases where, rather than doubling the weight of one half of one variance stratum and assigning a zero weight to the other, the weight of one unit is multiplied by a replicate factor of (1+δ), while the other is multiplied by (1- δ), the result is that In this way, by setting δ equal to the square root of the finite population correction factor, the jackknife variance estimator is able to incorporate a finite population correction factor into the variance estimator. In practice, variance strata are also grouped to make sure that the number of replicates is not too large (the total number of variance strata is usually 62 for NAEP). The randomization from the original sample distribution guarantees that the sum of squares NAEP 2018-2019 OMB Clearance: Appendix B Page 35 7/22/2016 2:46 PM 35 of 60 contributed by each replicate will be close to the target expected value. For triples, the replicate factors are perturbed to something other than 1.0 for two different replicate factors, rather than just one as in the case of pairs. Again in the simple case where replicate factors that are less than 1 are all set to 0, with the replicate weight factors calculated as follows. For unit i in variance stratum r where weight wi is the full sample base weight. Furthermore, for r' = r + 31 (mod 62): And for all other values r*, other than r and r´,wi(r*) = 1. with for replicate r, where is the In the case of stratified random sampling, this formula reduces to replacing sample mean from a “2/3” sample of 2nr/3 units from the nr sample units in the replicate stratum, and replacing with for is the sample mean from another overlapping “2/3” sample of 2nr/3 units from the nrsample units in the replicate r', where replicate stratum. The r-th and r´-th replicates can be written as: From these formulas, expressions for the r-th and r´-th components of the jackknife variance estimator are obtained (ignoring other sums of squares from other grouped components attached to those replicates): These sums of squares have expectations as follows, using the general formula for sampling variances: NAEP 2018-2019 OMB Clearance: Appendix B Page 36 7/22/2016 2:46 PM 36 of 60 Thus, as desired again. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_repwts_appdx.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 37 7/22/2016 2:46 PM 37 of 60 Quality Control on Weighting Procedures for the 2011 Assessment Given the complexity of the weighting procedures utilized in NAEP, a range of quality control (QC) checks was conducted throughout the weighting process to identify potential problems with collected student-level demographic data or with specific weighting procedures. The QC processes included checks performed within each step of the weighting process; Main QC Findings of Interest Participation, Exclusion, and Accommodation Rates Nonresponse Bias Analyses checks performed across adjacent steps of the weighting process; review of participation, exclusion, and accommodation rates; checking demographic data of individual schools; comparisons with 2009 demographic data; and nonresponse bias analyses. To validate the weighting process, extensive tabulations of various school and student characteristics at different stages of the process were conducted. The school-level characteristics included in the tabulations were minority enrollment, median income (based on the school ZIP code area), and urban-centric locale. At the student level, the tabulations included race/ethnicity, gender, relative age, students with disability (SD) status, English language learners (ELL) status, and participation status in National School Lunch Program (NSLP). http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_qa.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 38 7/22/2016 2:46 PM 38 of 60 Main Quality Control Findings of Interest for the 2011 Assessment Final participation, exclusion, and accommodation rates are presented in quality control tables for each grade and subject by geographic domain and school type. School-level participation rates have been calculated according to National Center for Education Statistics (NCES) standards as they have been for previous assessments. School-level participation rates were below 85 percent for private schools at all three grades (4, 8, and 12), for public schools of the Bureau of Indian Education (BIE) for grades 4 and 8, and for Colorado for grade 8. Student-level participation rates were also below 85 percent for students in Detroit for grade 8 public schools. As required by NCES standards, nonresponse bias analyses were conducted on each reporting group falling below the 85 percent participation threshold. Final Participation, Exclusion, and Accommodation Rates Grade 4 Mathematics Grade 4 Reading Grade 8 Mathematics Grade 8 Reading Grade 8 Science Grade 8 Writing (WCBA) Grade 12 Writing (WCBA) http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_qa_findings.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 39 7/22/2016 2:46 PM 39 of 60 Participation, Exclusion, and Accommodation Rates for Grade 8 Mathematics for the 2011 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 8 mathematics assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 8 mathematics assessment, by school type and jurisdiction: 2011 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded Weighted percent of students excluded Weighted student participation rates (percent) after makeups 8,700 97.57 88.42 209,000 2.45 92.72 9.74 National Northeast all Midwest all South all West all 8,600 97.54 88.31 204,000 2.46 92.71 9.67 1,400 2,400 2,700 2,100 95.45 98.76 97.65 97.80 79.60 91.49 89.96 89.74 33,200 45,900 73,300 50,200 2.25 2.62 3.22 1.30 91.82 93.33 93.13 92.19 14.40 9.84 8.35 8.11 National public Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois Indiana 7,500 125 167 136 126 257 130 119 68 102 233 132 81 113 223 113 99.79 100.00 99.90 99.02 100.00 100.00 99.87 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 99.76 100.00 97.93 99.08 100.00 100.00 97.31 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 192,000 3,200 2,900 3,200 3,200 8,400 3,200 3,100 3,200 3,000 7,100 4,800 3,300 3,400 4,700 3,200 2.64 1.21 3.15 1.14 1.36 1.08 0.85 1.31 3.08 4.37 1.83 2.73 1.87 1.28 2.32 2.56 92.53 93.72 89.48 92.97 92.53 91.81 92.71 92.90 93.15 89.99 92.56 92.94 91.67 94.32 93.35 92.93 10.14 3.59 14.39 8.87 11.63 7.47 9.96 12.26 10.86 15.28 16.13 7.41 10.69 7.23 11.56 12.15 School type and jurisdiction All all1 Weighted percent of students accommodated 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. NAEP 2018-2019 OMB Clearance: Appendix B Page 40 7/22/2016 2:46 PM 40 of 60 School type and jurisdiction Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 138 148 156 163 143 176 154 180 168 121 136 200 169 100 96 116 136 174 159 209 194 149 144 168 61 116 261 123 236 125 124 108 140 117 179 100.00 100.00 100.00 100.00 100.00 99.05 99.46 100.00 100.00 100.00 100.00 99.86 100.00 99.70 100.00 100.00 99.09 99.08 100.00 99.99 100.00 100.00 99.10 100.00 100.00 100.00 100.00 100.00 99.09 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 98.82 98.47 100.00 100.00 100.00 100.00 98.41 100.00 97.35 100.00 100.00 99.40 99.67 100.00 99.47 100.00 100.00 99.26 100.00 100.00 100.00 100.00 100.00 99.63 100.00 100.00 100.00 100.00 100.00 100.00 3,000 3,200 4,600 3,100 3,100 4,300 4,500 4,900 3,500 3,100 3,000 3,000 3,000 3,300 3,100 3,000 4,000 4,700 5,000 2,700 4,300 3,000 3,400 4,500 3,100 3,200 3,500 3,300 9,100 3,500 2,400 3,200 3,700 3,200 4,100 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 1.44 1.33 3.33 1.44 1.55 6.32 4.03 3.60 2.11 1.07 1.35 1.58 3.59 3.07 1.75 4.21 1.96 1.38 1.84 4.26 5.02 9.82 1.43 2.43 1.26 3.76 1.72 3.77 5.16 2.72 1.12 2.87 1.66 1.51 2.00 93.13 93.02 93.50 92.63 91.83 92.30 91.65 92.84 93.04 93.56 93.71 89.95 93.50 93.73 90.87 92.20 91.30 91.04 91.83 94.66 92.54 92.29 93.12 91.60 92.06 93.57 94.31 91.46 93.67 91.14 93.96 93.47 91.82 93.34 92.92 13.93 9.08 7.96 12.76 14.44 6.55 14.99 7.81 8.65 6.19 10.32 9.34 9.13 8.62 13.94 13.88 9.83 18.35 12.36 8.97 9.68 4.19 10.54 13.12 13.48 7.84 7.29 7.85 5.11 8.43 14.82 9.27 10.17 9.24 14.18 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. Page 41 NAEP 2018-2019 OMB Clearance: Appendix B 7/22/2016 2:46 PM 41 of 60 School type and jurisdiction Wyoming BIE Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 108 116 100.00 83.16 100.00 84.68 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 2,500 1,100 1.25 1.85 92.42 91.35 10.81 11.45 DoDEA 2 Trial Urban (TUDA) Districts Albuquerque Atlanta Austin Baltimore City Boston Charlotte Chicago Cleveland Dallas Detroit Fresno Hillsborough Houston Jefferson County, KY Los Angeles Miami Milwaukee New York City Philadelphia San Diego District of Columbia (TUDA) 72 98.56 95.31 2,100 2.68 95.40 7.98 34 26 24 75 45 38 117 71 41 63 23 50 50 46 71 86 59 91 60 31 50 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 1,400 1,600 1,800 1,400 1,400 1,700 2,200 1,300 1,700 1,900 1,500 1,600 2,500 1,700 2,300 2,900 1,400 2,500 1,500 1,300 1,700 3.42 2.50 4.64 12.50 5.79 1.32 3.33 5.59 4.89 8.07 1.19 1.84 5.61 3.16 1.30 1.87 4.90 1.04 6.71 2.83 6.58 89.29 92.80 91.43 87.44 92.02 92.39 95.57 91.35 93.88 84.39 91.56 93.23 92.79 91.83 92.39 92.99 91.92 91.35 90.86 94.84 88.34 12.40 8.38 9.07 7.50 18.94 11.29 15.84 24.17 6.24 8.23 6.64 20.81 5.47 8.08 9.21 17.82 25.48 24.15 17.82 7.94 18.13 National private Catholic Non-Catholic private Lutheran Conservative Christian Other private 930 332 598 141 149 308 74.40 93.23 57.54 92.73 72.51 45.71 69.89 92.09 58.75 91.45 74.78 45.85 8,800 4,600 4,200 1,000 1,200 2,000 0.50 0.53 0.47 0.33 0.47 0.49 94.65 95.07 94.26 96.31 94.46 93.90 4.54 3.32 5.70 2.13 3.67 7.06 Puerto Rico 110 100.00 100.00 5,100 1.03 93.14 17.11 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. NAEP 2018-2019 OMB Clearance: Appendix B Page 42 7/22/2016 2:46 PM 42 of 60 http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_gr8math_2011.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 43 7/22/2016 2:46 PM 43 of 60 Participation, Exclusion, and Accommodation Rates for Grade 8 Writing (WCBA) for the 2011 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 8 writing [WCBA] assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 8 writing (WCBA) assessment, by school type and jurisdiction: 2011 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 1,000 97.27 87.29 1,000 97.27 178 204 400 265 National public National private Catholic Non-Catholic private School type and jurisdiction All all1 National Northeast all Midwest all South all West all Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 27,400 1.72 93.99 8.05 87.29 27,400 1.72 93.99 8.05 95.36 98.83 97.15 97.43 80.39 93.61 86.74 86.34 4,400 5,300 10,700 7,000 2.08 1.78 1.71 1.38 93.20 94.19 94.62 93.43 11.42 10.14 6.48 5.80 890 99.73 99.86 25,200 1.84 93.99 8.49 157 50 107 71.21 95.53 52.06 68.66 95.23 56.42 2,200 1,100 1,100 0.28 0.53 0.08 93.97 94.72 93.35 3.01 3.12 2.92 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_g8writing_2011.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 44 7/22/2016 2:46 PM 44 of 60 Participation, Exclusion, and Accommodation Rates for Grade 12 Writing (WCBA) for the 2011 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 12 writing [WCBA] assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 12 writing (WCBA) assessment, by school type and jurisdiction: 2011 School type and jurisdiction All National all1 Northeast all Midwest all South all West all National public National private Catholic Non-Catholic private Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 1,400 93.52 89.26 1,400 93.52 245 266 495 367 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 36,500 2.33 86.97 6.65 89.26 36,500 2.33 86.97 6.65 91.91 96.93 94.66 89.70 79.32 95.76 90.91 85.00 6,400 7,300 13,200 9,700 2.09 2.20 2.72 2.02 84.18 86.40 88.36 87.73 8.87 8.79 4.79 5.70 1,200 96.04 97.12 33,400 2.52 86.96 6.84 177 55 122 67.23 76.60 58.35 67.45 75.42 65.42 3,100 1,500 1,600 0.27 0.11 0.42 87.16 85.96 88.34 4.68 3.31 6.02 1 Includes national public, national private, Bureau of Indian Education, and Department of Defense Education Activity schools located in the United States. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Writing Assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_gr12writing_2011.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 45 7/22/2016 2:46 PM 45 of 60 Participation, Exclusion, and Accommodation Rates for Grade 4 Mathematics for the 2011 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 4 mathematics assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 4 mathematics assessment, by school type and jurisdiction: 2011 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded Weighted percent of students excluded Weighted student participation rates (percent) after makeups 9,400 97.44 91.77 242,000 2.05 94.53 11.62 National Northeast all Midwest all South all West all 9,200 97.41 91.65 237,000 2.06 94.53 11.51 1,600 2,400 2,900 2,300 95.19 97.76 97.64 98.27 84.70 91.40 93.31 94.56 37,800 51,900 84,200 60,400 1.78 1.82 2.65 1.54 94.25 94.37 94.73 94.58 15.43 11.48 10.67 10.16 National public Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois Indiana 8,200 117 202 127 123 292 124 116 109 153 235 174 118 137 193 117 99.83 98.95 100.00 99.03 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 99.86 99.76 100.00 99.25 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 225,000 3,300 3,100 4,200 3,900 10,100 4,000 3,400 3,900 2,400 8,000 6,000 3,800 4,000 5,600 3,900 2.22 1.16 2.81 1.02 1.00 1.56 1.14 1.27 3.59 5.24 1.59 1.65 1.76 1.21 2.26 2.15 94.44 95.00 92.61 94.29 94.87 95.27 92.20 93.37 94.12 94.54 94.51 94.48 93.35 95.34 93.33 94.69 12.12 4.17 17.86 15.32 14.07 7.07 14.32 15.55 11.92 14.27 18.87 10.35 11.27 8.51 12.84 14.06 School type and jurisdiction All all1 Weighted percent of students accommodated 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico.. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. NAEP 2018-2019 OMB Clearance: Appendix B Page 46 7/22/2016 2:46 PM 46 of 60 School type and jurisdiction Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 141 148 158 134 166 173 202 172 148 117 131 206 181 118 133 118 155 158 173 272 205 140 149 167 125 114 209 119 306 130 226 115 141 152 190 100.00 99.18 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 99.17 100.00 100.00 100.00 99.97 100.00 100.00 99.08 100.00 100.00 100.00 100.00 100.00 99.08 100.00 100.00 100.00 100.00 100.00 100.00 100.00 99.30 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 98.83 100.00 100.00 100.00 99.61 100.00 100.00 99.36 100.00 100.00 100.00 100.00 100.00 99.23 100.00 100.00 100.00 100.00 100.00 100.00 3,700 3,500 5,500 3,900 3,500 5,200 5,600 4,700 4,100 3,400 3,900 3,500 3,400 4,400 3,600 3,600 4,700 5,200 5,900 3,400 4,900 3,500 4,100 5,200 3,500 3,700 3,600 3,900 11,000 4,500 3,000 4,100 4,400 3,400 4,900 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 1.41 1.65 3.08 1.74 1.59 5.61 3.17 2.15 1.50 0.83 1.65 1.52 1.50 2.29 1.74 3.30 2.61 1.33 1.78 3.59 2.32 8.27 2.67 1.39 0.93 1.27 1.78 3.40 4.15 2.02 1.58 2.08 1.90 1.50 1.66 94.94 94.25 94.48 93.65 94.48 94.66 94.16 94.11 94.02 94.99 93.58 94.24 95.57 94.93 93.95 94.52 93.89 94.13 94.01 95.26 94.03 95.42 93.47 94.13 94.41 94.15 95.31 93.79 95.24 93.69 94.12 94.76 94.31 94.58 94.98 14.81 12.54 8.54 17.54 15.07 11.17 15.25 8.75 12.52 5.87 10.16 7.68 14.36 22.47 15.33 14.19 14.86 20.59 12.44 9.31 13.21 6.72 14.60 12.58 13.31 10.28 9.00 10.37 8.31 10.20 14.70 11.71 13.75 8.78 15.56 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico.. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. Page 47 NAEP 2018-2019 OMB Clearance: Appendix B 7/22/2016 2:46 PM 47 of 60 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded Wyoming BIE 202 135 100.00 83.26 100.00 84.85 DoDEA 2 Trial Urban (TUDA) Districts Albuquerque Atlanta Austin Baltimore City Boston Charlotte Chicago Cleveland Dallas Detroit Fresno Hillsborough Houston Jefferson County, KY Los Angeles Miami Milwaukee New York City Philadelphia San Diego District of Columbia (TUDA) 120 98.91 52 66 55 70 86 57 95 86 55 58 53 56 86 56 76 87 68 82 57 54 106 National private Catholic Non-Catholic private Lutheran Conservative Christian Other private Puerto Rico School type and jurisdiction Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 3,200 1,200 1.61 1.38 93.72 91.91 11.97 16.51 97.32 3,800 2.74 94.06 10.38 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 2,000 2,100 2,000 1,700 1,900 1,900 2,700 1,600 2,000 1,500 2,200 1,800 3,100 2,200 2,600 2,900 1,500 2,700 1,700 1,900 1,600 2.74 0.99 3.96 11.23 4.64 1.15 2.35 5.59 2.94 5.71 1.27 1.68 4.12 4.91 1.76 2.85 2.75 1.63 3.88 2.61 6.48 93.15 96.13 94.12 93.05 93.74 94.44 94.44 94.44 96.63 88.79 94.15 95.05 95.17 95.22 94.87 96.27 94.32 94.24 94.52 94.87 93.98 19.19 8.38 16.52 8.11 16.87 11.73 20.21 20.97 7.70 6.41 6.77 25.94 14.46 9.50 9.35 22.60 27.73 27.11 15.93 8.20 15.15 748 264 484 107 123 254 73.51 96.27 55.34 94.87 73.13 42.23 68.47 95.93 56.27 92.38 70.86 44.57 6,300 3,400 2,900 722 925 1,300 0.30 0.25 0.35 0.36 0.30 0.37 95.60 95.86 95.37 96.62 94.16 95.74 4.59 3.75 5.33 3.31 2.79 6.73 139 100.00 100.00 4,800 0.47 94.52 22.94 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools, but not schools in Puerto Rico.. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. NAEP 2018-2019 OMB Clearance: Appendix B Page 48 7/22/2016 2:46 PM 48 of 60 http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_g4math_2011.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 49 7/22/2016 2:46 PM 49 of 60 Participation, Exclusion, and Accommodation Rates for Grade 4 Reading for the 2011 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 4 reading assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 4 reading assessment, by school type and jurisdiction: 2011 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 9,200 97.41 91.65 National Northeast all Midwest all South all West all 9,200 97.41 91.65 1,600 2,400 2,900 2,300 95.19 97.76 97.64 98.27 National public Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois Indiana 8,200 117 202 127 123 292 124 116 109 153 235 174 118 137 193 117 99.83 98.95 100.00 99.03 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 School type and jurisdiction All all1 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 245,000 3.65 94.57 9.64 245,000 3.65 94.57 9.64 84.70 91.40 93.31 94.56 39,100 53,700 87,100 62,600 3.75 2.45 5.19 2.19 94.18 94.61 94.61 94.76 13.51 10.44 8.07 8.75 99.86 99.76 100.00 99.25 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 233,000 3,400 3,200 4,400 4,100 10,500 4,100 3,500 4,000 2,400 8,300 6,300 3,900 4,100 5,800 4,000 3.92 2.27 2.03 1.42 1.26 2.21 1.43 2.20 6.98 3.26 2.17 6.31 2.27 1.81 1.63 1.20 94.55 95.31 92.56 94.43 94.70 95.22 92.67 93.99 95.06 94.66 94.55 94.42 93.39 95.46 93.82 95.23 10.12 3.52 19.65 12.84 13.45 6.10 13.44 14.80 7.62 16.24 18.02 5.64 11.39 7.40 13.30 14.49 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. NAEP 2018-2019 OMB Clearance: Appendix B Page 50 7/22/2016 2:46 PM 50 of 60 School type and jurisdiction Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 141 148 158 134 166 173 202 172 148 117 131 206 181 118 133 118 155 158 173 272 205 140 149 167 125 114 209 119 306 130 226 115 141 152 190 100.00 99.18 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 99.17 100.00 100.00 100.00 99.97 100.00 100.00 99.08 100.00 100.00 100.00 100.00 100.00 99.08 100.00 100.00 100.00 100.00 100.00 100.00 100.00 99.30 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 98.83 100.00 100.00 100.00 99.61 100.00 100.00 99.36 100.00 100.00 100.00 100.00 100.00 99.23 100.00 100.00 100.00 100.00 100.00 100.00 3,800 3,600 5,700 4,000 3,700 5,400 5,800 4,900 4,300 3,500 4,100 3,600 3,600 4,500 3,700 3,800 4,900 5,400 6,100 3,600 5,000 3,600 4,300 5,300 3,600 3,900 3,700 4,000 11,300 4,700 3,100 4,200 4,500 3,500 5,100 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 1.01 2.21 8.73 1.33 1.56 10.33 5.69 3.52 1.56 1.06 1.65 4.25 4.33 1.12 2.78 9.09 5.71 2.56 2.21 6.49 5.77 4.97 2.62 2.91 2.06 2.74 3.18 7.05 9.93 4.14 2.38 2.79 2.84 1.70 1.86 95.68 95.16 94.41 93.88 93.86 94.44 94.48 94.40 94.46 93.78 94.56 93.94 95.31 95.59 93.93 94.76 93.43 93.75 93.81 96.01 94.22 95.14 94.58 94.28 95.01 94.30 95.71 94.71 94.83 94.15 93.54 94.73 95.44 95.16 94.53 14.99 11.63 3.78 17.26 14.74 6.72 12.31 7.10 10.47 5.60 9.29 5.13 10.97 17.76 13.85 8.85 10.06 19.39 12.12 6.34 9.43 9.30 12.70 11.17 12.39 7.26 7.61 6.82 3.14 7.66 13.64 9.77 11.67 7.94 15.95 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. Page 51 NAEP 2018-2019 OMB Clearance: Appendix B 7/22/2016 2:46 PM 51 of 60 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded Wyoming BIE 202 135 100.00 83.26 100.00 84.85 DoDEA 2 Trial Urban (TUDA) Districts Albuquerque Atlanta Austin Baltimore City Boston Charlotte Chicago Cleveland Dallas Detroit Fresno Hillsborough Houston Jefferson County, KY Los Angeles Miami Milwaukee New York City Philadelphia San Diego District of Columbia (TUDA) 120 98.91 52 66 55 70 86 57 95 86 55 58 53 56 86 56 76 87 68 82 57 54 106 National private Catholic Non-Catholic private Lutheran Conservative Christian Other private 748 264 484 107 123 254 School type and jurisdiction Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 3,400 1,300 1.98 1.62 94.66 90.96 12.48 14.88 97.32 4,000 6.74 94.10 6.70 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 2,100 2,200 2,100 1,700 2,000 2,000 2,700 1,600 2,000 1,500 2,200 1,900 3,200 2,200 2,700 3,000 1,600 2,800 1,800 2,000 1,700 5.11 3.81 16.49 16.89 8.06 1.63 2.10 5.41 18.48 7.01 2.30 2.55 14.45 9.61 1.89 3.84 2.67 2.49 3.41 3.61 3.93 92.87 96.23 94.26 92.70 94.48 94.57 95.27 93.03 95.49 88.99 93.71 94.61 95.27 94.80 95.11 95.68 94.75 93.01 94.40 95.08 94.99 12.48 5.57 4.50 3.44 14.36 10.24 19.13 20.79 2.80 5.39 6.01 24.39 3.57 5.11 8.93 21.80 28.77 26.24 16.23 7.29 18.04 73.51 96.27 55.34 94.87 73.13 42.23 68.47 95.93 56.27 92.38 70.86 44.57 6,500 3,500 3,000 753 950 1,300 0.51 0.45 0.56 1.17 0.00 0.74 94.91 95.49 94.41 96.23 95.40 93.71 4.23 3.52 4.84 2.69 1.96 6.48 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_gr4reading_2011.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 52 7/22/2016 2:46 PM 52 of 60 NAEP 2018-2019 OMB Clearance: Appendix B Page 53 7/22/2016 2:46 PM 53 of 60 Participation, Exclusion, and Accommodation Rates for Grade 8 Reading for the 2011 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 8 reading assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 8 reading assessment, by school type and jurisdiction: 2011 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 8,600 97.54 88.31 National Northeast all Midwest all South all West all 8,600 97.54 88.31 1,400 2,400 2,700 2,100 95.45 98.76 97.65 97.80 National public Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois Indiana 7,500 125 167 136 126 257 130 119 68 102 233 132 81 113 223 113 99.79 100.00 99.90 99.02 100.00 100.00 99.87 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 School type and jurisdiction All all1 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 197,000 3.23 93.01 8.58 197,000 3.23 93.01 8.58 79.60 91.49 89.96 89.74 32,200 44,400 70,800 48,500 3.65 2.87 4.00 2.07 92.23 93.60 93.09 92.92 12.98 9.34 7.01 7.12 99.76 100.00 97.93 99.08 100.00 100.00 97.31 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 186,000 3,000 2,900 3,100 3,100 8,100 3,100 2,900 3,100 2,900 6,900 4,600 3,200 3,200 4,600 3,100 3.48 2.07 1.82 1.17 1.50 2.17 1.57 2.25 5.25 2.84 2.33 4.40 2.21 1.77 1.62 2.08 92.84 94.17 91.24 93.72 93.81 93.19 92.12 92.28 93.01 89.51 91.62 93.50 92.40 94.08 93.67 92.91 8.97 3.84 15.69 8.48 11.47 6.31 9.83 12.02 8.95 16.48 15.10 6.27 10.01 6.24 11.91 12.55 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. NAEP 2018-2019 OMB Clearance: Appendix B Page 54 7/22/2016 2:46 PM 54 of 60 School type and jurisdiction Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 138 148 156 163 143 176 154 180 168 121 136 200 169 100 96 116 136 174 159 209 194 149 144 168 61 116 261 123 236 125 124 108 140 117 179 100.00 100.00 100.00 100.00 100.00 99.05 99.46 100.00 100.00 100.00 100.00 99.86 100.00 99.70 100.00 100.00 99.09 99.08 100.00 99.99 100.00 100.00 99.10 100.00 100.00 100.00 100.00 100.00 99.09 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 98.82 98.47 100.00 100.00 100.00 100.00 98.41 100.00 97.35 100.00 100.00 99.40 99.67 100.00 99.47 100.00 100.00 99.26 100.00 100.00 100.00 100.00 100.00 99.63 100.00 100.00 100.00 100.00 100.00 100.00 2,900 3,100 4,400 3,000 3,000 4,200 4,400 4,700 3,400 3,000 2,800 2,900 2,900 3,200 3,000 2,900 3,900 4,600 4,900 2,600 4,200 2,900 3,300 4,400 3,000 3,100 3,300 3,200 8,800 3,400 2,300 3,100 3,600 3,100 4,000 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 0.75 1.90 7.22 1.00 1.73 8.43 6.32 4.83 2.84 0.96 1.39 4.03 4.73 1.93 4.15 7.08 5.74 3.10 2.06 7.90 5.75 4.34 2.13 3.10 1.16 5.30 3.22 6.31 6.04 3.70 2.76 3.62 2.11 1.45 2.20 92.53 93.46 94.27 92.69 92.31 91.82 92.18 93.15 92.58 92.33 94.09 92.04 93.78 92.86 92.22 92.32 91.29 91.33 92.09 93.50 93.25 92.52 92.34 91.91 92.66 93.72 94.70 91.99 93.74 92.06 93.05 93.66 92.07 92.44 93.81 13.68 8.49 4.21 13.36 14.05 4.03 12.42 6.58 7.43 5.63 10.63 6.47 7.42 8.87 11.57 10.71 6.30 16.61 11.66 5.62 8.72 9.37 8.93 12.73 13.70 4.14 5.66 4.63 2.91 6.05 12.94 7.73 9.42 7.19 13.64 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. Page 55 NAEP 2018-2019 OMB Clearance: Appendix B 7/22/2016 2:46 PM 55 of 60 School type and jurisdiction Wyoming BIE Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 108 116 100.00 83.16 100.00 84.68 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 2,400 1,100 1.96 2.01 92.67 89.62 10.77 12.89 DoDEA 2 Trial Urban (TUDA) Districts Albuquerque Atlanta Austin Baltimore City Boston Charlotte Chicago Cleveland Dallas Detroit Fresno Hillsborough Houston Jefferson County, KY Los Angeles Miami Milwaukee New York City Philadelphia San Diego District of Columbia (TUDA) 72 98.56 95.31 2,000 3.26 91.78 7.80 34 26 24 75 45 38 117 71 41 63 23 50 50 46 71 86 59 91 60 31 50 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 1,400 1,600 1,700 1,300 1,400 1,600 2,100 1,200 1,600 1,900 1,500 1,500 2,500 1,600 2,300 2,800 1,400 2,500 1,500 1,300 1,600 7.29 3.55 9.01 16.95 9.66 2.04 2.28 5.21 5.95 7.96 1.90 1.77 6.40 6.83 2.04 3.77 3.33 2.63 4.69 1.49 3.86 88.93 92.38 93.15 88.94 89.97 92.97 94.92 91.23 92.60 85.41 92.21 94.45 94.12 91.58 91.67 92.91 90.89 91.54 91.11 95.63 87.69 9.06 6.29 4.96 3.31 14.19 9.66 15.81 24.77 4.86 7.95 6.37 20.97 3.87 6.06 8.84 15.63 27.81 22.49 19.54 9.03 19.31 National private Catholic Non-Catholic private Lutheran Conservative Christian Other private 930 332 598 141 149 308 74.40 93.23 57.54 92.73 72.51 45.71 69.89 92.09 58.75 91.45 74.78 45.85 8,600 4,500 4,100 1,000 1,200 1,900 0.47 0.38 0.56 0.48 0.29 0.69 94.77 95.36 94.22 95.12 93.55 94.39 4.32 3.29 5.29 2.79 2.00 7.03 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. NAEP 2018-2019 OMB Clearance: Appendix B Page 56 7/22/2016 2:46 PM 56 of 60 http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_gr8reading_2011.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 57 7/22/2016 2:46 PM 57 of 60 Participation, Exclusion, and Accommodation Rates for Grade 8 Science for the 2011 Assessment The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 8 science assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column headings of the table. The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school population that is represented by the responding schools in the sample. These rates differ because schools differ in size. Participation, exclusion, and accommodation rates, grade 8 science assessment, by school type and jurisdiction: 2011 School type and jurisdiction All all1 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded 8,600 97.32 88.19 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 141,000 1.57 92.87 10.58 National Northeast all Midwest all South all West all 8,600 97.32 88.19 141,000 1.57 92.87 10.58 1,400 2,400 2,700 2,100 95.45 98.76 97.65 96.86 79.60 91.49 89.96 89.18 24,000 32,400 47,700 35,100 1.36 1.60 1.70 1.49 92.02 93.40 93.10 92.68 15.71 10.41 9.71 8.38 National public Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois 7,500 125 167 136 126 257 130 119 68 102 233 132 81 113 223 99.54 100.00 99.90 99.02 100.00 100.00 84.41 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 99.57 100.00 97.93 99.08 100.00 100.00 87.04 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 138,000 2,600 2,400 2,700 2,700 2,800 2,200 2,500 2,600 3,000 2,700 2,800 2,700 2,800 3,900 1.69 1.08 1.08 0.88 0.95 1.76 0.90 1.32 1.69 1.49 1.24 1.55 1.99 1.46 1.14 92.81 93.22 89.90 93.19 94.08 92.96 92.56 91.41 92.10 87.56 93.10 92.81 92.59 93.06 94.03 11.10 4.13 16.37 9.33 11.63 7.76 10.32 12.55 12.20 17.52 16.28 8.43 10.80 6.66 12.40 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. NAEP 2018-2019 OMB Clearance: Appendix B Page 58 7/22/2016 2:46 PM 58 of 60 School type and jurisdiction Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia 113 138 148 156 163 143 176 154 180 168 121 136 200 169 100 96 116 136 174 159 209 194 149 144 168 61 116 261 123 236 125 124 108 140 117 100.00 100.00 100.00 100.00 100.00 100.00 99.05 99.46 100.00 100.00 100.00 100.00 99.86 100.00 99.70 100.00 100.00 99.09 99.08 100.00 99.99 100.00 100.00 99.10 100.00 100.00 100.00 100.00 100.00 99.09 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 98.82 98.47 100.00 100.00 100.00 100.00 98.41 100.00 97.35 100.00 100.00 99.40 99.67 100.00 99.47 100.00 100.00 99.26 100.00 100.00 100.00 100.00 100.00 99.63 100.00 100.00 100.00 100.00 100.00 2,600 2,500 2,600 3,800 2,600 2,600 2,600 2,700 2,600 2,900 2,500 2,400 2,500 2,500 2,700 2,600 2,500 3,300 3,900 2,900 2,200 2,600 2,400 2,800 2,600 2,600 2,700 2,800 2,700 3,200 2,900 2,000 2,600 3,000 2,700 Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 1.29 0.94 1.41 2.72 1.21 1.83 1.93 3.20 2.74 1.96 0.92 1.23 1.53 1.44 1.23 2.15 1.22 1.75 1.39 1.61 3.22 2.13 2.86 1.55 1.03 0.65 1.19 1.22 1.43 2.36 1.83 1.39 2.68 1.88 1.60 93.83 92.75 94.45 93.04 93.36 92.66 92.54 92.20 92.28 92.13 92.49 93.44 91.02 94.57 93.07 90.78 91.77 91.94 91.24 92.21 94.63 92.62 92.26 92.66 93.28 92.15 94.22 95.08 92.35 93.04 91.80 93.95 94.01 91.86 93.48 12.92 14.34 9.06 8.19 13.21 13.97 10.87 15.97 8.36 8.48 6.22 9.93 9.08 11.59 11.17 13.11 17.29 10.28 18.41 12.12 10.14 12.38 10.04 10.18 14.73 14.31 9.39 8.01 10.05 7.69 9.47 14.15 9.96 9.68 9.14 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. Page 59 NAEP 2018-2019 OMB Clearance: Appendix B 7/22/2016 2:46 PM 59 of 60 Number of schools in original sample, rounded School participation rates (percent) before substitution (weighted by base weight and enrollment) School participation rates (percent) before substitution (weighted by base weight only) Number of students sampled, rounded Wisconsin Wyoming BIE 179 108 116 100.00 100.00 83.16 100.00 100.00 84.68 DoDEA 2 72 98.56 930 332 598 74.40 93.23 57.54 School type and jurisdiction National private Catholic Non-Catholic private Weighted percent of students excluded Weighted student participation rates (percent) after makeups Weighted percent of students accommodated 2,400 2,000 112 1.91 1.31 0.00 93.21 92.27 88.14 13.62 11.27 6.07 95.31 1,700 1.33 94.33 9.55 69.89 92.09 58.75 903 476 427 0.18 0.35 0.00 93.65 94.27 93.03 4.74 4.65 4.84 1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department of Defense Education Activity schools. 2 Department of Defense Education Activity. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Assessment. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_g8science_2011.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 60 7/22/2016 2:46 PM 60 of 60 Nonresponse Bias Analyses for the 2011 Assessment NCES statistical standards call for a nonresponse bias analysis to be conducted for a sample with a response rate below 85 percent at any stage of sampling. Weighted school response rates for the 2011 assessment indicated a need for school nonresponse bias analyses for private school samples in grades 4 and 8 (operational subjects), for public school samples for the Bureau of Indian Education (BIE) in grades 4 and 8, and for private school samples in grades 8 and 12 (computer-based writing). Student nonresponse bias analyses were necessary for students in Detroit for grade 8 public schools (mathematics). Additionally, a student nonresponse bias analysis was required to handle the special case of session nonresponse in the science sample in grade 8 Colorado public schools. Thus, six separate school-level analyses and two separate student-level analyses were conducted. The procedures and results from these analyses are summarized briefly below. The analyses conducted consider only certain characteristics of schools and students. They do not directly consider the effects of the nonresponse on student achievement, the primary focus of NAEP. Thus, these analyses cannot be conclusive of either the existence or absence of nonresponse bias for student achievement. For more details, please see the NAEP 2011 NRBA report (818KB). Each school-level analysis was conducted in three parts. The first part of the analysis looked for potential nonresponse bias that was introduced through school nonresponse. The second part of the analysis examined the remaining potential for nonresponse bias after accounting for the mitigating effects of substitution. The third part of the analysis examined the remaining potential for nonresponse bias after accounting for the mitigating effects of both school substitution and school-level nonresponse weight adjustments. The characteristics examined were Census region, reporting subgroup (private school type), urban-centric locale, and size of school (categorical). Based on the school characteristics available, for the private school samples at grades 8 and 12, there does not appear to be evidence of substantial potential bias resulting from school substitution or school nonresponse. However, the analyses suggest that a potential for nonresponse bias remains for the grade 4 private school samples. This result is evidently related to the fact that, among non-Catholic schools, larger schools were less likely to respond. Thus, when making adjustments to address the underrepresentation of non-Catholic schools among the respondents, the result is to overrepresent smaller schools at the expense of larger ones. The limited school sample sizes involved means that it is not possible to make adjustments that account fully for all school characteristics. Please see the full report for more details. Each student-level analysis was conducted in two parts. The first part of the analysis examined the potential for nonresponse bias that was introduced through student nonresponse. The second part of the analysis examined the potential for bias after accounting for the effects of nonresponse weight adjustments. The characteristics examined were gender, race/ethnicity, relative age, National School Lunch Program eligibility, student disability (SD) status, and English language learner (ELL) status. For Colorado, additional school characteristics were examined: Census region, urban-centric locale, size of school (categorical), and state-based achievement (categorical). Based on the student characteristics available, for the grade 8 Detroit student samples, there does not appear to be evidence of substantial potential bias resulting from student nonresponse. The same result can be concluded for grade 8 Colorado student samples, when considering student characteristics. However, analyses of the school characteristics suggest that a potential for nonresponse bias remains. Please see the full report for more details. http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_qa_nonresp_bias_analyses.aspx NAEP 2018-2019 OMB Clearance: Appendix B Page 61 7/22/2016 2:46 PM NATIONAL CENTER FOR EDUCATION STATISTICS NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS Appendix C 2018 Sampling Memo National Assessment of Educational Progress (NAEP) 2018 and 2019 OMB# 1850-0928 v.5 March 2017 Date: February 22, 2017 To: Bill Ward, NCES Amy Dresher, ETS Ed Kulick, ETS Dianne Walsh Rick Rogers William Wall Rob Dymowski Lisa Rodriguez Chris Averett Kavemuii Murangi From: Leslie Wallace Reviewer: David Hubble Subject: Sample Design for 2018 NAEP – Overview I. Memo: 2018-m03v01psu/m01v01s David Freund, ETS Cathy White, Pearson Keith Rust Dwight Brock Lauren Byrne John Burke Joel Wakesberg Lloyd Hicks Sipeng Wang Jason Schuknecht Introduction For 2018, the sample design involves several components, all of which are national assessments of one kind or another. 1. Operational Digitally-based Assessment (DBA) in Civics at grade 8; 2. Operational DBA in U.S. History at grade 8; 3. Operational DBA in Geography at grade 8; 4. Operational assessment in Technology and Engineering Literacy (TEL) at grade 8; 5. Pilot tests at grades 4, 8, or 12: a. Math DBA at grade 12 b. Reading DBA at grade 12 c. Science integrated DBA at grades 4, 8, and 12; 6. Bridge Paper-based Assessment (PBA) in Civics at grade 8; 7. Bridge PBA in U.S. History at grade 8; 8. Bridge PBA in Geography at grade 8; \\westat.com\dfs\NAEPLIB\2018\Memos\School Sampling\2018-m01v01s.docx Page 15 Memorandum 2018-m03v01psu/-m01v01s -2- February 22, 2017 In addition, the following special studies will be conducted: 9. Oral Reading Fluency (ORF) at grade 4 10. Reading Scenario-based Task (SBT) Special Study at grades 4, 8, and 12. 11. NTPS-NAEP Linking Study at grades 4, 8, and 12. Note that this study will likely involve a subsample of the NAEP schools at each grade. Details are unclear at this time, so the study is not discussed in the remainder of this memo. The target sample sizes of assessed students for the various components are shown in Table 1 (which also shows an estimate of the required number of participating schools). All of these assessments are to take place in the typical NAEP testing window of late January to early March 2018. Note that the Pilot assessments and special studies are conducted in public schools only, whereas the Social Studies and TEL assessments are conducted in both public and private schools. Table 1. 2018 NAEP Sample Sizes (Public and Private) Session* Grade 4 Science Integrated DBA (P) Reading SBT (S) ORF (S) Total Schools Grade 8 Civics DBA (O) Geography DBA (O) U.S. History DBA (O) Civics PBA (B) Geography PBA (B) U.S. History PBA (B) TEL (O) Science Integrated DBA (P) Reading SBT (S) Total Schools Grade 12 Math DBA (P) Reading DBA (P) Science Integrated DBA (P) Reading SBT (S) Total Schools DA DB DO PC PH DL DA DA GRAND TOTAL Schools Public school students Total students 16,100 2,200 2,000 20,300 0 0 0 0 16,100 2,200 2,000 20,300 520 7,200 7,200 9,000 7,200 7,200 9,000 14,400 15,900 2,200 79,300 800 800 1,000 800 800 1,000 1,600 0 0 6,800 8,000 8,000 10,000 8,000 8,000 10,000 16,000 15,900 2,200 86,100 2,050 10,500 4,500 17,500 2,200 34,700 0 0 0 0 0 10,500 4,500 17,500 2,200 34,700 870 134,300 6,800 141,100 3,440 (O) = Operational, (B) = Bridge, (P) = Pilot, and (S) = Special Study * The session designations are not final. NAEP 2018-2019 OMB Clearance: Appendix C Private school students Page 2 Memorandum 2018-m03v01psu/-m01v01s II. -3- February 22, 2017 Assessment Types From a sampling and operations point of view, many types of assessment sessions can be distinguished. The detailed target counts of assessed students are provided in Table 1. 1. The DBA Civics, U.S. History, and Geography spiral is only at grade 8. This spiral must be assessed in a different physical session from the others, but will be in the same schools as the PBA Civics, U.S. History, and Geography session types (see immediately below). The session has a target of 26,000 assessed students (8,000 Civics, 10,000 U.S. History and 8,000 Geography). 2. The PBA U.S. History and Geography spiral is only at grade 8. This session spiral has a total target of 18,000 assessed students (10,000 U.S. History and 8,000 Geography). The PBA Civics session is also only at grade 8. This session has a target of 8,000 assessed students. Both of these PBA Social Studies sessions will be conducted in the same schools as the DBA Social Studies session described above. 3. The Technology and Engineering Literacy (TEL) operational assessment for grade 8 will be computer delivered. Because of the different delivery method, this assessment must be in separate sessions. In fact, out of concern for overburdening schools conducting Social Studies assessments, an additional set of Primary Sampling Units (PSUs), with minimum overlap with Social Studies PSUs, will be used for conducting TEL with a target of 16,000 assessed students. 4. The DBA Pilot/SBT at grades 4, 8, and 12 will be tablet delivered and conducted only in public schools. Because of the different delivery method and out of concern for overburdening schools conducting social studies and TEL assessments, a third set of PSUs will be used for conducting the DBA Pilot/SBT. One session will be administered at each grade. At grades 4 and 8, the session will be a DBA Science Pilot/SBT spiral, with targets of 18,300 assessed students at grade 4 and 18,100 assessed students at grade 8. At grade 12, the session will be a DBA Science/Reading/Math/SBT spiral, with a target of 34,700 assessed students. 5. The ORF Special Study at grade 4 will be conducted in some of the DBA Pilot (public) schools. This will be a separate session with a target of 2,000 assessed students. III. Primary Sampling Units Selection and Overlap Control There are three separate PSU samples for 2018: one each for Social Studies, TEL, and DBA Pilot/Special Studies. As the U.S. History, Geography, and Civics assessments are national, with a total original sample size of assessed students of about 52,000 at grade 8, for reasons of operational efficiency in conducting the assessments a sample of PSUs was selected, and all sampled schools will be drawn from within the sampled PSUs. With a smaller sample size of about 16,000 assessed students for the computer delivered TEL assessment in grade 8, a separate sample of PSUs was selected with the largest PSUs being in common to both PSU samples. Finally, with a total sample size of 73,100 assessed students across grades 4, 8, and 12 for the DBA Pilot and Special Studies, a third sample of PSUs was selected with the largest PSUs being in common in all three PSU samples. NAEP 2018-2019 OMB Clearance: Appendix C Page 3 Memorandum 2018-m03v01psu/-m01v01s -4- February 22, 2017 The PSUs were created from aggregates of counties. Data on counties were obtained from the 2010 Census, and the definitions of Metropolitan Statistical Areas (MeSAs) used were the December 2009 Office of Management and Budget (OMB) definitions. Each Metropolitan Statistical Area (MeSA) constitutes a PSU, except that MeSAs that cross Census region boundaries were split into their individual regional components. Non-metropolitan PSUs were formed by aggregating counties into geographic units of sufficient minimum size to provide enough schools to constitute a workload of about 1% of the total sample. These PSUs were made of contiguous counties where possible, and almost contiguous counties (separated by MeSA counties) otherwise. Each PSU falls within a single state. This process generated a frame of approximately 1,000 PSUs. The PSUs were stratified, using characteristics aggregated from county-level characteristics, found by analysis to be related to NAEP achievement in past assessments. A sample of 105 PSUs was selected for the Social Studies samples. The 29 largest MeSAs were selected with certainty, and the remaining sample was a stratified probability proportional to size (PPS) sample, where the size measure was a function of the number of children as given in the most recent population estimates prepared by the U.S. Census Bureau. For the Social Studies sample, 76 such strata were formed and a single PSU was selected from each stratum for a total of 105 PSUs. For the TEL sample, the same certainty PSUs were selected. However, the 76 strata were formed and paired and a single PSU was selected from one stratum in each of the 38 pairs for a total of 67 PSUs. For the DBA Pilot/Special Studies samples, the same certainty PSUs were again selected and the 76 strata were formed and paired. However, a single PSU was selected from each of the strata in the 38 pairs not used for the TEL sample for a total of 67 PSUs. The three PSU samples were selected in such a way as to minimize overlap between them. This was done to reduce the chance that a school is selected for more than one of the Social Studies, TEL, or DBA Pilot/Special Studies assessments. Due to the fact that three PSU samples were selected and that one of them consisted of 105 PSUs, overlap among the three PSU samples, though minimized, was not entirely avoided. Five noncertainty PSUs overlap between the Social Studies and TEL samples, and (a different) five noncertainty PSUs overlap between the Social Studies and DBA Pilot/Special Studies samples. IV. Stratification and Oversampling As in the recent past, the plan is to draw separate public and private school samples. This approach has proven to be useful, in that, selecting the samples separately has three advantages: 1) it permits the timing of sample selection to vary between public and private schools, should this prove necessary; 2) it allows us to readily assume different response and eligibility rates for public schools and private schools; and 3) it makes it easier to use different sort variables for public schools and private schools. It also allows for the possibility of a late change of mind concerning the sample sizes that differ between public and private schools. Note that the DBA Pilot and Special Studies designs do not include private school components as the assessment goals could be better met through other means in this case. Explicit stratification will take place at the PSU level. For schools within PSUs, stratification gains will be achieved by sorting the school file prior to systematic selection. As in past national samples, the expectation is that, within the set of certainty MeSA PSUs within a census region, PSU will not necessarily be the highest level sort variable. Thus, type of location will be used as the primary sort variable. Consider for example the large MeSAs in the Midwest region. The design is aimed primarily at getting the correct balance of city, suburban, town, and rural schools, as a priority over getting exactly a proportional representation from each MeSA (Chicago, Detroit, Minneapolis), although of course it should be possible to get a high degree of control over both of these characteristics. The sort of the NAEP 2018-2019 OMB Clearance: Appendix C Page 4 Memorandum 2018-m03v01psu/-m01v01s -5- February 22, 2017 schools will use other variables beyond the type of location variable, such as a race/ethnicity percentage variable. The exact set of variables used in sorting the schools prior to sampling will be specified in the particular sampling specification memos. In addition, we will implement three different kinds of oversampling of public schools. First, in order to increase the likelihood that the results for American Indian/Alaskan Native (AIAN)1 students can be reported for the operational samples, we will oversample high-AIAN public schools for Social Studies and TEL at grade 8. That is, a public school with over 5 percent AIAN enrollment will be given four times the chance of selection of a public school of the same size with a lower AIAN percentage. Recent research into oversampling schemes that could benefit AIAN students indicates that this approach should be effective in increasing the sample sizes of AIAN students, without inducing undesirably large design effects on the sample, either overall or for particular subgroups. In addition, high minority public schools for Social Studies and TEL that are not oversampled for AIAN enrollment will be oversampled for Black and Hispanic enrollment. That is, as used in past national assessments, a public school with over 15 percent Black and Hispanic combined enrollment will be given twice the chance of selection of a public school of the same size with a lower percentage of these two groups. This approach is effective in increasing the sample sizes of Black and Hispanic students, without inducing undesirably large design effects on the sample, either overall or for particular subgroups. The second kind of oversampling to be implemented will be oversampling of public schools based on National School Lunch Program (NSLP) eligibility in order to accommodate the ORF Special Study. That is, for the grade 4 DBA Pilot/Special Study samples, a public school with over 75 percent student eligibility for the NSLP will be given twice the chance of selection of a public school of the same size with a lower percentage of NSLP student eligibility. The third kind of oversampling to be implemented will be oversampling of high-minority public schools for the grade 8 DBA Pilot/SBT and grade 12 DBA Pilot/SBT samples. That is, as used in past national assessments, a public school with over 15 percent Black and Hispanic combined enrollment will be given twice the chance of selection of a public school of the same size with a lower percentage of these two groups. This approach is effective in increasing the sample sizes of Black and Hispanic students, without inducing undesirably large design effects on the sample, either overall or for particular subgroups. Beyond this, we will not implement the oversampling of Black and Hispanic students at the student level in schools not being oversampled at the school level as has been done in the past because such studentlevel oversampling is incompatible with the digital mode of assessment. The updated preliminary 2015/16 CCD and the updated 2015/16 PSS school files were approved for use by NCES. They serve as the basis for the public and private school frames for the 2018 NAEP. V. New Schools To compensate for the fact that the CCD file used to create the NAEP public school sampling frames is out of date at the time of frame construction, we will supplement the samples for the Social Studies and TEL assessments with a sample of new public schools for grade 8. New school samples will not be developed for the private school samples or the DBA Pilot/Special Studies samples. The new school samples will be drawn using a two-stage design. At the first stage, a national sample of school districts will be selected from the Social Studies and TEL sample PSUs. The sampled districts will be asked to review lists of their respective schools and identify new schools. Frames of new schools for grade 8 will be constructed from these updates, and new schools will be drawn with NAEP 2018-2019 OMB Clearance: Appendix C Page 5 Memorandum 2018-m03v01psu/-m01v01s -6- February 22, 2017 probability proportional to size using the same sample rates as their corresponding original school samples. Note that the student and school sample sizes in Table 1 do not reflect these new school samples. However, some schools from the original sample will prove to be closed or otherwise ineligible, and the new school procedure essentially compensates for the sample losses from these sources, as well as ensuring full coverage of the population. VI. Within-PSU Overlap Control with Other Samples In keeping with the efforts at the PSU level to reduce potential overlap between the Social Studies and TEL samples, methods will be employed to reduce overlap during sample school selection within the PSUs that contain more than one sample. In addition to the overlap control efforts between the Social Studies and TEL samples, methods will be employed to reduce overlap during NAEP school selection with the International Computer and Information Literacy Survey (ICILS). The ICILS is a national sample of schools at grade 8 (not PSU-based) and is being conducted in the spring of 2018. With this approach we expect it to be possible to avoid any school overlap among the Social Studies, TEL, and ICILS school samples at grade 8. Concurrent with the selection of the Social Studies and TEL samples, the DBA Pilot/Special Studies schools will be selected independently. No effort will be made to minimize overlap between the DBA Pilot/Special Studies samples at grade 8 and the Social Studies, TEL, or ICILS samples because the level of effort required to implement overlap control among four samples in order to avoid a few schools that might overlap is not justified for a pilot or special study. DBA Pilot/Special Studies schools at grade 8 that are also selected for Social Studies, TEL, or ICILS will be treated as nonrespondents and their substitutes will be recruited. Within the DBA Pilot/Special Studies sample, schools may sometimes be selected to participate at more than one grade. The Keyfitz method will be used to compute conditional probabilities to reduce the overlap between the samples within grade 8. That is, in the Social Studies PSUs, the conditional probabilities of selection for the Social Studies schools will be based on the ICILS school sampling outcome. Also, in the 33 TEL noncertainty PSUs that do not overlap with Social Studies, the conditional probabilities of selection for the TEL schools will be based on the ICILS school sampling outcome. Finally, in the 29 certainty PSUs and the 5 noncertainty PSUs that overlap between Social Studies and TEL, the conditional probabilities of selection for the TEL schools will be based on the Social Studies and ICILS school sampling outcomes. Specifically, this will be done to reduce overlap between Social Studies and TEL sample schools, between Social Studies and ICILS sample schools, and between TEL and ICILS sample schools. VII. Substitute Samples Substitute samples will be selected for each of the 2018 samples in the following order for public schools: Social Studies, TEL, and then DBA Pilot/Special Studies. Within the DBA Pilot/Special Studies sample, the order for selecting substitute schools will be from “oldest” to “youngest”. That is, grade 12, 8, and then 4. The order for selecting substitute samples for private schools will be Social Studies and then TEL. This ordering of samples and grades is necessary since no school can be selected as a substitute more than once. It is more critical for operational samples to precede non-operational ones and higher grades to precede lower grades due to having fewer schools available to serve as substitutes at the higher grades. Selecting substitutes will done separately for both public and private schools. The general steps NAEP 2018-2019 OMB Clearance: Appendix C Page 6 Memorandum 2018-m03v01psu/-m01v01s -7- February 22, 2017 for selecting substitutes are to put the school frames in their original sampling sort order, and take the 'nearest neighbor' of each original sampled school, excluding schools selected for any of the NAEP 2018 samples, schools already selected to serve as a substitute school, and schools which cross PSU or state boundaries, as potential substitutes. The nearest neighbor is the school adjacent (immediately preceding or succeeding) the original school in the sorted frame with the closer estimated grade enrollment value. If estimated grade enrollment of both potential substitute schools differs from the original school by the exact same amount, the selection procedure randomly chooses one of the schools. If neither the preceding or succeeding school is eligible to be a substitute, then the sampled school is not assigned a substitute. In addition, the few sampled private schools whose school affiliation is unknown will not get substitutes nor could such private schools not in sample serve as substitute schools. Also, new schools will not get substitute schools nor serve as substitutes. VIII. Student Sampling Student sample sizes within each school are determined as the combined result of several factors: them. 1. We wish to take all students in relatively small schools. 2. We wish to avoid the situation where all but a few students (e.g., more than 90%, but fewer than 100%) are tested. 3. We do not wish to have a sample that is too clustered for any one assessment subject. 4. We do not wish to have many physical sessions that contain only a very small number of students, as this is inefficient. 5. We wish to minimize the number of unique combinations of session types in the schools and to avoid three session types in a given sample school. 6. We do not wish to overburden the schools with unduly large student samples. 7. For the DBAs, we can use up to 25 tablets in a school at one time. The plans below reflect the design that results from considering each of these factors and balancing Social Studies: Grade 8 Schools We will select all students, up to 75. In schools with more than 75 students, we will select 75. There are three session types: DBA Civics/U.S. History/Geography (DBA), PBA Civics (C) and PBA U.S. History/Geography (H/G). The proportion of students assigned is 1/2 for the DBA session, 9/26 for the H/G session and 2/13 for the C session. Schools will be assigned one or two sessions; no schools will be assigned all three sessions. Minimum session size is 12 within schools with 12 or more students. This assignment of the DBA, C, and H/G sessions, based on the number of students in the school, is detailed in Table 2. NAEP 2018-2019 OMB Clearance: Appendix C Page 7 Memorandum 2018-m03v01psu/-m01v01s Table 2. -8- February 22, 2017 Grade 8 social studies school session allocations and proportions Grade 8 Enrollment size Probability of being assigned DBA and C Proportion of sample students assigned to DBA (in schools with DBA and C) Probability of being assigned DBA and H/G with two DBA sessions Proportion of sample students assigned to DBA (in schools with DBA and H/G and two DBA sessions) 1 to 23 24 to 50 51 and higher 0 4/13 6/13 NA 1/2 2/3 0 0 1/26 0 0 2/3 Probability of being assigned DBA and H/G with one DBA session Proportion of sample students assigned to DBA (in schools with DBA and H/G and one DBA session) 0 9/13 1/2 NA 1/2 1/3 Probability of being assigned DBA only 1/2 0 0 Probability of being assigned H/G only 9/26 0 0 Probability of being assigned C only 2/13 0 0 TEL: Grade 8 Schools We will select all students, up to 30. In schools with more than 30 students we will select 30. All students will be assigned to the TEL assessment. DBA Pilot/SBT/ORF: Grade 4 Schools There are two session types: DBA Pilot/SBT (DBA) and ORF. The proportion of students assigned is about 9/10 for the DBA session and about 1/10 for the ORF session. Some of the schools will be assigned only DBA. In these schools, we will select all students, up to 50. In schools with more than 50 students we will select 50. All students in these schools will be assigned to the DBA session. The other schools have the potential for doing both DBA and ORF sessions. In these schools we will select all students up to 37. In schools with more than 37 students, we will select 37—25 for DBA and 12 for ORF. Minimum DBA session size is 12 within schools with 12 or more students. Maximum ORF session size is 12. In order to achieve the session size constraints, every school that has the potential for doing both sessions will not always be assigned both sessions. This assignment of the DBA and ORF sessions, based on the number of students in the school, is detailed in Table 3. NAEP 2018-2019 OMB Clearance: Appendix C Page 8 Memorandum 2018-m03v01psu/-m01v01s Table 3. -9- February 22, 2017 Grade 4 school session allocations and proportions Grade 4 Enrollment size Probability of being assigned DBA only (and so selecting up to 50 students) 1 to 12 13 to 24 25 to 37 38 to 49 50 and higher 9/10 4/5 7/10 2/3 17/27 Probability of being assigned DBA and ORF (and so selecting up to 37 students) Proportion of sample students assigned to DBA (in schools with DBA and ORF session types) 0 1/5 3/10 1/3 10/27 NA 1/2 25/37 25/37 25/37 Probability of being assigned ORF only 1/10 0 0 0 0 DBA Pilot/SBT: Grade 8 and 12 Schools We will select all students, up to 50. In schools with more than 50 students we will select 50. All students will be assigned to the DBA Pilot/SBT session for grade 8 and 12. IX. Weighting Requirements Social Studies Samples These samples will have a single set of weights for each subject (DBA Civics, DBA U.S. History, DBA Geography, PBA Civics, PBA U.S. History, and PBA Geography at grade 8) applied to reflect probabilities of selection, school and student nonresponse, any trimming, and the random assignment to the particular subject. There will be a separate replication scheme by grade and public/private. For each subject, we will also provide weights for the combined DBA and PBA samples (Civics, U.S. History, and Geography at grade 8). TEL Sample As with the Social Studies samples, the TEL sample at grade 8 will be fully weighted. Pilot Test and Special Studies Samples As with the Social Studies and TEL samples, the ORF sample at grade 4 will be fully weighted. We will not weight the students in the Pilot samples or Reading SBT studies at grades 4, 8, and 12. However, preliminary weights will be available for these samples. NAEP 2018-2019 OMB Clearance: Appendix C Page 9 Endnotes 1 As states, districts, and schools are only required to report race/ethnicity data at the 7-category level (and specifically because this is how the data are recorded on the Common Core of Data, used as the sampling frame), the data used to oversample high AIAN percentage schools are the percent of students who are non-Hispanic AIAN students with no other race category. This is also the basis for the primary reporting results of AIAN students for NAEP. Note that the oversampling is at the school level, so that students who report multiple races, including AIAN, who are in schools with a high percentage of AIAN students, will also be oversampled. However, as noted, current NAEP primary reporting practices will not report such students as AIAN. NAEP 2018-2019 OMB Clearance: Appendix C Page 10
| File Type | application/pdf |
| File Title | Appendix A (Statute Authorizing NAEP) |
| Author | joconnell |
| File Modified | 2017-06-22 |
| File Created | 2017-04-04 |