Document
FY15 Non Response Bias Report - Compensation
ICR 201606-2900-003 · OMB 2900-0782 · Object 70310501.
Document [pdf]
Download: pdf | txt
Voice of the Veteran Line of Business Tracking Study Compensation Service Fiscal Year 2015 Non-Response Bias Analysis VETERANS BENEFITS ADMINISTRATION [FY15 REPORT] Table of Contents Executive Summary .................................................................................................................................. 4 Introduction ............................................................................................................................................. 5 Methodology ............................................................................................................................................ 6 2.1 J.D. Power Index Model ................................................................................................................... 6 2.2 Sampling......................................................................................................................................... 10 2.3 Data Collection ............................................................................................................................... 10 Non-Response Bias Analysis................................................................................................................... 11 3.1 Survey Yield .................................................................................................................................... 20 3.2 Missing Data Patterns and Mechanisms ........................................................................................ 27 3.3 Margin of Error .............................................................................................................................. 28 3.3.1 Sampling Distribution ....................................................................................................... 29 3.3.2 Distribution of Overall Satisfaction Index Scores.............................................................. 30 3.3.3 Analysis for Demographic Differences .............................................................................. 31 3.3.4 Data Imputation Analysis for Demographic Differences .................................................. 37 Findings .................................................................................................................................................. 40 Conclusion .............................................................................................................................................. 41 References ............................................................................................................................................. 42 List of Appendices Appendix A Missing Data Patterns and Mechanisms .................................................................... 43 Appendix B Item Response Rates .................................................................................................. 45 Appendix C Study Overview ........................................................................................................... 49 1.1 Study Background .................................................................................................................. 49 1.2 Methodology ......................................................................................................................... 49 1.2.1 Sample Criteria .................................................................................................................... 50 1.2.2 Sample File Generation ................................................................................................... 50 1.2.3 Data Transfer .................................................................................................................... 51 1.2.4 Data Cleaning .................................................................................................................... 53 1.2.5 Sample Cleaning Rules Glossary..................................................................................... 53 1.2.6 Sample Selection .............................................................................................................. 54 1.2.7 Fielding/Sampling Frequency ......................................................................................... 54 1.2.8 Order Generation and Fulfillment Process ................................................................... 54 1.2.9 Data Collection ................................................................................................................. 56 1.2.10 Reporting......................................................................................................................... 56 ii Appendix D .................................................................................................................................... 57 Approaches to Mitigating the Effect of Non-Response Bias .......................................................... 57 1.1 Approach 1: Strategies to Maximize Response Rates ........................................................... 57 1.2 Approach 2: Correcting Unit Non-response Bias with Sample Weighting and Survey Raking58 Strategies to Improve Response Rate ............................................................................................ 59 Appendix E Impact of FAR 8.8 ........................................................................................................ 60 1.1 Impact .................................................................................................................................... 60 Appendix F Survey Questionnaire.................................................................................................. 61 Appendix G List of Acronyms………………………………………………………………..………….……………………….78 iii Executive Summary The Voice of the Veteran (VOV) Line of Business Tracking Satisfaction Research Study was developed to establish continuous satisfaction measurement and incorporate direct Veteran feedback in the decision making process in order to improve the level of service to Servicemembers, Veterans, and their beneficiaries. As part of this study, two surveys were fielded in Fiscal Year 2015 (FY15) for the Department of Veterans Affairs (VA), Veterans’ Benefits Administration (VBA) Compensation Service. One survey was based upon the access of the benefit and the other on the ongoing servicing of the benefit. The Access survey yielded a response rate of 21.96% (2.55% increase from FY14) and the Servicing survey yielded a response rate of 25.03% (3.64% increase from FY14). These rates were lower than the estimated response rate submitted with the information collection request (ICR) as well as lower than the Office of Management and Budget’s standard of 80% (at the overall unit response rate). OMB’s “Standards and Guidelines for Statistical Surveys,” Section 3.2, Guideline 3.2.9, notes that a non-response analysis should be conducted for surveys with an overall unit response rate of less than 80%. Therefore, J.D. Power (JDP) conducted the necessary statistical tests in accordance with OMB’s guidelines in order to verify the validity of Compensation’s Service’s survey results for FY15. The initial 2015 analyses for these reports were done in consultation with Dr. Don Dillman, a professor at Washington State University. Dr. Dillman is regarded as a key survey method expert on non-response bias research and the report conforms to sound statistical research practices in accordance with OMB standards. The analysis performed also includes an iterative survey raking procedure to derive sample weightings based on a simultaneous balancing analysis of the demographic differences. After adjusting for demographic differences between survey respondents and nonrespondents, the statistical tests performed on the survey responses for the Compensation Service surveys collected, illustrate that no differences were found in the Overall Satisfaction Index Score and Advocacy rating (likelihood to inform others about VA benefits). The sample for the Access population was defined as individuals who have received a decision in the past 30 days and includes those who were found eligible on a new or subsequent claim and those who have been denied and are not appealing the decision. The Access Overall Satisfaction score (663) and advocacy rating (3.53 on rating 1-4) are not impacted in any meaningful way by non-response bias. The sample for the Servicing population was defined as individuals who began receiving compensation benefits within the last six to eighteen months. The Servicing Overall Satisfaction score (630) and advocacy rating (3.50 on rating 1-4) are not impacted by non-response bias. This analysis confirms that the data collected during Fiscal Year 2015 is valid for use by VBA. 4 Introduction In an effort to achieve the highest level customer service, VBA partnered with J.D. Power to conduct Veteran satisfaction research on its behalf. VBA’s Voice of the Veteran (VOV) Satisfaction Initiative was established to continuously measure and improve the level of service to Servicemembers, Veterans, and their beneficiaries. The intent of this initiative is to: Reinstate VBA’s customer satisfaction research program in order to incorporate Veteran feedback into the decision-making process Identify the critical factors to Veterans’ satisfaction with benefits and services provided by VBA Provide continuous feedback to validate effectiveness of new initiatives and process changes Provide decision-makers and stakeholders with timely and actionable feedback on a continuous basis Identify and document best practices, and act as a vehicle to celebrate successful interactions and experiences The VOV Line of Business Tracking Satisfaction Research Study was developed to continuously field customer satisfaction survey instruments to provide Veteran and beneficiary feedback on the following VBA lines of business and benefit programs: Compensation, Pension, Education, Vocational Rehabilitation and Employment, and Loan Guaranty (including Specially Adapted Housing). In support of this effort, in FY15, JDP fielded a survey instrument regarding the Access and Servicing process on behalf of the Compensation program. The purpose of the Access and Servicing process surveys was to identify the factors critical to Veteran satisfaction with the access and receipt of benefits issued by VBA and to improve the level of services provided. The survey instruments for Servicing and the Access process were developed in collaboration with VA’s Compensation Service, and in accordance with OMB’s guidelines concerning statistical collection procedures and methods. After the initial survey instrument was designed, cognitive labs using the “think aloud” method were conducted to evaluate user experience when completing the survey. Prior to the FY15 fielding of the Servicing and Access process survey, a benchmark pilot study was conducted from October 2012 through January 2013 to further assess the effectiveness of the methodology and conformance to OMB’s standards. Additionally, we have also fielded the study in 2014 and the 2015 fielding will be the third iteration. 5 Methodology 2.1 J.D. Power Index Model J.D. Power defines customer satisfaction as a measure of how well product or service experiences fit the expectations of customers. All JDP index models assume a two-tiered regression model involving factors and attributes. Each customer experience is influenced by several factors (i.e. first tier), which in turn, are influenced by several attributes or drivers (i.e. second tier). A diagram of the index model follows on the subsequent page. In order to begin the index model calculation, each set of attributes within a factor is used to predict the Overall Satisfaction Index score (sub-OSAT) for that factor. An importance weight is assigned to each attribute, where the weight of “importance” of each attribute is defined as the ability of that attribute to predict Overall Satisfaction. A multiple regression model is used to estimate the attribute weights. This model produces the “bottom” level weights and is computed for each factor separately. The bottom level weights are rescaled so that they add up to a rating of 1 point within each subcategory. As a result, the percentage of total explained variation in the sub-OSAT that is due to a particular attribute constitutes that attribute’s importance weight within its respective factor. Following the calculation of attribute (i.e. bottom level) weights, the factor (i.e. top-level) weights are calculated. Factor scores are calculated by taking the sum of the product of the attribute rating scores and the attribute importance weights. This model produces the “top” level weights and these weights are rescaled so that they add up to a score of 1 point. Thus, the percentage of the total explained variation in the Overall Satisfaction Index score that is due to a particular sub-OSAT constitutes that factor’s importance weight. After all factor scores are computed, they are weighted so that some contribute more to Overall Satisfaction than others, based on the index importance weights. The index score is subsequently calculated by taking the sum of the product of all of the factor scores and the factor importance weights. Finally, both the index and factor scores are multiplied by 100 so that the range of each is 100 (if all attributes were rated 1 point) to 1,000 (if all attributes were rated 10 points). By applying the importance weights derived from the two-tiered modeling approach, JDP creates a weighted index score that ranges from a low of 100 to a high of 1,000 points. This index approach has the benefit of being highly reliable and valid and provides increased ability to discriminate the performance levels of companies and organizations. 6 Compensation Access and Servicing Process Index Weights In working with Compensation’s subject matter experts and leadership, the design of its survey encompasses the factors and attributes as outlined in the tables on the next page. The factors (Benefit Information, Contact with VA, Benefit Application, and Benefit Entitlement) and attributes (Ease of Accessing Information, Availability of Information, etc.) represent Access and Servicing Index Models in FY15. The corresponding weights for each factor and attribute are the weights based on the above index model calculation. The weights are derived from the relative importance of each factor or attribute to the respondents. 7 Table 2.0. Access: Index Model Weights Access Index Model Weights Effective Weight Benefit Information Contact with VA Application Process Clarity of Info on Appeal Benefit Entitlement 18.28% 11.23% 31.18% 2.88% 36.43% Table 2.1. Access: Weights by Attribute Access Weights by Attribute Effective Weight Benefit Information Ease of accessing information 3.88% Availability of information 2.67% Clarity of information 3.20% Usefulness of information 3.52% Frequency of information 5.02% Application Process Ease of completing the application 7.75% Timeliness of eligibility notification 13.31% Flexibility of application methods 10.12% Contact with VA 11.23% Clarity of Info on Appeal 2.88% Benefit Entitlement (Timeliness of receiving benefit) 36.43% 8 Table 2.2. Servicing: Index Model Weights Servicing Index Model Weights Effective Weight Benefit Information Contact with VA Benefit Entitlement 26.49% 12.49% 61.02% Table 2.3. Servicing: Weights by Attribute Servicing Weights by Attribute Effective Weight Benefit Information Ease of accessing information 5.17% Availability of information 3.22% Clarity of information 4.41% Usefulness of information 6.23% Frequency of Information 7.46% Benefit Entitlement Disability evaluation rating percentage 26.96% Timeliness of receiving benefit/services 14.59% Clarity of your disability rating 19.47% Contact with VA 12.49% 9 2.2 Sampling The Servicing survey was fielded to Veterans and beneficiaries who began receiving compensation benefits six to eighteen months ago. The Access survey is fielded to Veterans and beneficiaries who received a decision for their application for compensation benefits within the past 30 days. These individuals may include those who were found eligible on a new or subsequent claim and those who have been denied and are not appealing the decision. J.D. Power mailed approximately 160,000 surveys for the Access survey and 60,000 for the Servicing survey to Veterans (and surviving spouses) across the nation in FY15. The target number of completed surveys was 48,000 for Access and 18,000 for Servicing. The actual number of completed surveys received for Access was 36,605 and for Servicing it was 16,030. The samples used in this study was provided by the Office of Performance Analysis and Integrity (PA&I) on behalf of Compensation and delivered to JDP. The sample was a random sample from the available records provided in the sample file. See Appendix D, Sample Plan Overview for further detail on sampling. Methodology Fielding Frequency Total Mail-outs in FY15 Access Mixed Monthly 160,000 Servicing Mixed Annually 60,000 Survey Instrument 2.3 Data Collection During the survey fielding period, both self-administered online survey returns and selfadministered paper surveys were collected. While verbatim responses are recorded by a live survey processor, responses from paper surveys are scanned through automated imaging software. Survey returns undergo quality assurance to validate the accuracy of responses captured. Respondents received two separate mailings, and had the option of completing the survey on paper or online: 1st Mailing: Postcard, introducing the study to the respondent, which included an online survey link and a unique access code login for the online survey. 2nd Mailing: Survey Package, which included a cover letter with the online survey link, a paper survey, and a business reply envelope. 10 Each time the surveys were deployed, the survey packages were subject to a proof approval process that utilized three levels of approvals by J.D. Power, Benefits Assistance Service (BAS), and VA Publications Services Division (VAPSD). After the print vendor mailed the postcards and survey packages, mail receipts were sent to VBA. During the survey fielding period, JDP provided a toll-free survey hotline and dedicated e-mail address to answer survey-related inquiries and to provide assistance to respondents for completing the surveys. The telephone and e-mail helpdesk was staffed with three JDP employees who answered inquiries during regular business hours (8:00am-5:00pm PST, Monday thru Friday). A voice message system was available to receive phone messages so after-hours calls could be responded to the following business day. An automatically generated e-mail response was sent to all e-mail inquiries informing respondents that their e-mail was received and they would receive a response within 24 hours. JDP helpdesk representatives logged each survey-related inquiry in a password protected spreadsheet documenting the reason for the inquiry, the resolution provided, and the contact information of each caller. At the end of each month, a log containing all inquiries was provided to the Contracting Officer Representative (COR) for review. If non-survey related high-severity benefit inquiries were received, J.D. Power contacted the COR immediately with the respondent’s contact information. Throughout the course of the program, weekly status meetings were held between JDP and BAS to discuss survey administration. Biweekly status meetings were held between the Government Printing Office print vendor, JDP, BAS and VAPSD to discuss the printing and mailing of the survey materials. Non-Response Bias Analysis The purpose of the non-response bias analysis is to ascertain the possible causes of variance in response rates among different respondent demographics and/or determine if any bias has been introduced with a low response rate. Given that the Voice of the Veteran Compensation Access survey had an overall unit response rate of approximately 22% and the Voice of the Veteran Compensation Servicing survey had an overall unit response rate of 25% in FY15, the following section examines whether a low response rate or other factors may have caused respondent bias to occur. The Office of Management and Budget’s Questions and Answers, “When Designing Surveys for Information Collections” dated January 2006, and “Standards and Guidelines for Statistical Surveys” dated September 2006 (see References) provide guidelines on acceptable survey design and response rates. OMB guidelines recommend a non-response bias evaluation for surveys with an overall unit response rate of less than 80%. In addition to the above referenced documents prepared by OMB, J.D. Power assessed other source documents that were written and published by the Federal Committee on Statistical Methodology, “Statistical Policy Working Paper 17, Survey Coverage” (1990) and “Statistical Policy Working Paper 31, Measuring and Reporting Sources of Error in Surveys” (2001). 11 While high response rates are always desirable in surveys, JDP finds an 80% response rate is not achievable for most voluntary, satisfaction-based, survey research studies (Malhotra & Birks, 2007). In particular, survey research studies that do not provide an incentive are subject to not achieving an 80% response rate. To better illustrate this point, the Dillman Method for survey fielding was discussed in Dillman, D. A. (2014, pp. 22), detailing the efforts to attain an 80% response rate. A survey instrument was fielded to 600 students at the University of Washington, the same institution that sponsored the study. After five attempts to solicit a response in a closed university setting, as well as offering a monetary incentive to complete the study, the 80% response rate was not achieved and instead garnered only a 77% response rate. The JDP team met with the VA Contracting officer Representative to discuss current trends and realistic response rates. As noted JDP does not believe that an 80% response rate is achievable and this concern was shared with the Benefits Assistance Service team. JDP conducted the following non-response bias analysis to determine if the respondents (i.e. those who completed the survey) were different in a meaningful way from the nonrespondents (i.e. those who were sent a survey, but did not complete it). Chi-squared analyses consist of comparisons between respondents and non-respondents on available demographic variables such as gender, age, race, geographical region, war participation (service era), and military service branch. The U.S. states were converted to standard USA census regions (Midwest, Northeast, South, and West) in order to aggregate the data and enhance regional comparisons. J.D. Power research indicates that there is an absence of systematic statistical differences of respondents’ overall satisfaction on the mail and online survey results. Research does suggest differences can occur between mixed mode survey methodologies (mail, online, and phone), but these are primarily related to (a) social desirability and interviewer bias associated with phone surveys (see Baum, Chandonnet, Fentress, and Rasinowich, 2012, p. 2, for a review) and (b) that older respondents tend to respond by mail more often than online. The non-response bias analysis was conducted across both mail and online survey collection modes. However, as a verification check, we examined potential differences in mail vs. online survey responses by utilizing a t-test analysis on the OSAT index and advocacy rating which serve as measures of Veterans’ overall satisfaction and benefits advocacy. The overall satisfaction index is defined in the Methodology section of this report. The advocacy rating is defined as Veterans’ likelihood to inform others about VA benefits. 12 Throughout this report, we are conducting statistical analyses to compare survey respondents and non-respondents. Frequently used statistical tests can include the T-Test, Chi-Square, or Analyses of Variance (ANOVA). These tests generate relevant t-statistics, Chi-Squares, or F statistics that are reported. The magnitude of the statistic’s value (either positive or negative) measures the size of the difference relative to the variation in the data. If the statistic is not large enough to generate a probability (p-value) less than .05, then it falls below the accepted standard probability cut-off level that indicates whether a statistical difference is significant. If a difference is not significant, statisticians regard these results as part of the normal sample variation that occurs within the same population. Throughout this report, the probability pvalue standard of “must be less than .05 to be significant” is used for all statistics reported. Table 3a.e shows there were statistical differences found for Compensation Access between the mail and online methodologies on both Overall Satisfaction and Advocacy. The results show that satisfaction was higher for mail respondents whereas advocacy was higher for online respondents. Table 3a.e. Access: T-Test Analysis of Mail vs. Online Survey Results Rating Measure Overall Satisfaction Index (100 - 1000 range) Likelihood to inform others about VA benefits (rating 1 - 4) Mail Online t-statistic p-value 667 651 5.43 <.0001 3.52 3.58 -8.15 <.0001 For the Access sample, significant differences were found with the population based on gender such that the Access sample had a lower percentage of females compared to the population: Table 3b.e. Access: Comparing Gender for Respondents and Non-Respondents Gender by Respondent Type (%) Female Male Survey Respondents NonRespondents Total 7 93 11 89 11 89 Statistic Chi-Square DF 1 Value 502.54 Prob <.0001 For the Access sample, significant differences were found with the population based on age generation, such that a larger number of older Veterans and a fewer number of generation X and YZ Veterans completed the survey: 13 Table 3c.e. Access: Comparing Age Generation for Respondents and Non-Respondents Age Generation by Respondent Type (%) Baby-Boomer (ages 50-68) Generation X (ages 37-49) Generation YZ (ages 18-36) Pre-Boomer (ages 69+) Survey Respondents NonRespondents Total 49 27 32 14 22 20 10 42 35 27 9 13 Statistic DF Value Prob Chi-Square 3 17992 <.0001 For the Access sample, significant differences were found with the population based on race. The differences indicate that survey respondents were less likely to be White or Black, and were more likely to fall into Other category: Table 3d.e. Access: Comparing Race for Respondents and Non-Respondents Race by Respondent Type (%) Asian Black White Other Survey Respondents NonRespondents Total 2 10 46 42 4 17 62 18 4 15 59 22 Statistic Chi-Square DF 3 Value 7100 Prob <.0001 For the Access survey, significant differences were found with the population based on geographical region such that survey respondents were more from the Midwest and less from the South: Table 3e.e. Access: Comparing Census Region for Respondents and Non-Respondents U.S. Census Region by Respondent Type (%) Midwest Northeast South West Survey Respondents NonRespondents Total 27 15 35 23 21 13 41 26 22 13 40 25 Statistic Chi-Square DF 4 Value 817 Prob <.0001 14 For the Access sample, significant differences were found with the population based on branch of service. The effects show that the survey respondents were slightly more likely to be in the Navy and less likely to be in the Marines: Table 3f.e. Access: Comparing Military Service Branch for Respondents and Non-Respondents Military Service Branch by Respondent Type (%) Air Force Army Marines Navy Other Survey Respondents NonRespondents Total 18 48 11 21 2 18 49 15 17 2 18 49 14 18 2 Statistic Chi-Square DF 4 Value 513 Prob <.0001 For the Access survey, significant differences were found in war service era with less surveys returned by Operation Enduring Freedom (OEF) and Operation Iraqi Freedom (OIF) veterans than by other war Veterans: Table 3g.e. Access: Comparing War Participation in OIF and OEF for Respondents and NonRespondents OIF and OEF War Service by Respondent Type (%) Survey Respondents NonRespondents Total 82 18 59 41 64 36 All others OEF/OIF Statistic Chi-Square DF 1 Value 6110 Prob <.0001 Note: OIF is Operation Iraqi Freedom and OEF is Operation Enduring Freedom. For the Access sample, significant differences were found with the population based on Benefit Award. The data shows that more surveys were completed by veterans who receive lesser awards under $500 than those who received $1,501 or more: Table 3h.e. Access: Comparing Benefit Award for Respondents and Non-Respondents Benefit Award by Respondent Type (%) Survey NonRespondents Respondents $500 or less $501-$1000 $1001-$1500 $1501 or more 53 15 13 18 43 17 17 23 Total Statistic Chi-Square DF 3 Value Prob 1086 <.0001 45 17 16 22 15 For the Access sample, significant differences were found with the population based on Disability Entitlement. The data shows that more surveys were completed by Vietnam and Peacetime veterans and less were completed by Gulf War Veterans: Table 3i.e. Access: Comparing Entitlement for Respondents and Non-Respondents Disability Entitlement by Respondent Type (%) Survey NonRespondents Respondents 31 17 46 6 Gulf War Peacetime Vietnam Era Other Total 67 12 19 2 Statistic Chi-Square DF 3 Value 15347 Prob <.0001 60 13 24 3 For the Access sample, significant differences were found with the population based on days of active service, such that survey respondents were more likely to have served 1,000 or less days and less likely to have served 1,001 to 4,000 days compared to the population: Table 3j.e. Access: Comparing Days of Active Service for Respondents and Non-Respondents Days of Active Service by Respondent Type (%) Survey Respondents NonRespondents Total 62 34 40 11 26 23 8 19 17 18 21 20 1000 days or less 1001-2000 days 2001-4000 days 4001 days or more Statistic Chi-Square DF 3 Value 9400 Prob <.0001 For the Access sample, a Chi-square test showed war period differences such that a larger number of Vietnam Veterans and a fewer number of Gulf War Veterans completed the survey: Table 3k.e. Access: Comparing War Period for Respondents and Non-Respondents War Period by Respondent Type (%) Survey NonRespondents Respondents Gulf War Korean Conflict Peacetime Era Vietnam Era World War I & II 31 5 17 46 2 67 1 12 19 1 Total 60 2 13 24 1 Statistic DF Value Prob Chi-Square 4 15390 <.0001 16 For Compensation Servicing, there were significant differences found between mail and online survey respondents. Similar to Access, the results show that satisfaction was higher for mail respondents whereas advocacy was higher for the online respondents: Table 3a.s. Servicing: T-Test Analysis of Mail vs. Online Survey Results Rating Measure Overall Satisfaction Index (100 - 1000 range) Likelihood to inform others about VA benefits (rating 1 - 4) Mail Online t-statistic p-value 636 612 5.44 <.001 3.48 3.54 -4.32 <.001 For the Servicing sample, significant differences were found with gender such that the Servicing sample had a lower percentage of females compared to nonrespondents: Table 3b.s. Servicing: Comparing Gender for Respondents and Non-Respondents Gender by Respondent Type (%) Female Male Survey Respondents NonRespondents Total 6 94 10 90 9 91 Statistic Chi-Square DF 1 Value 140 Prob <.0001 For the Servicing sample, significant differences were found with the population based on age generation, such that a larger number of older Veterans and a fewer number of generation X and YZ Veterans completed the survey: Table 3c.s. Servicing: Comparing Age Generation for Respondents and Non-Respondents Age Generation by Respondent Type (%) Baby-Boomer (ages 50-68) Generation X (ages 37-49) Generation YZ (ages 18-36) Pre-Boomer (ages 69+) Survey Respondents NonRespondents Total 57 44 47 9 20 17 5 17 14 29 19 21 Statistic DF Value Prob Chi-Square 3 2834 <.0001 For the Servicing sample, significant differences were found with the population based on race. The differences indicate that survey respondents were less likely to be White or Black, and more likely to fall into Other category: 17 Table 3d.s. Servicing: Comparing Race for Respondents and Non-Respondents Race by Respondent Type (%) Asian Black Other White Survey Respondents NonRespondents Total 3 11 36 50 4 15 23 58 4 14 26 56 Statistic Chi-Square DF 3 Value 681 Prob <.0001 For the Servicing survey, significant differences were found with the population based on geographical region such that survey respondents were more from the Midwest and West and less from the South: Table 3e.s. Servicing: Comparing Census Region for Respondents and Non-Respondents U.S. Census Region by Respondent Type (%) Midwest Northeast South West Survey Respondents NonRespondents Total 22 18 33 28 20 18 36 26 21 18 35 26 Statistic Chi-Square DF 3 Value 69 Prob <.0001 For the Servicing sample, significant differences were found with the population based on branch of service. The effects show that the survey respondents were slightly more likely to be in the Navy and less likely to be in the Marines: Table 3f.s. Servicing: Comparing Military Service Branch for Respondents and Non-Respondents Military Service Branch by Respondent Type (%) Air Force Army Marines Navy Other Survey Respondents NonRespondents Total 18 53 11 17 1 15 55 13 15 1 16 54 13 16 1 Statistic Chi-Square DF 4 Value 93 Prob <.0001 For the Servicing survey, significant differences were found in war service era with less surveys returned by OEF/OIF veterans than by other war Veterans: 18 Table 3g.s. Servicing: Comparing War Participation in OIF and OEF for Respondents and NonRespondents OIF and OEF War Service by Respondent Type (%) Survey Respondents NonRespondents Total 88 12 75 25 79 21 All others OEF/OIF Statistic Chi-Square DF 1 Value 1136 Prob <.0001 Note: OIF is Operation Iraqi Freedom and OEF is Operation Enduring Freedom. For the Servicing sample, significant differences were found with the population based on Benefit Award. Fewer surveys were completed by veterans who receive awards under $500 than those who received $1,001 or more: Table 3h.s. Servicing: Comparing Benefit Award for Respondents and Non-Respondents Benefit Award by Respondent Type (%) Survey NonRespondent Respondents s $500 or less $501-$1,000 $1,001-$1,500 $1,501 or more 20 18 20 42 Total 22 18 19 41 Statistic Chi-Square DF 3 Value Prob 30 <.0001 21 18 19 42 For the Servicing sample, significant differences were found with the population based on Disability Entitlement. The data shows that more surveys were completed by Vietnam and Peacetime veterans and less were completed by Gulf War Veterans: Table 3i.s. Servicing: Comparing Entitlement for Respondents and Non-Respondents Disability Entitlement by Respondent Type (%) Survey NonRespondents Respondents Gulf War Other Peacetime Vietnam Era 28 5 15 52 47 5 14 34 Total Statistic Chi-Square DF 3 Value 1938 Prob <.0001 42 5 15 38 For the Servicing sample, significant differences were found with the population based on days of active service, such that survey respondents were more likely to have served 1,000 or less days and less likely to have served 1,001 to 4,000 days compared to the population: 19 Table 3j.s. Servicing: Comparing Days of Active Service for Respondents and Non-Respondents Days of Active Service by Respondent Type (%) Survey Respondents NonRespondents Total 63 50 54 8 17 15 7 12 11 23 20 21 1000 days or less 1001-2000 days 2001-4000 days 4001 days or more Statistic Chi-Square DF 3 Value 1307 Prob <.0001 For the Servicing sample, a Chi-square test showed war period differences such that a larger number of Vietnam Veterans completed the survey and fewer numbers of Gulf War Veterans completed the survey: Table 3k.s. Servicing: Comparing War Period for Respondents and Non-Respondents War Period by Respondent Type (%) Survey NonRespondents Respondents Gulf War Korean Conflict Peacetime Era Vietnam Era World War I & II 28 3 15 52 2 47 3 14 34 2 Total 42 3 15 38 2 Statistic DF Value Prob Chi-Square 4 1959 <.0001 3.1 Survey Yield In accordance with OMB “Standards and Guidelines for Statistical Surveys,” an agency must appropriately measure, adjust for, report, and analyze unit and item non-response, when the intended response for a targeted population is not met.1 In assessing Compensation’s data in accordance with Section 3.2, and Guidelines 3.2.1-3.2.3, the unweighted unit response rate was calculated as the ratio of the number of completed cases to the number of in-scope sample cases (Ellis, 2000; AAPOR, 2000). Table 3.1a.e below shows the sample distribution and response rate for the Compensation Access target population: 1 As defined by OMB and FCSM, unit non-response occurs when a respondent fails to respond to all required response items (i.e., fails to fill out or return a data collection instrument); item non-response occurs when a respondent fails to respond to one or more relevant item(s) on a survey 20 Table 3.1a.e. Sample Distribution and Response Rates for Compensation Access Population Total Compensation Access Population FY2015 Total records received Duplicate records in sample file Duplicate record history Invalid Address Invalid Values Blanks Do Not Contact 2 Total records available after cleaning Total records selected Undeliverable addresses Total mailed (excludes undeliverable) Total completed mail surveys Total completed online surveys Total completed surveys 3 Total completed surveys with Overall Index Score 4 Total Sample Response Rate 5 Eligible Sample Response Rate 315,883 3,754 3,743 15,146 20,219 0 2,035 270,986 160,000 6,531 153,469 28,150 8,455 36,605 35,138 21.96% 23.85% 2 Glossary of sample cleaning rules included in Appendix E. Findings in the report are based on the Total Completed Surveys with Overall Index Score (N=35,138). 4 Response rate calculation per OMB Standards and Guidelines for Statistical Surveys, Section 3.2, Guideline 3.2.9 (includes undeliverables as number of non-contacted sample units known to be eligible). 5 Response rate calculation per Council of American Survey Research Organizations (CASRO) (includes number of completed interviews with reporting units/number of eligible reporting units in sample). The American Association for Public Opinion Research (APPOR) also uses this method for calculation and cites CASRO (APPOR Standard Definitions, 2008, pp. 34). 3 21 Table 3.1a.s below shows the sample distribution and response rate for Compensation Servicing target population: Table 3.1a.s. Sample Distribution and Response Rates for Compensation Servicing Population Total Compensation Servicing Population FY2015 Total records received Duplicate records in sample file Duplicate record history Invalid address Invalid values Blanks Do not contact 6 Total records available after cleaning Total records selected Undeliverable addresses Total mailed (excludes undeliverable) Total completed mail surveys Total completed online surveys Total completed surveys 7 Total completed surveys with Overall Index Score 8 Total Sample Response Rate 9 Eligible Sample Response Rate 449,568 7,479 34,557 14,586 11,405 0 552 380,989 60,000 1,833 58,167 11,962 4,068 16,030 15,015 25.03% 27.56% Of the 315,883 total records received from Access, 44,897 records were purged from the sample due to cleaning rules such as duplicate records, invalid addresses and values, blanks, and do not contact opt outs. From the 44,897records purged, 3,743 records were purged due to duplicate records across VBA’s other business line surveys (i.e. duplicate record history). In Servicing we received a total of 315,883 records but we purged 68,579 records from the sample due to cleaning rules such as duplicate records, invalid addresses and values, blanks, and do not contact opt outs. Also, from the 68,579 records that were purged, there were 34,557 records that were cleaned due to duplicate records across other business lines. 6 Glossary of sample cleaning rules included in Appendix E. Findings in the report are based on the Total Completed Surveys with Overall Index Score (N=15,015). 8 Response rate calculation per OMB Standards and Guidelines for Statistical Surveys, Section 3.2, Guideline 3.2.9 (includes undeliverables as number of noncontacted sample units known to be eligible). 9 Response rate calculation per Council of American Survey Research Organizations (CASRO) (includes number of completed interviews with reporting units/number of eligible reporting units in sample). The American Association for Public Opinion Research (APPOR) also uses this method for calculation and cites CASRO (APPOR Standard Definitions, 2008, pp. 34). 7 22 The purpose of the cleaning rules is to prevent respondents from being re-contacted if they were previously selected to participate in any of VBA’s business line surveys in the past 12 months. The cleaning rule is a JDP and survey research best practice and is intended to promote proper conduct in market research. About 14% of the total records provided for Access and about 15% of the total records provided for Servicing were removed from the sample due to these cleaning rules. It is unlikely that the cleaning rules impacted the unit non-response and we were able to secure the designated number of records for both Servicing and Access. Table 3.1b.e. Access: Weight/Person for Completed Surveys per Population Completed Surveys 36,605 Access 2015 Population 315,883 Weight/Person 9 In Table 3.1b.e, the 9 in the Weight/Person column means that every survey completed/returned represents the views of 9 Veterans using Compensation Access-benefits. This was calculated by dividing the number of completed surveys into the population number. Table 3.1b.s. Servicing: Weight/Person for Completed Surveys per Population Completed Surveys 16,030 Servicing 2015 Population 449,568 Weight/Person 28 In Table 3.1b.s, the 28 in the Weight/Person column means that every survey completed and returned represents the views of 28 Veterans using Compensation Servicing-benefits. This was calculated by dividing the number of completed surveys into the population number. To confirm the sample’s representativeness for both Access and Servicing, a comparison was conducted among the total records provided and the records available after cleaning. The intent of this analysis was to determine whether the cleaning rules caused the remaining sample to vary in a meaningful way from the original sampling frame. Table 3.1c.e (Access) and Table 3.1c.s (Servicing) indicate characteristics such as Gender, Age, and Geographical Region are similar among the total records provided and the records available after cleaning. Comparisons by state yield differences that are mostly less than 1% point, with a few exceptions in Access where differences were wider for certain age ranges. Overall, these comparisons suggest the cleaning rules did not significantly alter the proportion of respondent characteristics provided in the original sampling frame. 23 Table 3.1c. Access: Comparing Gender, Generation, and U.S. States to Total Population Gender Female Male Generation Baby Boomer Generation X Generation YZ Pre-Boomer U.S. State AK AL AR AZ CA CO CT DC DE FL GA HI IA ID IL IN KS KY LA MA MD ME MI MN MO MS MT NC ND NE NH NJ NM NV NY OH Total Population (%) Records Available (%) % Point Difference 10.66 89.34 11.1 88.9 0.44 -0.44 31.68 19.64 33.59 15.09 30.88 21.04 35.76 12.32 -0.8 1.4 2.17 -2.77 0.47 1.82 0.85 1.81 10.31 2.43 0.63 0.11 0.26 6.60 4.66 0.72 0.86 0.57 2.04 2.10 0.87 1.22 1.48 1.51 1.82 0.39 2.31 2.03 2.24 1.00 0.29 4.39 0.23 0.80 0.38 1.13 0.69 1.19 2.79 2.96 0.5 1.84 0.86 1.86 10.39 2.5 0.63 0.11 0.26 6.7 4.61 0.72 0.86 0.59 2.01 2.01 0.87 1.21 1.47 1.44 1.86 0.41 2.13 1.86 2.22 1.03 0.3 4.42 0.24 0.78 0.38 1.08 0.72 1.25 2.76 2.91 0.03 0.02 0.01 0.05 0.09 0.07 -0.01 0 0 0.1 -0.05 -0.01 0 0.03 -0.03 -0.09 0 -0.01 0 -0.07 0.03 0.01 -0.17 -0.17 -0.02 0.03 0.02 0.03 0.01 -0.02 -0.01 -0.06 0.02 0.06 -0.03 -0.06 24 Table 3.1c. Access: Comparing Gender, Generation, and U.S. States to Total Population (Continued) OK OR PA RI SC SD TN TX UT VA VT WA WI WV WY Total Population (%) Records Available (%) % Point Difference 2.37 1.61 2.76 0.27 2.69 0.30 2.87 10.40 0.69 4.01 0.13 3.00 1.22 0.67 0.28 2.34 1.61 2.68 0.27 2.65 0.32 2.87 10.62 0.7 4.07 0.13 3.05 1.21 0.69 0.3 -0.03 0 -0.07 0 -0.04 0.01 0 0.21 0.02 0.06 0 0.05 -0.01 0.02 0.02 Table 3.1c.s. Servicing: Comparing Gender, Generation, and U.S. States to Total Population Gender Female Male Generation Baby Boomer Generation X Generation YZ Pre-Boomer U.S. State AK AL AR AZ CA CO CT DC DE FL GA HI Total Population (%) Records Available (%) % Point Difference 10.15 89.85 9.41 90.59 -0.74 0.74 47.65 16.89 13.86 21.61 48.15 16.6 13.35 21.91 0.5 -0.29 -0.51 0.3 0.35 2.09 1.03 2.04 8.87 1.94 0.57 0.09 0.26 6.98 4.33 0.65 0.35 1.95 1.02 2.01 9.06 1.94 0.53 0.08 0.26 7.29 4.29 0.66 0 -0.14 -0.01 -0.03 0.19 0 -0.04 -0.01 0 0.31 -0.04 0 25 Table 3.1c.s Servicing: Comparing Gender, Generation, and U.S. States to Total Population (Continued) IA ID IL IN KS KY LA MA MD ME MI MN MO MS MT NC ND NE NH NJ NM NV NY OH OK OR PA RI SC SD TN TX UT VA VT WA WI WV WY Total Population (%) Records Available (%) % Point Difference 1.09 0.71 2.22 1.72 0.96 1.47 1.7 1.53 1.24 0.55 2.76 2.09 2.63 1.18 0.47 4.69 0.34 0.89 0.44 1.4 0.94 1.06 3.05 3.01 2.6 1.61 2.73 0.31 2.76 0.44 3.02 8.97 0.72 2.99 0.2 2.51 1.35 1.11 0.25 1.08 0.71 2.21 1.64 0.96 1.46 1.64 1.52 1.16 0.56 2.7 2.07 2.64 1.15 0.49 4.94 0.35 0.92 0.43 1.34 0.92 1.03 2.99 3 2.6 1.59 2.65 0.31 2.69 0.46 3.04 9.26 0.71 3.02 0.2 2.61 1.28 1.12 0.25 -0.01 0 -0.01 -0.08 0 -0.01 -0.05 0 -0.08 0.01 -0.06 -0.02 0.01 -0.03 0.01 0.25 0.01 0.03 -0.01 -0.06 -0.02 -0.02 -0.06 -0.01 -0.01 -0.02 -0.08 -0.01 -0.07 0.02 0.02 0.29 -0.01 0.02 0 0.1 -0.07 0.01 0 26 3.2 Missing Data Patterns and Mechanisms In accordance with the OMB “Standards and Guidelines for Statistical Surveys” Guidelines 3.2.9 and 3.2.11, an investigation of missing data patterns was conducted on the 36,605 total surveys received for Access and 16,030 total surveys received for Servicing. In order to assess the distribution of missing data, a procedure was performed to process missing values involving iterative multiple imputation chains using expectation–maximization (MCMC) algorithms and divide these into distribution interval groupings, Pierchala, Carl E. (2001). This was done on the key measures of the Overall Satisfaction Index (see Appendix A for calculation) and Advocacy ratings related to Veterans’ likelihood to recommend VA benefits. As shown in Tables 3.2.e and 3.2.s for Access and Servicing respectively, there were no indications of unusual patterns for missing data. For more discussion of missing data mechanisms (MCAR, MAR, and MNAR), please see Appendix A. Table 3.2.e. Access: Missing Data Patterns in Satisfaction and Advocacy (0 = missing, 1 = data) Group Means Group 1 2 3 4 Overall Satisfaction Likelihood to inform others Freq Percent OSAT Index Age % Male 0 0 1 1 0 1 0 1 298 843 425 30764 1% 3% 1% 95% 656 667 654 663 62 67 64 60 91% 95% 94% 93% Table 3.2.s. Servicing: Missing Data Patterns in Satisfaction and Advocacy (0 = missing, 1 = data) Group Means Group Overall Satisfaction Likelihood to inform others Freq Percent OSAT Index Age % Male 0 0 1 1 0 1 0 1 37 322 251 14402 0% 2% 2% 96% 587 627 612 630 76 69 66 63 91% 95% 90% 94% 0 0 1 1 27 3.3 Margin of Error The margin of error expresses the maximum expected difference between the true population parameter and a sample estimate of that parameter. It is often used to indicate the accuracy of survey results. The larger the margin of error around an estimated value, the less accurate the estimated value will be. Larger samples are more likely to yield results close to the true population quantity and thus have smaller margins of error than smaller samples. Based on a sample of 153,469 Veterans, the Overall Satisfaction Index for the Access study is 662 and has a margin of error of 2 index points, on a 1,000 point scale, at the 95% confidence level. This indicates that if the survey were repeated many times with different samples, the true mean Overall Satisfaction Index would fall within 2 index points 95% of the time. Table 3.3.e below demonstrates relative decreases in margin of error as the study sample size increases. A 30% response rate (46,041 completes) would be associated with a margin of error of 2 index points, similar to the margin of error for a 40% response rate (61,388 completes). Results from this analysis indicate the Overall Satisfaction Index (OSAT) calculated from the Access study is an accurate measurement of the true population mean. Table 3.3.e. Access: Margin of Error for Larger Sample Sizes Sample Response Rate Completes (N) OSAT (mean) Standard Deviation Standard Error Margin of error (95% confidence interval) 153,469 153,469 153,469 153,469 153,469 153,469 153,469 23.85% 20% 30% 40% 50% 60% 80% 36,605 30,694 46,041 61,388 76,735 92,081 122,775 662 662 662 662 662 662 662 213 213 213 213 213 213 213 1.1 1.2 1.0 0.9 0.8 0.7 0.6 2 2 2 2 2 1 1 Based on a sample of 58,167 Veterans, the FY15 Overall Satisfaction Index for the Servicing study is 630 and has a margin of error of 3 index points, on a 1,000 point scale, at the 95% confidence level. This indicates that if the survey were repeated many times with different samples, the true mean Overall Satisfaction Index would fall within 3 index points 95% of the time. Table 3.3.s below demonstrates relative decreases in margin of error as the study sample size increases. A 30% response rate (17,450 completes) would be associated with a margin of error of 3 index points, similar to the margin of error for a 40% response rate (23,267 completes). Results from this analysis indicate the Overall Satisfaction Index (OSAT) calculated from the Servicing study is an accurate measurement of the true population mean, which is reported on a 1,000 point scale. 28 Table 3.3.s. Servicing: Margin of Error for Larger Sample Sizes Sample Response Rate Completes (N) OSAT (mean) Standard Deviation Standard Error Margin of error (95% confidence interval) 58,167 58,167 58,167 58,167 58,167 58,167 58,167 27.56% 20% 30% 40% 50% 60% 80% 16,030 11,633 17,450 23,267 29,084 34,900 46,534 630 630 630 630 630 630 630 222 222 222 222 222 222 222 1.8 2.1 1.7 1.5 1.3 1.2 1.0 3 4 3 3 3 2 2 In the margin of error analysis noted on the previous page and in subsequent analyses included in this report, the Overall Satisfaction Index Score is the main dependent variable and is the basis for the analysis. The Overall Satisfaction Index score is the survey metric that VBA utilizes to measure customer satisfaction and benchmark performance against other industries. It is the primary measurement in all reports. The Overall Satisfaction Index encompasses all aspects of the customer experience10, and can therefore be used as a reliable indicator for the presence or absence of respondent bias in the survey results as a whole. For these reasons, the Overall Satisfaction Index score is used as the main dependent variable in the margin of error analysis and subsequent t-test analyses included in this report. 3.3.1 Sampling Distribution Respondent characteristics such as gender and age were compared to that of the total sample to determine whether respondents and non-respondents differed on key variables of interest. Compared to the population of all eligible respondents (Access 160,000, Servicing 60,000), survey respondents demonstrate the same gender characteristics. For Access, Table 3.3.1.e below illustrates 7% of survey respondents were female and 93% were male, similar to the total sample population. The distribution of age shows that survey respondents tend to be older. 10 Explanation of J.D. Power Index Model Calculation included in Methodology. 29 Table 3.3.1.e. Access: Comparing Gender and Age of Survey Respondents to Total Sample Gender Female Male Age Generation Baby Boomer Generation X Generation YZ Pre-Boomer Respondents (%) Sample Size (N) Total Sample (%) Sample Size (N) % Point Difference 7 93 2209 29662 11 89 16364 138938 4 -4 48 14 10 28 16286 4608 3341 9420 32 20 35 13 50676 31938 56152 21234 -17 6 25 -15 For Servicing, Table 3.3.1.s below illustrates 6% of survey respondents were female and 94% were male, similar to the total sample population. The distribution of age shows that survey respondents tend to be older. Table 3.3.1.s. Servicing: Comparing Gender and Age of Survey Respondents to the Total Sample Gender Female Male Age Generation Baby Boomer Generation X Generation YZ Pre-Boomer Respondents (%) Sample Size (N) Total Sample (%) Sample Size (N) % Point Difference 6 94 973 14474 9 91 5048 53220 2 -2 56 9 5 30 8959 1447 747 4874 47 17 14 22 28384 10169 8276 13171 -9 8 9 -8 3.3.2 Distribution of Overall Satisfaction Index Scores Following the comparison of sampling distributions, a comparison of Overall Satisfaction Index scores was conducted to determine whether differences in age and gender among respondents correlate with differences in overall satisfaction. For Access, Table 3.3.2.e below indicates differences in Overall Satisfaction Index scores are notable between gender groups. On average, females tend to rate their experience 14 index points lower than males (648 vs. 662). Comparing age groups reveals that Pre-Boomer had the highest overall satisfaction with Generation YZ much lower. 30 Table 3.3.2.e. Access: Overall Satisfaction Scores for Gender and Age Groups Characteristics Gender Female Male Age Generation Baby Boomer Generation X Generation YZ Pre-Boomer OSAT (mean) Standard Deviation Sample Size (N) 648 662 215 214 2164 28469 658 642 634 692 214 224 218 202 15803 4544 3281 8702 For Servicing, Table 3.3.2.s below indicates differences in Overall Satisfaction Index scores are notable between gender groups. On average, females tend to rate their experience 5 index points higher than males (633 vs. 628). Comparing age groups reveals that Pre-Boomer had the highest overall satisfaction with Generation XYZ much lower. Table 3.3.2.s. Servicing: Overall Satisfaction Scores for Gender and Age Groups Characteristics Gender Female Male Age Generation Baby Boomer Generation X Generation YZ Pre-Boomer OSAT (mean) Standard Deviation Sample Size (N) 633 628 225 223 914 13569 624 596 593 658 223 229 225 215 8530 1405 726 4351 3.3.3 Analysis for Demographic Differences T-test analyses were conducted to determine whether differences in demographic groups produced statistical differences in Overall Satisfaction (OSAT) scores. T-tests are typically used to determine whether or not the difference between two groups’ averages most likely reflect a meaningful difference in the population from which the groups were sampled. For Access, both gender and war participation differences were significantly different such that males and non-OEF/OIF veterans had higher levels of overall satisfaction: 31 Table 3.3.3a.e. Access: T-Test Analysis for Pairs of Characteristics in Veterans’ Satisfaction Characteristics Gender Female vs. Male War Participation OEF/OIF vs. All Others T-Test Statistic Statistical Difference (95% confidence level) -2.95 Yes 7.22 Yes For Servicing, the differences for gender and war participation were both statistically significant: Table 3.3.3a.s. Servicing: T-Test Analysis for Pairs of Characteristics in Veterans’ Satisfaction Characteristics Gender Female vs. Male War Participation OEF/OIF vs. All Others T-Test Statistic Statistical Difference (95% confidence level) 0.66 No 1.97 Yes Analyses of Variance (ANOVA) were conducted to determine whether differences in demographic groups produced statistical differences in Overall Satisfaction scores. This analysis is typically used to determine whether or not the difference among the average of three or more groups most likely reflects a meaningful difference in the population from which the groups were sampled. For Access, differences in Overall Satisfaction by generation were significant (F = 92.30, p-value < .0001) such that older survey respondents had higher levels of satisfaction: Table 3.3.3b.e. Access: Overall Satisfaction for Generation Generation OSAT (mean) Sample Size (N) Baby Boomer Generation X Generation YZ Pre-Boomer 658 642 634 692 15803 4544 3281 8702 For Access, differences in Overall Satisfaction by region were significant (F = 89.03, p-value < .0001) such that the Midwest respondents had the highest levels of satisfaction: 32 Table 3.3.3c.e. Access: Overall Satisfaction for Regions Regions OSAT (mean) Sample Size (N) Midwest Northeast South West 688 680 644 651 8556 4776 11243 7498 For Access, racial differences in Overall Satisfaction were not significant (F = 100.56, p-value < .0001): Table 3.3.3d.e. Access: Overall Satisfaction for Race Race OSAT (mean) Sample Size (N) Asian Black Other White 684 638 683 635 587 2405 10370 11207 For Access, differences in Overall Satisfaction by branch of service were significant (F = 5.95, pvalue < .0001) such that respondents from the Army had the highest levels of satisfaction: Table 3.3.3e.e. Access: Overall Satisfaction for Military Service Branches Military Service Branches OSAT (mean) Sample Size (N) Air Force Army Marines Navy Other 659 667 659 662 629 5914 15378 3633 6798 607 For Access, differences in Overall Satisfaction by Benefit Award were significant (F = 283.72, pvalue < .0001) such that respondents receiving the highest awards had the highest levels of satisfaction: Table 3.3.3f.e. Access: Overall Satisfaction for Benefit Award Benefit Award OSAT (mean) Sample Size (N) $500 or less $501-$1000 $1001-$1500 $1501 or more 635 665 691 722 17252 4882 4289 5904 33 For Access, differences in Overall Satisfaction by Entitlement were significant (F = 106.26, pvalue < .0001) such that respondents in the Other category had the highest levels of satisfaction: Table 3.3.3g.e. Access: Overall Satisfaction for Entitlement Entitlement OSAT (mean) Sample Size (N) Gulf War Disability Other Peacetime Disability Vietnam Era Disability 638 718 656 674 9894 2018 5575 14843 For Access, differences in Overall Satisfaction by days of active service were significant (F = 81.17, p-value < .0001) such that respondents who had “1000 days or less” had the highest levels of satisfaction: Table 3.3.3h.e. Access: Overall Satisfaction for Days of Active Service Days of Active Service 1,000 days or less 1,001-2,000 days 2,001-4,000 days 4,001 days or more OSAT (mean) Sample Size (N) 677 638 629 645 20,148 3,637 2,717 5,828 For Access, differences in Overall Satisfaction by War Period were significant (F = 79.83, p-value < .0001) such that respondents from the Korean and World War conflicts had the highest levels of satisfaction: Table 3.3.3i.e. Access: Overall Satisfaction for War Period War Period Gulf War Korean Conflict Peacetime Era Vietnam Era World War I & II OSAT (mean) Sample Size (N) 638 718 656 674 718 9896 1473 5575 14844 542 For Servicing, differences in Overall Satisfaction by generation were significant (F = 43.58, pvalue < .001) such that older respondents had the highest levels of satisfaction: 34 Table 3.3.3b.s. Servicing: Overall Satisfaction for Generation Generation OSAT (mean) Sample Size (N) Baby Boomer Generation X Generation YZ Pre-Boomer 624 596 593 658 8530 1405 726 4351 For Servicing, differences in Overall Satisfaction by region were significant (F = 51.26, p-value < .0001) such that respondents from the Northeast had the highest levels of satisfaction: Table 3.3.3c.s. Servicing: Overall Satisfaction for Regions Regions OSAT (mean) Sample Size (N) Midwest Northeast South West 639 673 614 612 3259 2593 4788 4040 For Servicing, racial differences in Overall Satisfaction were significant (F = 28.23, p-value < .001) such that Asian respondents had the highest levels: Table 3.3.3d.s. Servicing: Overall Satisfaction for Race Race OSAT (mean) Sample Size (N) Asian Black Other White 641 595 650 609 306 1095 3444 4791 For Servicing, differences in Overall Satisfaction by branch of service were not significant (F = 0.63, p-value = 0.6398): Table 3.3.3e.s. Servicing: Overall Satisfaction for Military Service Branches Military Service OSAT (mean) Sample Size (N) Air Force Army Marines Navy Other 628 631 627 631 610 2629 7944 1698 2529 212 35 For Servicing, differences in Overall Satisfaction by Benefit Award were significant (F = 207.51, pvalue < .0001) such that those respondents with the highest awards had the highest levels of satisfaction: Table 3.3.3f.s. Servicing: Overall Satisfaction for Benefit Award Benefit Award OSAT (mean) Sample Size (N) $500 or less $501-$1000 $1001-$1500 $1501 or more 574 589 618 679 2945 2761 2981 6325 For Servicing, differences in Overall Satisfaction by Entitlement were significant (F = 52.45, pvalue < .0001) such that respondents in the Other category had the highest levels: Table 3.3.3g.s. Servicing: Overall Satisfaction for Entitlement Entitlement OSAT (mean) Sample Size (N) Gulf War Disability Other Peacetime Disability Vietnam Era Disability 609 675 599 646 4169 813 2299 7731 For Servicing, differences in Overall Satisfaction by days of active service were significant (F = 37.38, p-value <.0001) such that respondents with “1000 days or less” had the highest levels: Table 3.3.3h.s. Servicing: Overall Satisfaction for Days of Active Service Days of Active Service 1000 days or less 1001-2000 days 2001-4000 days 4001 days or more OSAT (mean) Sample Size (N) 641 582 592 626 9403 1197 1029 3383 For Servicing, differences in Overall Satisfaction by War Period were significant (F = 38.95, pvalue < .0001) such that respondents from the Korean and World War conflicts had the highest levels: 36 Table 3.3.3i.s. Servicing: Overall Satisfaction for War Period War Period OSAT (mean) Sample Size (N) 609 668 599 646 684 4182 480 2303 7737 310 Gulf War Korean Conflict Peacetime Era Vietnam Era World War I & II 3.3.4 Data Imputation Analysis for Demographic Differences A pairwise comparison t-test analysis was done to evaluate whether data imputation for missing values across significant demographic differences shown in section 3.3.3 would impact Overall Satisfaction Index Scores. This analysis included survey raking across demographic differences as one level of comparison. These results (Tables 3.3.4a.e and 3.3.4a.s) show that there were no significant differences between the non-imputed mean and the imputed mean of the satisfaction index across demographics, sample sizes, nor survey raked values. We want to highlight that after statistical adjustment for the differences found between respondents and non-respondents reported earlier, there were no differences in overall satisfaction levels. These results support the conclusion that the survey’s findings for Veterans’ overall satisfaction ratings are accurate. Table 3.3.4a.e. Access: Comparison of Imputed vs. Non-Imputed on Veterans’ Satisfaction T-Test Analysis on Imputed vs. Non-Imputed raked for Demographic Differences Overall Satisfaction Index (100 - 1000 range) Imputed demographics (32,330 final sample size) Imputed survey-raked demographics (32,330 final sample size) Imputed survey-raked demographics (33,655 total respondents) mean (imputed) mean (nonimputed) t-statistic p-value 662.70 662.65 -0.03 0.97 653.46 653.60 0.08 0.94 654.23 653.32 -0.54 0.59 Note: Non-imputed is based on the 32,330 final cleaned sample size used in this report. 37 Table 3.3.4a.s. Servicing: Comparison of Imputed vs. Non-Imputed on Veterans’ Satisfaction T-Test Analysis on Imputed vs. Non-Imputed raked for Demographic Differences Overall Satisfaction Index (100 - 1000 range) Imputed demographics (15,012 final sample size) Imputed survey-raked demographics (15,012 final sample size) Imputed survey-raked demographics (16,027 total respondents) mean (imputed) mean (nonimputed) t-statistic p-value 630.52 629.77 -0.29 0.77 622.34 621.65 -0.27 0.79 623.30 621.19 -0.83 0.40 Note: Non-imputed is based on the 15,012 final cleaned sample size used in this report. Survey Raking for Sample Weights to Adjust for Differences and Compare Overall Satisfaction and Advocacy Ratings The procedure known as “raking” adjusts a set of data so that its marginal totals match specified control totals on a specified set of variables. The term suggests an analogy with the process of smoothing the soil in a garden plot by alternately working it back and forth with a rake in two perpendicular directions (Izrael and Battaglia (2004)). Survey raking is an iterative sample-balancing algorithm-based technique that provides sample weighting convergence across multiple variables and multiple categories (Battaglia, Izrael, Hoaglin, and Frankel (2009)). In keeping with OMB “Standards and Guidelines for Statistical Surveys” Guidelines 3.2.12 and 3.2.13, JDP selected the best statistical method to simultaneously adjust for multiple differences between groups by applying a survey raking procedure, see Anderson, L., and R.D. Fricker, Jr. (2015). The JDP raking procedure is a proprietary improved version based on the excellent methods initially developed by Izrael and Battaglia (2000, 2004) and Battaglia, Izrael, Hoaglin, and Frankel (2004). JDP raking improvements are primarily related to better handling of low cell values during iterative convergence processing. For this analysis, 50 iterations were set (although fewer were needed) to converge on the best sample weights (.2 estimation margin) to simultaneously adjust for non-response bias in age, race, region, and war (service era) demographic categories. For additional background about survey raking methodologies, see Wallace and Rust (1996). The estimated population distributions are used as convergence targets. In this case, the dataset of all eligible respondents for Access (160,000) and Servicing (60,000) was used as the estimated population to derive sample weightings for the Access survey respondents (36,605) and the Servicing survey respondents (16,030). 38 In accordance with OMB “Standards and Guidelines for Statistical Surveys” Guideline 3.2.13, a series of t-tests were conducted to determine whether non-response bias in demographic areas produced statistical differences in Overall Satisfaction Index scores and Advocacy ratings. Typically, t-tests are used to determine whether differences between the averages and variances of two groups reflect a meaningful difference in the population. The sample weightings derived from the survey raking procedure were included in the t-tests to equalize the survey respondent differences with non-respondents. For Access, there were significant differences in Overall Satisfaction Index scores and Advocacy when the data was adjusted for demographic differences between survey respondents and non-respondents. However, the actual differences in ratings are small (663 vs 654 for OSAT, and 3.53 and 3.55 for Advocacy). The effect size for Overall Satisfaction as measured by Cohen’s D = .04 is considered less than a “small” effect (Cohen’s D = 0.20). Likewise, the effect size for advocacy was also very small (Cohen’s D = -0.03). This suggests that the statistical significance was magnified by the large sample numbers. In conclusion, we would point to the findings reported in Table 3.3.4a.e, where the overall results support the conclusion that the survey’s findings for Veterans’ overall satisfaction ratings are accurate. Table 3.3.4b.e. Access: Overall Satisfaction and Advocacy for Survey Respondents Unweighted and Weighted Analysis of Survey Respondent Scores with Weighted Adjustment for Non-Response Bias Standard Standard Rating Mean Mean tpDeviation Deviation Measure (Unweighted) (Weighted) statistic value (Unweighted) (Weighted) Overall Satisfaction Index (100 1000 range) Likelihood to inform others about VA benefits (rating 1 - 4) 663 654 213 218 5.33 <.001 3.53 3.55 .62 .62 -3.59 <.001 For Servicing, there were significant differences in Overall Satisfaction but not advocacy when the data was adjusted for demographic differences between survey respondents and nonrespondents. However, the actual differences in Satisfaction index are small (630 vs 622). The effect size for Overall Satisfaction as measured by Cohen’s D = .04 is considered less than a “small” effect (Cohen’s D = 0.20). This suggests that the statistical significance was magnified by the large sample numbers. In conclusion, we would point to the findings reported in Table 3.3.4a.s, where the overall results support the conclusion that the survey’s findings for Veterans’ overall satisfaction ratings are accurate. 39 Table 3.3.4b.s. Servicing: Overall Satisfaction and Advocacy for Survey Respondents Unweighted and Weighted Analysis of Survey Respondent Scores with Weighted Adjustment for Non-Response Bias Standard Standard Rating Mean Mean tpDeviation Deviation Measure (Unweighted) (Weighted) statistic value (Unweighted) (Weighted) Overall Satisfaction Index (100 1000 range) Likelihood to inform others about VA benefits (rating 1 - 4) 630 622 222 224 3.15 <.01 3.50 3.50 .66 .66 -.90 .37 Findings Results from the non-response bias analysis indicate that the Overall Customer Satisfaction Index score and the Advocacy ratings from the Compensation study reflects the experience of all Veterans and beneficiaries who: 1) received a decision for their application for compensation benefits with the past 30 days, 2) may include individuals who were found eligible on a new or subsequent claim, 3) those who have been denied and are not appealing the decision and 4) those who began receiving benefits six to eighteen months ago Sample Cleaning: Initial comparisons on Age, Gender, and Geographical characteristics between the total records provided and the records available after cleaning (see Survey Yield, Section 3.1), suggests the sample utilized in the study exhibits similar characteristics as the total sample. Additional comparisons (see Margin of Error and Sampling Distribution, Section 3.3,) suggest the sample cleaning rules did not impact the sample’s representativeness and the results are conclusive. Non-Response Bias Analysis: Results from the non-response bias analysis did show group differences for age, gender, region, race, benefit award, entitlement type, military branch, days of service, and war participation between survey respondents and non-respondents. After correcting for these differences using a recommended sample-balancing survey raking method to derive sample weights (see Margin of Error, Section 3.3.4 Data Imputation Analysis for Demographic Variables), no differences were found in the Overall Satisfaction Index scores and Advocacy ratings (likelihood inform others about VA benefits). Some differences were found between weighted and unweighted survey indices but the effects were shown to be small and the statistical significance was due to the large sample numbers. 40 Item Response Rate Calculations: Results from the survey item response rate calculations indicate high item response rates, with none falling below OMB guidelines (see Appendix B for Item Response Rates). According to OMB Guideline 3.2.10, given that neither study had a response rate lower than 70%, a non-response bias analysis was not necessary at the item level. The research and approach taken by JDP are in accordance with sound market research and current best practices from the American Association for Public Opinion Research (AAPOR) regarding response rate recommendations: “Results that show the least bias have turned out, in some cases, to come from surveys with less than optimal response rates. Experimental comparisons have also revealed few significant differences between estimates from surveys with low response rates and short field periods and surveys with high response rates and long field periods.” See AAPOR “Response Rates – An Overview” (2015) and Special Issue of Public Opinion Quarterly "Nonresponse Bias in Household Surveys" (Singer, 2006). Conclusion The Overall Satisfaction Index score and Advocacy rating (likelihood to inform others about VA benefits) are not impacted in any meaningful way by non-response bias. This analysis confirms that the data collected during Fiscal Year 2015 is valid. The FY15 Voice of the Veteran Line of Business Tracking Satisfaction Study data for the Compensation Access and Servicing surveys can be used to infer reliable overall customer satisfaction scores and advocacy ratings. The overall customer satisfaction index score reflects the experience of all Veterans and beneficiaries who: 1) received a decision for their application for compensation benefits with the past 30 days, 2) may include individuals who were found eligible on a new or subsequent claim, 3) those who have been denied and are not appealing the decision and 4) those who began receiving benefits six to eighteen months ago. The sample utilized in the study exhibits similar characteristics for age, gender, and geographical characteristics as the total sample provided. This indicates the sample cleaning rules did not impact the sample’s representativeness. While the results from the non-response bias analysis did show group differences on most of the demographic characteristics between survey respondents and non-respondents, there were no differences found in Veterans’ overall satisfaction and advocacy (likelihood to inform others about VA benefits). This was after correcting for these differences using a recommended sample-balancing survey raking method to derive sample weights. Some differences were found between weighted and unweighted survey indices but the effects were shown to be small and the statistical significance was due to the large sample numbers. JDP conducted all necessary statistical tests in accordance with OMB standards. J.D. Power certifies the results contained within this report. 41 References Anderson, L., and R.D. Fricker, Jr. (2015). Raking: An Important and Often Overlooked Survey Analysis Tool, Phalanx, http://faculty.nps.edu/rdfricke/docs/Analysis%20process_v4.pdf American Association for Public Opinion Research (2008). Standard Definitions: Final Disposition of Case Codes and Outcome Rates for Surveys. Ann Arbor, Michigan: AAPOR. (http://www.aapor.org/AAPORKentico/AAPOR_Main/media/MainSiteFiles/Standard_Definitions_07 _08_Final.pdf). American Association for Public Opinion Research (2015). “Response Rates – An Overview” http://www.aapor.org/AAPORKentico/Education-Resources/For-Researchers/Poll-SurveyFAQ/Response-Rates-An-Overview.aspx Battaglia, Michael P., Izrael, David, Hoaglin, David C., and Frankel, Martin R. (2004), “To Rake or Not To Rake Is Not the Question Anymore with the Enhanced Raking Macro.” Proceedings of the 29th Annual SAS Users Group International Conference, Paper 207. Battaglia, Michael P., Izrael, David, Hoaglin, David C., and Frankel, Martin R. (2009). Practical Considerations in Raking Survey Data. Survey Practice, Vol 2, No. 5. Baum, Herbert M., Ph.D.; Chandonnet, Anna M.A.; Fentress, Jack M.S., M.B.A.; and Rasinowich, Colleen, B.A. (2012). “Mixed-Mode Methods for Conducting Survey Research”. Data Recognition Corporation. http://www.datarecognitioncorp.com/survey-services/Documents/Mixed-ModeMethods-for-Conducting-Survey-Research.pdf Dillman, D. A. and J.D. Power (2015), Conference call discussion on non-response bias, avoidance methods, and post-hoc sample weighting between Dr. Dillman and JDP (Greg Truex, Jay Meyers, Ph.D., Lee Quintanar, Ph.D.), May 20, 2015 (2pm PDT). Dillman, D. A. (2014). Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method. Fourth Edition. John Wiley & Sons, Inc: New York. Ellis, J. M. (2000). Estimating the Number of Eligible Respondents for a Telephone Survey of LowIncidence Households. Paper presented at the annual meeting of the American Association for Public Opinion Research, Portland OR, May 21, 2000. Federal Committee on Statistical Methodology’s Statistical Policy Working Paper 31, Measuring and Reporting Sources of Error in Surveys (2001). Washington, D.C. Izrael, David, Hoaglin, David C., and Battaglia, Michael P. (2000), “A SAS Macro for Balancing a Weighted Sample.” Proceedings of the Twenty-Fifth Annual SAS Users Group International Conference, Paper 275. Izrael, David, Hoaglin, David C., and Battaglia, Michael P. (2004), “Tips and Tricks for Raking Survey Data (a.k.a. Sample Balancing).” Proceedings of the 2004 American Association for Public Opinion Research (AAPOR) Conference. 42 Malhotra, N.K, and Birks, D.F. (2007). Marketing Research: An Applied Approach, 3rd edition. Prentice Hall/Financial Times: England. Pierchala, Carl E. (2001). PROC MI® as the Basis for a Macro for the Study of Patterns of Missing Data. Northeast SAS Users Group. http://www.lexjansen.com/nesug/nesug03/st/st009.pdf Singer, E. (2006). Special Issue: Nonresponse Bias in Household Surveys. Public Opinion Quarterly, Vol 70, Issue 5. U.S. Office of Management and Budget (1990), "Survey Coverage", Statistical Policy Working Paper 17, Washington, D.C. U.S. Office of Management and Budget Publication (January 2006). “When Designing Surveys for Information Collections”. The Office of Management and Budget, 725 17th Street, NW. Washington, D.C. 20503 USA U.S. Office of Management and Budget Publication (September 2006). “Standards and Guidelines for Statistical Surveys”. The Office of Management and Budget, 725 17th Street, NW. Washington, D.C. 20503 USA U.S. Office of Management and Budget Publication (2008). VBA Compensation OMB - Part B Supporting statement for “Collections of Information Employing Statistical Methods”. Washington, D.C. Vogt, W. Paul, Vogt, Elaine R., Gardner, Dianne C., and Haeffele, Lynne M. (2014). Selecting the Right Analyses for Your Data - Quantitative, Qualitative, and Mixed Method. Guilford Press, New York, NY. Wallace, Leslie and Rust, Keith (1996). A Comparison of Raking and Poststratification Using 1994 NAEP Data. Leslie Wallace, West Inc., 584-589. 43 Appendix A Missing Data Patterns and Mechanisms An excellent discussion of missing data patterns, mechanisms, and research analysis methods is provided in Vogt, W. Paul, Vogt, Elaine R., Gardner, Dianne C., and Haeffele, Lynne M. (2014). An overview of the missing data types and issues is described below: Understanding the reasons why data is missing can be useful in analyzing the remaining data. If values are missing at random, the data sample may still be representative of the population; however, if the values are missing systematically, analysis may be harder. Missing completely at random. Values in a data set are missing completely at random (MCAR) if the events that lead to any particular data-item being missing are independent both of observable variables and of unobservable parameters of interest, and occur entirely at random. When data are MCAR, the analyses performed on the data are unbiased; however, data are rarely MCAR. Missing at random. Missing at random (MAR) is an alternative, and occurs when related to a particular variable, but is not related to the value of the variable that has missing data. An example of this is accidentally omitting an answer on a questionnaire. Missing not at random. Missing not at random (MNAR) is data that is missing for a specific reason (i.e. the value of the variable that is missing is related to the reason it is missing). An example of this is if a certain question on a questionnaire tends to be skipped deliberately by participants with certain characteristics. Graphical models can be used to describe the missing data mechanism in detail. While it is clear that MNAR can introduce statistical bias, there is no definitive test, see Vogt et al. (2014). It is also clear that MCAR is rarely evident in research data and most tests of it will fail. However, MAR is fully acceptable for valid statistical analyses (Vogt et. al, 2014). MAR is essentially “missing partially at random” whereby the intra-group missingness remains random despite some differences between group tendencies. Graphical data representations are the typical tool used in assessment as described above and in Pierchala, Carl E. (2001). See Section 3.2 Missing Data Patterns and Mechanisms for findings specific to Compensation’s data. 44 Appendix B Item Response Rates In accordance with OMB “Standards and Guidelines for Statistical Surveys,” Section 3.2, Guidelines 3.2.6-3.2.7, the item response rate was calculated as the ratio of the number of respondents for whom an in-scope response was obtained to the number of respondents who were asked to answer that item. The number asked to answer an item is the number of unitlevel respondents minus the number of respondents with a valid skip pattern. In addition to item response rate, total item response rate was calculated as the product of the overall unit response rate and the item response rate for each item. The purpose of these calculations is to assess the item non-response, which occurs when one or more survey items are left blank in an otherwise completed questionnaire. Tables B1.e and B1.s display the item and total item response rates for these surveys. The OMB “Standards and Guidelines for Statistical Surveys” (guideline 3.2.10) states an item non-response analysis should be conducted for items with an item response rate of less than 70%. Since none of the survey item response rates fall below 70%, for Access or Servicing, an item-level analysis of non-response bias was not necessary. The Access item response rates range from 84% to 100% with a 96% average while Servicing response rates range from 81% to 100%, with a 95% average. Table B1.e. Access Item and Total Item Response Rate11 Question Number 1 2 3 4 5a 5b 5c 5d 5e 5f 6 Item Response Rate 93% 99% 98% 99% 99% 99% 99% 98% 97% 98% 100% Unit Response Rate 20% 22% 22% 22% 22% 22% 22% 22% 21% 22% 22% 11 Open capture question for additional comments about your experience and e-mail opt in questions display “N/A” and were not included in item and total item response rate calculations 45 Table B1.e. Access Item and Total Item Response Rate (Continued) 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24a 24b 24c 24d 25 26 27 28a 28b 28c 29 30 31 32 33 34 35 97% 98% 95% 100% 94% 98% 98% 97% 99% 99% 100% 99% 97% 97% 98% 98% 97% 97% 97% 94% 98% 99% 99% 97% 96% 95% 97% 99% 96% 98% N/A 97% N/A N/A 21% 22% 21% 22% 21% 22% 22% 21% 22% 22% 22% 22% 21% 21% 22% 22% 21% 21% 21% 21% 21% 22% 22% 21% 21% 21% 21% 22% 21% 21% N/A 21% N/A N/A 46 Table B1.s Servicing Item and Total Item Response Rate12 Question Number 1 2 3 4 5a 5b 5c 5d 5e 5f 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24a 24b 24c 24d 25 26 27 Item Response Rate 92% 100% 98% 99% 99% 99% 99% 98% 98% 98% 88% 96% 99% 94% 93% 99% 98% 81% 96% 91% 98% N/A 95% 84% 88% 90% 97% 89% 93% 92% 94% 96% 99% 98% 98% Unit Response Rate 23% 25% 24% 25% 25% 25% 25% 25% 24% 24% 22% 24% 25% 24% 23% 25% 25% 20% 24% 23% 25% N/A 24% 21% 22% 22% 24% 22% 23% 23% 23% 24% 25% 24% 25% 12 Open capture questions for additional comments about your experience and items unclear in letter and e-mail opt in questions display “N/A” and were not included in item and total item response rate calculations 47 Table B1.e. Access Item and Total Item Response Rate (Continued) 28 29 30 31 N/A 98% N/A N/A N/A 25% N/A N/A In the item response rate calculations above, JDP considered blanks as non-response for mail returns and “don’t know” selections in addition to blanks as non-response for online returns. “Don’t know” selections are included as non-response for online returns since respondents are forced to select a response in order to continue the survey. Similarly, “N/A” responses were also included as non-response for rating questions in online returns. For respondents taking the survey online, the respondent must answer each question before proceeding to the next question in the survey, “Not Applicable” or “N/A” could either mean that the respondent was answering “N/A” to the question or did not wish to answer it. Therefore, this response option was included as non-response. 48 Appendix C Study Overview 1.1 Study Background The Voice of the Veteran Satisfaction Initiative tracks Veteran satisfaction with the benefits and services received from VBA. The Voice of the Veteran Line of Business Tracking Satisfaction Research Study is ongoing survey research tracking for Veteran satisfaction with VBA’s lines of business: Compensation, Pension, Education, Vocational Rehabilitation & Employment (VR&E), and Loan Guaranty (LGY). As part of Executive Order 13571 Streamlining Service Delivery and Improving Customer Service, agencies that provide significant services directly to the public are to identify and survey customers, establish service standards and track performance against those standards, and benchmark customer service against the best in business. This program enables VBA to understand what is important to Veterans relative to benefits received and services provided. This program provides timely and actionable Veteran feedback on how well VBA is providing services. Insights from this program identify opportunities for improvement and measure the impact of improvement initiatives, as well as continuously measure performance outcomes. Compensation’s survey instrument is measures Veterans’ satisfaction with access and receipt of benefits process. In FY15, fielding occurred continuously on a monthly basis for Access and annually for Servicing. Surveys remained open in field until the end of each quarter. If any surveys were received after a quarter closed field, then those returns were counted in the next quarter’s number of returns. Methodology Fielding Frequency Total Mailouts Per Year Target Number of Completes Access Mixed – Mail and Online Monthly 160,000 48,000 Servicing Mixed – Mail and Online Annually 60,000 18,000 Survey 1.2 Methodology The respondents had the option of completing a paper survey or an online survey. Respondents were first sent a postcard with a link to the eSurvey to complete the survey online. Each respondent was issued a unique sequence number which is entered online prior to beginning the eSurvey. Three weeks after deployment of the postcard, a survey packet containing a cover letter, survey instrument, and Business Reply Envelope (BRE) was sent to non-responders (to the postcard mailing). The sample for mailings of the survey packet was cleaned to exclude anyone who completed the survey at least one week prior to the cleaning. 49 1.2.1 Sample Criteria The targeted populations were identified by Compensation Service. For Compensation Access the target population is defined as Veterans and beneficiaries who began receiving compensation benefits six to eighteen months ago. For Compensation Servicing the target population is defined as Veterans and beneficiaries: (1) who received a decision for their application for compensation benefits with the past 30 days, (2) those who were found eligible on a new or subsequent claim and (3) those who have been denied and are not appealing the decision. VBA was responsible for providing sample to JDP that meets the following sampling criteria: Sample Population Inclusion Criteria Frequency of Data Request Access Survey For Access the target population includes Veterans and beneficiaries who began receiving compensation benefits six to eighteen months ago. Monthly Servicing Survey For Servicing the target population includes Veterans and beneficiaries: (1) who received a decision for their application for compensation benefits with the past 30 days, (2) those who were found eligible on a new or subsequent claim and (3) those who have been denied and are not appealing the decision. Annually 1.2.2 Sample File Generation Compensation generates the sample files based upon the sampling definitions and submits sample files directly to BAS. BAS receives the sample files and sends to VADIR for processing. VADIR processes sample files (to remove SSN and append demographics/EDIPI) and returns to BAS. BAS transfers sample files (via EDX platform) to JDP and notifies JDP via email that sample files are ready for deployment. JDP cleans the sample file and selects the sample. Sample is transferred to Government Printing Office (GPO) print vendor (via EDX platform) for printing and mailing of the postcards and survey packages. 50 VOV_LOB Sample is transferred in accordance with the following schedule: Tracking_Production Schedule_10.06.15.pdf 1.2.3 Data Transfer The sample was posted by BAS once a month within the sampling folder on the VOV EDX site. Sample should be provided in a file layout consistent with the file layout provided for the study as outlined below. Compensation File Layout ACC_Code AcillaryDecisionDate Address_1 Address_2 ADDRESS_LINE_ONE ADDRESS_LINE_THREE ADDRESS_LINE_TWO ADDRESS_LINE3 ADDRESS_LINE4 ADDRESS_LINE5 ADDRESS_LINE6 AGE AID_ATTENDANCE_HOUSEBOUND AMOUNT_AWARDED Award_End_Reason BENEFICIARY_TYPE BENEFIT_TYPE BRANCH_OF_SERVICE CHAR_SVC_CD CHARACTER_OF_DISCHARGE CITY_NAME Claim_DT CLAIM_LATEST_STATUS CLAIM_NUMBER CLOTHING_ALLOWANCE CURRENT_CLAIM_STATUS DATE_OF_APPLICATION DATE_OF_BIRTH Date_of_Birth DATE_OF_BIRTH2 51 Compensation File Layout (Continued.) DATE_OF_DEATH DAYS_OF_ACTIVE_SERVICE DeathIndicator DIAG_CODE DISABILITY_RATING DPBarcode DPV_Code Email_Address ENTITLEMENT_CODE ENTITLEMENT_DATE EOD EVALUATION GENDER GENDER_1 HOMELESS INDIVIDUAL_UNEMPLOYABILITY LAST NAME LATEST_END_PRODUCT NO_OF_APPEALS NO_OF_DEPENDENTS NUM_DISABILITIES_CLAIMED NUM_OF_DEPENDENTS PAYEE_CODE PERIOD_OF_SERVICE PHONE_NUMBER POA_CD RAD REASON_CODE REGIONAL_OFFICE_CODE REGIONAL_OFFICE_CODE_1 SERVICE_REPRESENTATIVE SSN STATE NAME ZIP_CODE Date of Award Method of Application Number of Application Prior Education Level 52 1.2.4 Data Cleaning JDP processed the sample according to the following cleaning rules: 1. De-duplicate records within each business line and across surveys based on the unique identifier (EDI_PI or VA_ID) for each record. Note: EDIPI is Electronic Data Interchange Personal Identifier. a) Exception: For Pension Access (v1) and Pension Servicing (v8), de-duplicate records based on EDI_PI and Claim Number. b) When each new sample file is received, JDP cleans it against all sample selected from every sample batch that has been delivered 12 months prior to ensure a respondent does not receive a VA line of business survey more than once in a 12 month-period. In the case of duplicates occurring within the same sample month, priority is assigned to business lines with the lowest number of sample records. 2. Clean out records present on the JDP Do Not Contact list and clean against the National Change of Address (NCOA) list. 3. Clean out any respondents who do not have any EDI_PI or VA_ID included in their sample record. a) Exception: For Pension Access (v1) and Pension Servicing (v8), clean out records with blank EDI_PI and Claim Number. 4. Clean out any respondents not specified as a dependent/spouse who have a date of death (DOD) in their sample record. 5. Clean out any respondents who do not have any address included in their sample record. 6. Assign and maintain unique sampling identifiers to each sample record in order to track history of sampling. Exclude records that have been sampled in the past 12 months to ensure no respondent is mailed surveys more than once in a 12-month timeframe. This rule may not apply to those who completed a survey. 1.2.5 Sample Cleaning Rules Glossary Duplicate records in sample file – the record is cleaned out if there is more than one record within the same sample file for the same respondent Duplicate record history – The record is cleaned out if the record has been selected within the past 12 months for any of VBA’s business line surveys (i.e. Compensation, Pension, Education, Home Loan Guaranty, and Vocational Rehabilitation) regardless of whether the respondent completed the survey Invalid address – The record is cleaned out if JDP’s address verification software indicates an invalid address code Invalid values – The record is cleaned out if the “VA_ID” field is blank Blanks – The record is cleaned out if the “Name” field corresponding to the record is blank Do not contact – The record is cleaned out if the individual is listed on JDP’s Do Not Contact List 53 1.2.6 Sample Selection JDP selected sample records following the completion of the sample cleaning process. The following guidelines are referenced when selecting sample: 1. Total Sampling Targets: The table below summarizes the total sampling target per an RO per a fielding period. The “Sampling Target per RO” column indicates the minimum number of sample records that should be selected per an RO for each survey. If this minimum target number cannot be reached for a particular RO, sample from a different RO will be selected to make up the difference. Frequency Total Sampling Target Sampling Target Per Time Period Sampling Target Per RO Number of ROs Access Survey Monthly 160,000 13,333 300 57 Servicing Survey Annually 60,000 60,000 1,053 57 2. The same record cannot be selected for multiple surveys during the same wave. Respondents who have completed a survey within the past 12 months cannot be selected. Survey priority is based on the number of records in each sample file. The survey with the smallest number of records is given first priority. 3. Following sample selection, the JDP project teams receives an automated report confirming the number of records selected for each survey version. The JDP project team verifies that the sample selection quantities reflect the sample targets and approves the sample file for fielding. 1.2.7 Fielding/Sampling Frequency Survey Instrument Access Survey Servicing Survey Methodology Total Survey Instruments Targeted Number of Completes Number of Postcards (eSurvey) Number of Mail Packages Fielding Frequency Mixed – Mail and Online 160,000 48,000 160,000 160,000 Monthly Mixed – Mail and Online 60,000 18,000 60,000 60,000 Annually 54 1.2.8 Order Generation and Fulfillment Process Federal Acquisition Regulations (FAR 8.8) mandate government agencies solicit all printing requirements through the Government Printing Office. GPO utilizes print vendors to fulfill orders. A Data Transfer Agreement (DTA) must be in place with print vendor and contractor before BAS can obligate funds or transfer sample files to the print vendor and contractor. Prior to mailing the postcards and mail surveys, print orders must be generated for each survey. The entire process may take up to 2-4 weeks from inception of the print order to the mailing of the survey package or postcard. Below are the steps involved in order generation and order fulfillment. Order generation After sample is received by JDP, the sample files are cleaned and selected. Then Letter Work Orders (LWOs) are created to provide the print vendor with the necessary information to match the sample files to the correct survey instrument. (1 day) JDP creates the print order and sends over to BAS Contractor Officer’s Representative (COR). (Same day as above step) The COR then reviews, authorizes, and submits the print order. (1 day) The BAS Publication Officer and/or COR submits the orders to the VA Publications Services Division (VAPSD). (Same day as above step) The order is issued a control number by a VBA Management Analyst, Publications. (Variable timing) Once the control number is assigned, the order goes to VA Publication Services Division liaison to forward to GPO Contracting Officer. (Variable timing) Note: the amount of time an order is with VAPSD varies greatly, it could be from 3 days up to 20 days. The GPO Contracting Officer sends the printing and mailing order to the print vendor. Order fulfillment Once the order is placed, the GPO print vendor is allotted 9 business days to fulfill the order (2 days to generate proofs, 2 days for proof review/correction, and 5 days to print and mail). Upon receipt of the proofs from print vendor, JDP reviews and approves; then BAS reviews and approves; then VAPSD reviews and approves. The GPO Print Vendor then conducts the printing of the instruments and prepares to mail. The print vendor uses envelopes that were subcontracted. The GPO Print Vendor mails the postcards and/or survey packages. After the orders have been mailed, the print vendor provides the mail receipts to contractor, BAS and VAPSD. Upon order completion, VAPSD provides actual costs to BAS. 55 1.2.9 Data Collection During the survey fielding period, both online survey returns and paper surveys are collected as they are received and posted on a secure EDX site. Responses from paper surveys are scanned through automated imaging software while verbatim responses are recorded by a live survey processor. Survey returns must have all pages intact in order to be processed and counted as a return. Surveys with missing pages are counted as unusable. Returns are also considered unusable, if there is an indication that the individual completing the survey is not the individual selected from the sample file (i.e. the respondent name and/or address on the survey is replaced with a different name and/or address). During each day of fielding, a subset of survey returns undergo quality assurance to validate the accuracy of responses captured. If duplicate surveys are returned (as identified by the unique sampling identifier assigned to each sample record), the original survey return is processed while the duplicate survey is removed. In the case of duplicate survey returns from mixed methodology surveys, the date the survey was received is used to identify the original return while the subsequent return is removed postfielding. 1.2.10 Reporting Reporting occurs four times yearly for the Access survey. On a quarterly basis, the following deliverables are provided: Scorecard Data Matrices Data is loaded to the VOV reporting site Open ended comments (verbatims) On a semiannual (twice yearly) basis, the following deliverable is provided: Data Analysis and Presentation Reporting occurs once annually for the Servicing survey. On an annual basis, the following deliverables are provided: Scorecard Data Matrices Data is loaded to the VOV reporting site Open ended comments (verbatims) Data Analysis and Presentation 56 APPENDIX D Approaches to Mitigating the Effect of NonResponse Bias and Strategies to Improve the Response Rate The following section outlines two approaches used in FY 2015 to mitigate the potential of nonresponse bias. As mentioned earlier in the report, J.D. Power affirms that while high response rates are always desirable in surveys, an 80% response rate is typically not achievable for a voluntary, customer-satisfaction survey instrument (Malhotra & Birks, 2007), particularly those that do not provide an incentive (not recommended for this program). To illustrate this point, the Dillman Method for survey fielding was discussed in Dillman, D. A. (2014) – a survey instrument was fielded to 600 students at the University of Washington. After 5 attempts to solicit a response in a closed university setting, as well as offering a monetary incentive to complete the study, they were only able to garner a 77% response rate. The first approach to minimize non-response occurs before and during data collection and involves introducing measures to maximize survey response rates. The second approach is to make statistical adjustments after the data is collected. 1.1 Approach 1: Strategies to Maximize Response Rates Prior to, and during, fielding the survey, JDP implemented the following measures to reduce the chances of non-response: Respondents were provided with the promise of confidentiality on the survey cover letter and postcard, and assured that their survey responses would not impact their current or future eligibility for benefits. Following the first mailing, non-respondents were sent an additional survey mailing. Respondents were provided with a toll-free telephone number and dedicated e-mail address to contact JDP about survey-related inquiries (e.g., how to interpret questions and response items, the purpose of the survey, how to get another copy of the survey if their copy has been lost/damaged, etc.). Telephone calls and e-mails are responded to within 24 hours and answered during regular business hours (8:00-5:00pm PT). JDP ensured the web-based surveys were accessible to people with disabilities by maintaining 508 compliant standards. These standards include: Keyboard navigation rather than mouse or other pointing devices Customization options for color, size, and style of text displayed Compatibility with screen-readers to translate items displayed on the survey in audible output and/or Braille displays Customer support and technical support through JDP Help Desk toll-free phone number and email address Exclusion of non-text elements, image maps, animation, flashing or blinking text. 57 The survey fielding period was extended to offer opportunities to respond for subgroups having a propensity to respond late (e.g., males, young, full-time employed). The survey was developed and reviewed in order to enhance respondent understanding of the survey materials and to improve the relevancy of the data collected: Prior to fielding the Benchmark study, a series of cognitive labs was conducted with test users to ensure the survey questions were easily understood and correctly interpreted. Revisions were made to the survey based on test user feedback. (As per OMB Guideline 1.4.1) After the Benchmark study and prior to fielding the first year of the Tracking study, Compensation Service and JDP conducted a review of the survey instruments and modified the surveys to improve the relevancy of data collected. (As per OMB Guideline 1.4.2) 1.2 Approach 2: Correcting Unit Non-response Bias with Sample Weighting and Survey Raking As stated above, the two approaches to tackling non-response bias include implementing measures to maximize response rates during the fielding period and making post hoc statistical adjustments to the survey results afterwards. The following section discusses the statistical adjustments approach, which include weighting the data or imputing scores to correct the amount of non-response bias. An example of this approach would be the survey raking procedure described earlier in this paper. See the associated references in the “Survey Raking Procedure for Sample Weightings” section for more information. The procedure known as “raking” adjusts a set of data so that its marginal totals match specified control totals on a specified set of variables. The term “raking” suggests an analogy with the process of smoothing the soil in a garden plot by alternately working it back and forth with a rake in two perpendicular directions, Izrael and Battaglia (2004). If non-response bias was identified in the survey data, the non-response bias could be corrected mathematically with a post-stratification survey weight. JDP would weigh the survey data based on certain demographics (such as age, gender, region, etc.) of the total sample so that the weighted survey data would conform more to the demographics of the total sample. The implicit assumption in this approach is the distributions of characteristics of the nonrespondents within an adjustment class (such as an age group) are the same, on average, as those of the respondents within the same adjustment class. See Appendix B for the item response rate for each question in the survey. If the item response rate was not lower than 70%, as per OMB standards, the imputation of data is not necessary. In the case that a particular item-level response was less than 70%, JDP would recommend conducting additional analysis to determine the potential for other factors (i.e. missing or skip patterns in the survey instrument) to be the cause of non-response. 58 Strategies to Improve Response Rate In addition to the strategies listed above, JDP recommends considering the following strategies to improve response rates going forward: Issue ongoing public communications (e.g. press releases, post information on the VA website) to spread awareness and confirm the legitimacy of the VA Study. Educate VA employees and VSOs about the survey to encourage participation. Provide a list of frequently asked questions and answers to VSOs and VA employees to equip them with answering Veterans’ questions regarding the survey. Send e-mail invitations to Veterans rather than mailing postcards to make it easier for Veterans to complete the survey online. Reduce the length of the survey to improve respondents’ willingness to respond Reduce overall number of questions and number of response options for each question. Increase the number of contacts to respondents with additional reminders about the survey to encourage participation Provide respondents with an additional paper survey questionnaire. Reduce the frequency of mailings to reduce the opportunities for delays and errors in the GPO Print process. Revise the cover letter and postcard to express the importance of participation in the survey. Provide sample from the 30 day period immediately prior to the mailing rather than sample from 90 days prior to improve the recency of their experience with the benefit (which improves both participation and recollection). Alter the responsibility of sample file generation from Compensation to PA&I. The PA&I data pull will increase consistency. Change location of sequence number to directly follow survey link on postcard and cover letter. Alter formatting on postcard and cover letter to include color print to make materials more readable to increase participation. 59 Appendix E Impact of FAR 8.8 Federal Acquisition Regulation (FAR) 8.8 requires that printing must be conducted through the Government Printing Office (GPO). The following section outlines limiting factors of the VOV Line of Business Tracking Satisfaction Research Study that occurred as a result of the FAR requirement. Through the utilization of the GPO Print Vendor, the following occurred in FY15: o Quality issues included: Survey instruments were printed and mailed: Utilizing the sample population from one survey, but receiving a different survey (e.g., potential respondents from the pool of one business line received the survey for a different business line) Using a version of the instrument that was outdated; this version did not contain the current questions or responses that were being fielded Sent the wrong surveys to the wrong respondents o Ongoing timeliness delays occurred with each set of orders placed, as the order fulfillment process took a minimum of 2-4 weeks 1.1 Impact The project experienced ongoing delays in the printing and mailing of its postcards and survey packets for VBA’s lines of business. The delays affected the critical processes required to execute the VOV Program to its fullest potential. A multitude of quality issues were experienced throughout FY15 that negatively impacted the VOV Program response rates. The issues that occurred impacted: access to the online survey; readability of mail materials; level of effort required by respondents to take the survey; relevancy of survey; and the diminishment of brands (VA/JDP) associated with poor quality materials. 60 Appendix F NOTE: Questionnaire is not shown in the formatted version that respondents used to fill out survey. Survey Questionnaires [DO NOT DISPLAY/IDENTIFY SECTION HEADERS. DISPLAY SINGLE QUESTION PER PAGE.] [RESPONSE CODES APPEAR IN BRACKETS AT THE END OF EACH RESPONSE FOR SINGLE RESPONSES AND IN THE PROGRAMMING INSTRUCTIONS FOR MULTIPLE RESPONSES.] Servicing Questionnaire Benefit Information 1. How did you FIRST learn about VA benefit programs? (Mark only one) If you are unsure, please indicate the first way you remember learning about VA benefit programs. [RADIO BUTTONS. SINGLE RESPONSE.] a. VA website [1] b. c. d. e. f. g. h. eBenefits.va.gov [3] Social media websites (e.g., Facebook, Twitter, etc.) [11] Internet (excluding VA and social media sites) [14] Mail (from VA) [4] VA phone number (800-827-1000) [5] In person at a Regional Office/Visit from a VA employee [10] VA medical center/VA Vet Center [8] i. Transition Assistance Program/Disabled Transition Assistance Program briefings [6] j. Veterans Service Organizations (e.g., Amer. Legion, DAV, VFW, PVA, MOPH, etc.) k. Other Veterans [13] l. Friends or family [15] m. Other publications (e.g., Army Times, local newspaper, etc.) [16] n. Vocational Rehabilitation and Employment Service o. Other (Specify) ___________________[TEXT BOX, FORCE TEXT IF RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97] p. Don’t know or not sure [99] 2. What method(s) do you MOST FREQUENTLY use to obtain general information about VA benefits or services? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE.CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED] 61 a. b. c. d. e. f. g. h. i. j. k. l. m. n. o. p. q. VA website eBenefits.va.gov Social media websites (e.g., Facebook, Twitter, etc.) Other websites (excluding VA or social media sites) Phone Mail E-mail In person at a Regional Office VA medical center/VA Vet Center Veterans Service Organizations (e.g., Amer. Legion, DAV, VFW, PVA, MOPH, etc.) Disabled Veterans’ Outreach Program Friends or family Vocational Rehabilitation and Employment Service Other publications (e.g., Army Times, local newspaper, etc.) Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF RESPONSE IS SELECTED, 50 CHARACTER MAX.] Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE] None of the above [MUTUALLY EXCLUSIVE RESPONSE] 3. How frequently would you like to receive communications (e.g., e-mails, letters, newsletters, etc.) about VA benefits or services? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.] a. Weekly [1] b. Monthly [2] c. Quarterly (every 3 months) [3] d. Semi-annually (twice per year) [4] e. Annually (once per year) [5] f. Never [6] g. Don’t know or not sure [99] 4. How would you like to receive information from VA about benefits or services? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED] a. Phone b. Mail c. E-mail d. VA website e. Social media websites (e.g., Facebook, Twitter, etc.) f. In person at a Regional Office g. Veterans Service Organizations (e.g., Amer. Legion, DAV, VFW, PVA, MOPH, etc.) h. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF RESPONSE IS SELECTED, 50 CHARACTER MAX.] i. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE] 62 The following question asks you to rate various aspects of your experience with Compensation using a scale of 1 to 10 where 1 is Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON SAME PAGE AS THE QUESTION THAT FOLLOWS] 5. When thinking about your most frequently used methods of communication, please rate your experience in obtaining information about your benefit on the following items: (Mark only one per row) [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND ATTRIBUTES/RESPONSES IN ROWS (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, ALTERNATE SHADES IN ROWS. SINGLE RESPONSE PER ROW. RANDOMIZE ALL ATTRIBUTES EXCEPT THE LAST ONE.] a. Ease of accessing information [ALLOW N/A RESPONSE] [1-10, N/A=99] b. Availability of information [ALLOW N/A RESPONSE] [1-10, N/A=99] c. Clarity of information [ALLOW N/A RESPONSE] [1-10, N/A=99] d. Usefulness of information [ALLOW N/A RESPONSE] [1-10, N/A=99] e. Frequency of information provided by VA [ALLOW N/A RESPONSE] [110, N/A=99] f. Overall rating of information [1-10] Contact with VA 6. During the past 6 months, did you contact anyone from VA about your benefit? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.] a. Yes [1] b. No [0] (Ask Q7-Q12 if Q6 is yes, otherwise go to Q13) 7. Which of the following best describes the reason for your most recent contact? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.] a. Resolve a problem [1] b. Ask a question [2] c. Request a change to your records/provide information [3] 8. Can you briefly describe the nature of your most recent contact? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED] a. Update your dependency status b. Change your address or direct deposit information c. Report the death of an individual who received VA benefits d. Report that you did not receive your VA check or direct deposit e. Resolve a problem with your benefits f. Find out about a late benefit payment g. Report a problem with a VA customer service representative 63 h. i. j. k. Ask a general question Obtain information about submitting/re-opening a claim Check on the status of a claim Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF RESPONSE IS SELECTED, 50 CHARACTER MAX.] 9. Thinking about your most recent contact, how did you contact VA? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE] a. Phone [1] b. Online Chat c. Website [6] d. E-mail [7] e. Mail [9] f. In person [3] g. eBenefits.va.gov [10] 10. Was your most recent issue resolved? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE] a. Yes [1] b. No [0] (Ask Q11 if Q10 is No, otherwise go to Q12) 11. Why wasn’t your most recent issue resolved? [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED] a. Did not receive all of the information required b. Received incorrect information c. Was referred to the incorrect office/person d. Waiting for follow-up from VA e. Other (Specify) ____________________ [TEXT BOX, FORCE TEXT IF RESPONSE IS SELECTED, 50 CHARACTER MAX.] f. Don't know or not sure [MUTUALLY EXCLUSIVE RESPONSE] 12. Thinking of your most recent contact with the VA, how would you rate your overall customer service experience with the VA or VA representatives using a scale of 1 to 10 where 1 is Unacceptable, 10 is Outstanding, and 5 is Average? [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND SINGLE ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, SINGLE RESPONSE PER ROW.][1-10] Benefit Entitlement 64 13. Have you submitted a claim for an increase in your benefit in the past 6 months? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE] a. Yes [1] b. No [0] c. Don’t know or not sure [99] (Ask Q14 if Q13 is yes, otherwise go to Q22) 14. How did you submit your claim? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE] a. eBenefits.va.gov b. Mail [1] c. In person at a Regional Office [2] d. In person at a Veterans Service Organization (e.g., Amer. Legion, DAV, VFW, PVA, MOPH, etc.) [3] e. f. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF RESPONSE IS SELECTED, 50 CHARACTER MAX.] [97] g. Don’t know or not sure [99] (Ask Q15 if Q13 is yes, otherwise go to Q22) 15. After you submitted your claim, did you receive a notification/confirmation from VA that your claim was received? [RADIO BUTTONS. SINGLE RESPONSE] a. Yes [1] b. No [0] c. Don’t know or not sure [99] (Ask Q16-Q18 if Q15 is Yes, otherwise go to Q19) 16. Thinking about the notification/confirmation from VA, was it clear and easy to understand? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE] a. Not at all clear [1] b. Somewhat clear [2] c. Completely clear [3] d. Don’t know or not sure [99] e. I did not read the letter [96] (Ask Q17 if Q16 is “Not at all clear” or “Somewhat clear”, otherwise go to Q18) 17. What did you find unclear/didn’t understand in the notification/confirmation? (Open Capture) [OPEN-END. TEXT BOX. 1000 CHARACTERS MAX. ALLOW NO COMMENT, MUTUALLY EXCLUSIVE CHECK BOX. CODE NO COMMENT AS 0 IF UNCHECKED AND 1 IF CHECKED.] 18. Did you contact VA to obtain clarification about the notification(s)/confirmation(s)? [RADIO BUTTONS. SINGLE RESPONSE] a. Yes [1] b. No [0] 65 c. Don’t know or not sure [99] 19. Did VA require you to provide additional medical evidence beyond the information you provided with your original claim? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE] a. Yes [1] b. No [0] c. Don’t know or not sure [99] (Ask Q20 if Q19 is yes, otherwise go to Q22) 20. After you submitted your claim, did VA schedule a medical examination for you to be re-evaluated? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE] a. Yes [1] b. No [0] c. Don’t know or not sure [99] d. Not applicable [96] (Ask Q21 if Q20 is Yes, otherwise go to Q22) 21. Did the exam address your claimed condition(s)? [RADIO BUTTONS. SINGLE RESPONSE] a. Yes [1] b. No [0] c. Don’t know or not sure [99] 22. Have there been any interruptions to your benefit payments in the past 6 months? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE] a. Yes [1] b. No [0] c. Don’t know or not sure [99] (Ask Q23 if ‘Yes’ to Q22, otherwise go to Q24) 23. Did you receive a letter notifying you as to the reason why your benefit payment was interrupted and/or terminated? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE] a. Yes [1] b. No [0] c. Don’t know or not sure [99] The following question asks you to rate various aspects of your VA experience, using a scale of 1 to 10 where 1 is Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON SAME PAGE AS THE QUESTION THAT FOLLOWS] 24. Please rate your compensation benefit on the following items: (Mark only one per row) [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND ATTRIBUTES/RESPONSES IN ROWS (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, ALTERNATE SHADES IN ROWS. SINGLE RESPONSE PER ROW. RANDOMIZE ALL ATTRIBUTES EXCEPT THE LAST ONE.] 66 a. Combined disability evaluation rating percentage (e.g. 10% disabled) [ALLOW N/A RESPONSE] [1-10, N/A=99] b. Timeliness of receiving benefit [ALLOW N/A RESPONSE] [1-10, N/A=99] c. Clarity of your disability rating [ALLOW N/A RESPONSE] [1-10, N/A=99] d. Overall rating of your benefit payment[1-10] Overall Experience with Benefit Program 25. Thinking about ALL aspects of your experience with your compensation benefits, please rate VA overall, using a scale of 1 to 10 where 1 is Unacceptable, 10 is Outstanding, and 5 is Average. (Mark only one) [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND SINGLE ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, SINGLE RESPONSE PER ROW.] [1-10] Overall Experience with VA 26. Taking into consideration all of the non-medical benefits (e.g., education, compensation, pension, home loan guaranty, vocational rehabilitation and employment, insurance, etc.) you have applied for or currently receive, please rate your experience with VA overall, using a scale of 1 to 10 where 1 is Unacceptable, 10 is Outstanding, and 5 is Average. (Mark only one) [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND SINGLE ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, SINGLE RESPONSE PER ROW.] [1-10] 27. How likely are you to inform other Veterans or beneficiaries about your experience with VA benefits or services? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.] a. Definitely will not [1] b. Probably will not [2] c. Probably will [3] d. Definitely will [4] 28. Do you have any other comments or concerns about your experience? (Open Capture) [OPEN-END. TEXT BOX. 1000 CHARACTERS MAX. ALLOW NO COMMENT, MUTUALLY EXCLUSIVE CHECK BOX. CODE NO COMMENT AS 0 IF UNCHECKED AND 1 IF CHECKED] 67 Additional Questions 29. How are you currently using your benefit payment? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED] a. Rent/mortgage payment b. Paying bills c. Paying down debt d. Medical expenses e. Education expenses f. Establishing savings g. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF RESPONSE IS SELECTED, 50 CHARACTER MAX.] h. Prefer not to answer [MUTUALLY EXCLUSIVE RESPONSE] i. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE] As a reminder, your responses will be kept completely confidential and your e-mail address will not be sent to VA with any responses on this survey. [SHOW ON THE SAME PAGE AS THE QUESTION THAT FOLLOWS.] 30. Would you like to provide an e-mail address so VA can contact you with general information about VA benefits and services? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.] a. Yes [1] b. No [0] c. I do not have an e-mail address [96] d. Prefer not to answer [98] (Ask Q31 if Yes in Q30) 31. Please enter your preferred e-mail address where you would like to be contacted: (Open Capture) a. E-mail: [TEXT BOX. 100 CHARACTER MAX.] 68 Access Questionnaire Benefit Information 1. How did you FIRST learn about VA benefit programs? (Mark only one) If you are unsure, please indicate the first way you remember learning about VA benefit programs. [RADIO BUTTONS. SINGLE RESPONSE.] a. VA website [1] b. c. d. e. f. g. h. eBenefits.va.gov [3] Social media websites (e.g., Facebook, Twitter, etc.) [11] Internet (excluding VA and social media sites) [14] Mail (from VA) [4] VA phone number (800-827-1000) [5] In person at a Regional Office/Visit from a VA employee [10] VA medical center/VA Vet Center [8] i. Transition Assistance Program/Disabled Transition Assistance Program briefings [6] j. Veterans Service Organizations (e.g., Amer. Legion, DAV, VFW, PVA, MOPH, etc.) k. Other Veterans [13] l. Friends or family [15] m. Other publications (e.g., Army Times, local newspaper, etc.) [16] n. Vocational Rehabilitation and Employment Service o. Other (Specify) ___________________[TEXT BOX. FORCE TEXT IF RESPONSE IS SELECTED. 50 CHARACTER MAX.] [97] p. Don’t know or not sure [99] 2. What method(s) do you MOST FREQUENTLY use to obtain general information about VA’s benefits or services? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED] a. VA website b. c. d. e. f. g. h. i. eBenefits.va.gov Social media websites (e.g., Facebook, Twitter, etc.) Other websites (excluding VA or social media sites) Phone Mail E-mail In person at a Regional Office VA medical center/VA Vet Center j. Veterans Service Organizations (e.g., Amer. Legion, DAV, VFW, PVA, MOPH, etc.) k. Disabled Veterans’ Outreach Program 69 Friends or family Vocational Rehabilitation and Employment Service Other publications (e.g., Army Times, local newspaper, etc.) Other (Specify) ___________________[TEXT BOX. FORCE TEXT IF RESPONSE IS SELECTED. 50 CHARACTER MAX.] p. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE.] q. None of the above [MUTUALLY EXCLUSIVE RESPONSE.] l. m. n. o. 3. How frequently would you like to receive communications (e.g., e-mails, letters, newsletters, etc.) about VA benefits or services? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.] a. Weekly [1] b. Monthly [2] c. Quarterly (every 3 months) [3] d. Semi-annually (twice per year) [4] e. Annually (once per year) [5] f. Never [6] g. Don’t know or not sure [99] 4. How would you like to receive information from VA about applying for VA benefits or services? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED] a. Phone b. Mail c. E-mail d. VA website e. Social media websites (e.g., Facebook, Twitter, etc.) f. In person at a Regional Office g. Veterans Service Organizations (e.g., Amer. Legion, DAV, VFW, PVA, MOPH, etc.) h. Other (Specify) ___________________[TEXT BOX. FORCE TEXT IF RESPONSE IS SELECTED. 50 CHARACTER MAX.] i. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE.] The following question asks you to rate various aspects of your experience with Compensation using a scale of 1 to 10, where 1 is Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON SAME PAGE AS THE QUESTION THAT FOLLOWS] 5. When thinking about your most frequently used methods of communication please rate your experience in obtaining information about your benefit application on the following items: (Mark only one per row) [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND ATTRIBUTES/RESPONSES IN ROWS (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, ALTERNATE SHADES IN ROWS. SINGLE RESPONSE PER ROW. RANDOMIZE ALL ATTRIBUTES EXCEPT THE LAST ONE.] 70 Ease of accessing information [ALLOW N/A RESPONSE][1-10, N/A=99] Availability of information [ALLOW N/A RESPONSE] [1-10, N/A=99] Clarity of information [ALLOW N/A RESPONSE] [1-10, N/A=99] Usefulness of information [ALLOW N/A RESPONSE] [1-10, N/A=99] Frequency of information provided by VA [ALLOW N/A RESPONSE] [1-10, N/A=99] f. Overall rating of information [1-10] a. b. c. d. e. Contact with VA 6. During the past 6 months, did you contact anyone from VA about the benefit application process? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.] a. Yes [1] b. No [0] (Ask Q7-Q12 if Q6 is yes, otherwise go to Q13) 7. Which of the following best describes the reason for your most recent contact? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.] a. Resolve a problem [1] b. Ask a question [2] c. Request a change to your records/provide information [3] 8. Can you briefly describe the nature of your most recent contact? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED] a. Change your address or direct deposit information b. Report the death of an individual who received VA benefits c. Report that you did not receive your VA check or direct deposit d. Report a problem with a VA customer service representative e. Ask a general question f. Obtain information about submitting/re-opening a claim g. Check on the status of a claim h. Other (Specify) ___________________[TEXT BOX. FORCE TEXT IF RESPONSE IS SELECTED. 50 CHARACTER MAX.] 9. Thinking about your most recent contact, how did you contact VA? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.] a. Phone [1] b. Online Chat c. eBenefits.va.gov [10] d. Website [6] e. E-mail [7] f. Mail [9] g. In person [3] 71 10. Was your most recent issue resolved? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.] a. Yes [1] b. No [0] (Ask Q11 if Q10 is No, otherwise go to Q12 11. Why wasn’t your most recent issue resolved? [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED] a. Did not receive all of the information required b. Received incorrect information c. Was referred to the incorrect office/person d. Waiting for follow-up from VA e. Other (Specify) ____________________ [TEXT BOX. FORCE TEXT IF RESPONSE IS SELECTED. 50 CHARACTER MAX.] f. Don't know or not sure 12. Thinking of your most recent contact with the VA, how would you rate your overall customer service experience with the VA or VA representatives using a scale of 1 to 10 where 1 is Unacceptable, 10 is Outstanding, and 5 is Average? [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND SINGLE ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, SINGLE RESPONSE PER ROW.][1-10] Benefit Eligibility and Application Process 13. Thinking about your most recent application, did someone from VA (e.g., call center representative, regional office representative, etc.) provide you with information about the benefit application process? [RADIO BUTTONS. SINGLE RESPONSE] a. Yes [1] b. No [0] c. Don’t know or not sure [99] 14. Thinking about your most recent benefit application, what method did you use to apply for your benefit? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE] a. eBenefits.va.gov b. In person at a Regional Office [3] c. Mail [2] d. In person at a Veterans Service Organization (e.g., Amer. Legion, DAV, VFW, PVA, MOPH, etc.) [4] 72 e. Other (Specify) ___________________ [TEXT BOX. FORCE TEXT IF RESPONSE IS SELECTED. 50 CHARACTER MAX.] [97] f. Don’t know or not sure [99] 15. After you submitted your application, did you receive a notification/confirmation from VA that your claim was received? [RADIO BUTTONS. SINGLE REPONSE.] a. Yes [1] b. No [0] c. Don’t know or not sure [99] (Ask Q16-21 if Q15 is Yes, otherwise go to Q22) 16. Thinking about the notification/confirmation from VA, was it clear and easy to understand? (Mark only one) [RADIO BUTTONS. SINGLE REPONSE.] a. Not at all clear [1] b. Somewhat clear [2] c. Completely clear [3] d. Don’t know or not sure [99] e. I did not read the letter [96] 17. Did you contact VA to obtain clarification about any of the notifications/confirmations you received? [RADIO BUTTONS. SINGLE REPONSE.] a. Yes [1] b. No [0] c. Don’t know or not sure [99] 18. Did you provide VA with the documentation that was requested in the notification(s)/confirmation(s)? (Mark only one) [RADIO BUTTONS. SINGLE REPONSE.] a. Yes [1] b. No [0] c. Nothing was requested [96] d. Don’t know or not sure [99] (Ask Q19-Q20 if Q18 is yes, otherwise go to Q21) 19. How did you submit the documentation to VA that was requested in the notification/confirmation? (Mark only one) [RADIO BUTTONS. SINGLE REPONSE.] a. eBenefits.va.gov b. In person at a Regional Office [2] c. Mail d. Through a Veterans Service Organization(e.g., Amer. Legion, DAV, VFW, PVA, MOPH, etc.)[3] e. Other (Specify) ___________________[TEXT BOX. FORCE TEXT IF RESPONSE IS SELECTED. 50 CHARACTER MAX.] [97] f. Don’t know or not sure [99] 73 20. What is your preferred method to submit the documentation to VA that was requested in the notification/confirmation? (Mark only one) [RADIO BUTTONS. SINGLE REPONSE.] a. eBenefits.va.gov b. In person at a Regional Office [2] c. Mail d. Through a Veterans Service Organization (e.g., Amer. Legion, DAV, VFW, PVA, MOPH, etc.) [4] e. Other (Specify) ___________________[TEXT BOX. FORCE TEXT IF RESPONSE IS SELECTED. 50 CHARACTER MAX.] [97] f. Don’t know or not sure [99] 21. Did you receive a subsequent notification requesting information in support of your claim from VA? (Mark only one) [RADIO BUTTONS. SINGLE REPONSE.] a. Yes [1] b. No [0] c. Don’t know or not sure [99] 22. During the application process, did you have to provide the same information more than once? (Mark only one) [RADIO BUTTONS. SINGLE REPONSE.] a. Yes [1] b. No [0] c. Don’t know or not sure [99] (Ask Q23 if Q22 is Yes, otherwise go to Q24) 23. What information did you have to provide more than once? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED] a. Discharge papers (DD214) b. Service treatment records c. Private medical records d. Other (Specify) ___________________[TEXT BOX. FORCE TEXT IF RESPONSE IS SELECTED. 50 CHARACTER MAX.] e. Don’t know or not sure The following question asks you to rate various aspects of your experience with your benefit application using a scale of 1 to 10, where 1 is Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON SAME PAGE AS THE QUESTION THAT FOLLOWS] 24. Please rate your experience with the benefit application process on the following items: (Mark only one per row) [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND ATTRIBUTES/RESPONSES IN ROWS (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, ALTERNATE SHADES IN 74 ROWS. SINGLE RESPONSE PER ROW. RANDOMIZE ALL ATTRIBUTES EXCEPT THE LAST ONE.] a. Ease of completing the application [ALLOW N/A RESPONSE][1-10, N/A=99] b. Timeliness of eligibility/entitlement notification [ALLOW N/A RESPONSE] [110, N/A=99] c. Flexibility of application methods [ALLOW N/A RESPONSE] [1-10, N/A=99] d. Overall rating of application process [1-10] (Paper Only Instruction: Ask Q25-Q27 if previously found ineligible for VA benefit payments, otherwise go to Q28) 25. If you were previously found ineligible for VA benefit payments, did you understand why you were found ineligible? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE] a. Yes [1] b. No [0] c. Don’t know or not sure [99] d. Not applicable, never been found ineligible (Online Only Response) [96] (Online Instruction: Ask Q26-Q27 if Q25 is yes, otherwise go to Q28) 26. Were you provided information about how to appeal your decision? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE] a. Yes [1] b. No [0] c. Don’t know or not sure [99] 27. Using a scale of 1 to 10, where 1 is Unacceptable, 10 is Outstanding, and 5 is Average, please rate the clarity of the information you were provided about appealing your decision. [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND SINGLE ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, SINGLE RESPONSE PER ROW.][1-10] Benefit Entitlement The following question asks you to rate various aspects of your experience with your benefit payment using a scale of 1 to 10, where 1 is Unacceptable, 10 is Outstanding, and 5 is Average. [SHOW ON SAME PAGE AS THE QUESTION THAT FOLLOWS] 28. Please rate your benefit payment on the following items: (Mark only one per row) [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND ATTRIBUTES/RESPONSES IN ROWS (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, ALTERNATE SHADES IN ROWS. SINGLE RESPONSE PER ROW. RANDOMIZE ALL ATTRIBUTES EXCEPT THE LAST ONE.] 75 a. Amount of benefit payment [ALLOW N/A RESPONSE][1-10, N/A=99] b. Timeliness of receiving initial benefit payment [ALLOW N/A RESPONSE] [110, N/A=99] c. Overall rating of your benefit payment [1-10] Overall Application Experience 29. Thinking about ALL aspects of your experience applying for your compensation benefit, please rate VA overall, using a scale of 1 to 10 where 1 is Unacceptable, 10 is Outstanding, and 5 is Average. (Mark only one) [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND SINGLE ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, SINGLE RESPONSE PER ROW.] [1-10] Overall Experience with VA 30. Taking into consideration all of the non-medical benefits (e.g., education, compensation, pension, home loan guaranty, vocational rehabilitation and employment, insurance, etc.) you have applied for or currently receive, please rate your experience with VA overall, using a scale of 1 to 10 where 1 is Unacceptable, 10 is Outstanding, and 5 is Average. (Mark only one) [SHOW RESPONSES IN GRID WITH 10-POINT SCALE IN COLUMNS AND SINGLE ROW (SEE JDPA CONVENTIONS DOCUMENT PG. 1 FOR SPECIFIC DETAILS OF LAYOUT). EVENLY SPACED RADIO BUTTONS/COLUMNS, SINGLE RESPONSE PER ROW.] [1-10] 31. How likely are you to inform other Veterans or beneficiaries about your experience with VA benefits or services? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.] a. Definitely will not [1] b. Probably will not [2] c. Probably will [3] d. Definitely will [4] 32. Do you have any other comments or concerns about your experience? (Open Capture) [OPEN-END. TEXT BOX. 1000 CHARACTERS MAX. ALLOW NO COMMENT, MUTUALLY EXCLUSIVE CHECK BOX. CODE NO COMMENT AS 0 IF UNCHECKED AND 1 IF CHECKED] ____________________________________________________ Additional Questions 76 As a reminder, your responses will be kept completely confidential and will not affect any current or future benefits you may receive. [SHOW ON SAME PAGE AS THE QUESTION THAT FOLLOWS.] 33. How are you currently using or intending to use your benefit payment? (Mark all that apply) [CHECK BOXES. MULTIPLE RESPONSE. CODE EACH RESPONSE AS 0 IF UNCHECKED OR 1 IF CHECKED] a. Rent/mortgage payment b. Paying bills c. Paying down debt d. Education expenses e. Establishing savings f. Other (Specify) ___________________ [TEXT BOX, FORCE TEXT IF RESPONSE IS SELECTED, 50 CHARACTER MAX.] g. Prefer not to state [MUTUALLY EXCLUSIVE RESPONSE] h. Don’t know or not sure [MUTUALLY EXCLUSIVE RESPONSE] As a reminder, your responses will be kept completely confidential and your e-mail address will not be sent to VA with any responses on this survey. [SHOW ON THE SAME PAGE AS THE QUESTION THAT FOLLOWS.] 34. Would you like to provide an e-mail address so VA can contact you with general information about VA benefits and services? (Mark only one) [RADIO BUTTONS. SINGLE RESPONSE.] a. Yes [1] b. No [0] c. I do not have an e-mail address [96] d. Prefer not to answer [98] (Ask Q35 if Yes in Q34) 35. Please enter your preferred e-mail address where you would like to be contacted: (Open Capture) a. E-mail: [TEXT BOX. 100 CHARACTER MAX.] 77 Appendix G List of Acronyms AAPOR ANOVA BAS BPA BRE CAPS COR DTA EDIPI EDX FAR FY GPO ICR JDP LGY LWO MAR MCAR MCMC MNAR NPC OIF OEF OMB OSAT PA&I RO SSN US USA VA VADIR VAPSD VBA VOV VR&E VSO American Association for Public Opinion Research Analysis of Variance Benefits Assistance Service Blanket Purchase Agreement Business Reply Envelope Centralized Account Processing System Contracting Officer’s Representative Data Transfer Agreement Electronic Data Interchange Personal Identifier Enterprise Data Exchange Federal Acquisition Regulations Fiscal Year Government Printing Office Information Collection Request J.D. Power Loan Guaranty Service Letter Work Order Missing At Random Missing Completely At Random Markov chain Monte Carlo algorithm Missing Not At Random NPC, Inc. Integrated Print and Digital Solutions Operation Iraqi Freedom Operation Enduring Freedom Office of Management and Budget Overall Satisfaction Index Office of Performance Analysis & Integrity Regional Office Social Security Number United States United States of America Department of Veterans Affairs VA DoD Identity Repository VA Publications Services Division Veterans Benefits Administration Voice of the Veteran Vocational Rehabilitation and Employment Service Veterans Service Organizations 78
| File Type | application/pdf |
| File Title | Training Catalog, Department of Veterans Affairs |
| Subject | Training Catalog |
| Author | Department of Veterans Affairs, Office of Human Resources and Ad |
| File Modified | 2016-12-19 |
| File Created | 2016-12-19 |