A Final Look: California's Previously Uninsured after the ACA's Third Open Enrollment Period

Survey Methodology

This is the fourth in a series of surveys by the Kaiser Family Foundation (KFF) tracking the views and experiences of a group of Californians who were uninsured in the summer of 2013, prior to implementation of the ACA’s insurance market reforms and coverage expansions through Covered California and Medi-Cal. The first survey (Wave 1) was conducted from July 11-August 29, 2013, with a randomly selected group of individuals who were uninsured at the time of the interview, and was paid for entirely by KFF. The second survey (Wave 2) was conducted from April 1-June 15, 2014. The third survey (Wave 3) was conducted from February 18-May 13, 2015, and the current survey (wave 4) was conducted from February 10-June 20, 2016,   with the same longitudinal panel of respondents, whether they obtained coverage or remained uninsured. All four surveys were designed and analyzed by researchers at KFF. Social Science Research Solutions collaborated with KFF researchers on sample design, weighting, and supervised fieldwork. Fieldwork costs associated with Waves 2, 3, and 4 of the survey were paid for by The California Endowment.

The Wave 1 survey was conducted among a representative random sample of 2,001 adults ages 19-64 living in California who reported having been without health insurance for at least two months at the time of the interview¹ (NOTE: persons without a telephone could not be included in the random selection process). Computer-assisted telephone interviews conducted by landline (990) and cell phone (1,011, including 660 who had no landline telephone) were carried out in English and Spanish by SSRS. To increase efficiency in reaching this low-incidence, hard-to-reach group, both the landline and cell phone sampling frames oversampled areas with a lower-income population (since being uninsured is negatively correlated with income). The landline sample frame also oversampled households whose phone numbers were matched with directory listings indicating the presence of at least one person age 19-64 and a household income of less than $25,000. Additionally, 230 interviews (130 landline, 100 cell phone) were conducted with respondents who previously completed recent national SSRS omnibus surveys of the general public and indicated they were ages 19-64 and uninsured. These previous surveys were conducted with nationally representative, random-digit-dial landline and cell phone samples. Waves 2, 3, and 4 also consisted of computer-assisted telephone interviews conducted in English and Spanish by landline and cell phone, including those who had no landline phone.

In order to re-connect with respondents who may be more willing to complete the survey online, an abbreviated web version was introduced for Waves 2, 3, and 4 after attempts had been made to reach respondents by phone. The online questionnaire was offered in English and Spanish and was limited to key questions about insurance status, type of coverage, and reasons for obtaining coverage or remaining uninsured. A total of 51 respondents in Wave 2, 87 respondents in Wave 3, and 113 in Wave 4 completed the online version of the survey.

Screening for Waves 2, 3, and 4 involved verifying that the respondent had participated in Wave 1. Multiple attempts were made to reach every respondent from Wave 1 and encourage participation in later waves. Efforts included multiple dialing at various times of day and throughout the week, mailings and emails to those who provided such contact information, repeated dialing of non-working numbers, and attempts to find alternative phone numbers for non-working numbers.

A multi-stage weighting design was applied to ensure accurate representation of California’s nonelderly adult uninsured population prior to the ACA’s coverage expansions. The weighting process for Waves 2, 3, and 4 involved corrections for sample design, as well as sample weighting to match the weighted Wave 2 sample, weighted Wave 3 sample, and weighted Wave 4 sample to Wave 1 responses along demographic characteristics. As it did for Wave 1, the base weight for Waves 2, 3, and 4 accounted for the oversamples used in the sample design, as well as the likelihood of non-response for the sample from earlier omnibus surveys, number of eligible household members for the landline sample, and a correction to account for the fact that respondents with both a landline and cell phone have a higher probability of selection. Demographic weighting parameters for Waves 2, 3, and 4 were based on Wave 1 weighted demographics, which were adjusted for age, education, race/ethnicity, nativity (for Hispanics only), Hispanics by gender, presence of own child in household, marital status, California region, poverty level, and phone usage. For more information on weighting and data sources, see the Wave 1 methodology. All differences referred to in the report are statistically significant. Statistical tests of significance account for the effect of weighting, and, for trend analysis, testing takes into account the survey’s panel design.

A unique consideration for panel surveys such as the Kaiser Family Foundation California Longitudinal Panel Survey, is whether those who participate in subsequent waves are different in terms of their attitudes or demographics than those who refuse to participate again or were unable to be re-contacted. Of the total 2,001 respondents who completed Wave 1, 1,219 participated in Wave 2, 1,105 completed Wave 3, and 1001 completed Wave 4. These completion rates are within an expected range given that the uninsured are already an often a difficult to reach population since many are lower income, younger, undocumented immigrants, and members of racial/ethnic minority groups, and may change phone numbers or move more often than the public at large. After data collection was complete, data from Wave 1 and Wave 4 were compared to evaluate the impact of some respondents not completing Wave 4, referred to as attrition. While there are some differences in the unweighted demographics of those who completed Wave 4 and the full Wave 1 sample, these observable differences are corrected for by weighting. As shown in the table below the total weighted distributions are similar for Wave 1 and Wave 4 for age, gender, race/ethnicity, party identification, education and income. See the Wave 4 Attrition Appendix for more information on attrition.

Another consideration for panel surveys is the potential for “sensitization effects,” that is, what effect returning to the same people about the same topics has on their experiences or views. For example, after taking the baseline survey that covered many aspects of the coverage expansions under the ACA, were people more likely to seek out information about health insurance and enroll than they would have been otherwise? While there is no direct way to measure this effect on this survey, other analyses have found that these effects are minimal and short-lived,² and we do not believe they would have had a substantial impact on results presented here, particularly given all the other media coverage, advertising, and outreach targeted at this population during the fall and winter of 2013, 2014, and 2015.

The margin of sampling error including the design effect for the full sample is plus or minus 5 percentage points. For the recently insured, it is plus or minus 5 percentage points and for the remaining uninsured it is plus or minus 9 percentage points. Numbers of respondents and margin of sampling error for key subgroups are shown in the table below.