Data Note: Attempting to Measure Early Impact of the ACA through National Public Opinion Polls- A Note of Caution and What to Watch For

October 1, 2013 was a landmark date for the Affordable Care Act (ACA), with the start of open enrollment in the law’s Health Insurance Marketplaces and the growth of Medicaid enrollment in some states. Several weeks into open enrollment, many people – including journalists, policymakers, and the public at large – are eager for early data indicating how the law is working from the perspective of potential enrollees. In particular, given the problems with Healthcare.Gov and some of the state exchange websites, many people want quantitative data about people’s experiences attempting to purchase or enroll in some sort of health insurance coverage using these mechanisms. In the long run, data from large federal surveys will be available to help answer questions about the enrollment experience, but in the short term, some may turn to national public opinion polls to see what can be learned.

In this Data Note, we raise a note of caution about the possible pitfalls of using standard national public opinion polls to make judgments about Americans’ early experiences with health plan enrollment under the ACA.

What To Watch For: Incidence Of Key Groups Is Low

The primary issue with trying to assess the early impact of the ACA through national, random sample surveys that use a standard methodology and sample size comes down to raw numbers. There generally just aren’t enough people to look at.

Let’s start with the broadest group that stands to be impacted by the ACA’s coverage expansions: the roughly 40 million adults who are currently uninsured, comprising roughly 21 percent of the nonelderly adult population. While a substantial share by most measures, this amounts to relatively few interviews when translated to a nationally representative, random sample survey.

To make it more concrete, a typical national public opinion poll with a total sample size of 1,200 adults yields between 140 to 175 interviews with uninsured individuals.¹ The margin of sampling error (MOSE) around survey estimates based on this group is in the range of plus or minus 8 to 10 percentage points. While it’s possible to use a poll of this type to report broadly on the views and experiences of the uninsured, the sample size constrains pollsters’ ability to detect changes over time. For example, in a typical national public opinion poll with a MOSE of plus or minus 3 percentage points, if the survey finds 35 percent of people overall report a certain behavior, the chances are very high (95 out of 100) that if you interviewed the whole population, the “real” answer would lie between 32 percent and 38 percent.  For a subgroup the size of the uninsured, with a MOSE of plus or minus 10 percentage points, the likely range of answers widens from 25 percent to 45 percent. One of the key questions people will want to answer about ACA enrollment is whether the share of the uninsured who are attempting to sign up for coverage is increasing or decreasing. With such a wide MOSE range around each point estimate, it will be difficult for pollsters to answer this type of question by detecting changes over time. Further, starting with a group of this size makes it difficult (and sometimes impossible) to analyze subgroups of interest within the uninsured (for example, young adults).

Another group that the media has focused on recently amid reports of cancelled insurance policies is those who purchase insurance on the individual market. This group amounts to about 8.5 million adults, or roughly 5 percent of the nonelderly adult population, and represents fewer than 100 interviews out of a typical 1,200 person survey. In this case, the MOSE is so large that most researchers consider estimates based on this population to be too unreliable to report.²

Further complicating matters is the fact that looking at all the uninsured as one undifferentiated group misses some of the important complexities of the law. In fact, there are many individual characteristics that will impact people’s eligibility for various coverage options under the ACA. For example, subsidies to help people purchase coverage are available for those within a certain income range based on the federal poverty level (FPL), which is calculated using a formula that includes household size and income.³ Asking the right questions to determine where an individual falls on the FPL scale is complicated, and survey respondents are often hesitant to divulge all of the information necessary to make the calculation. While researchers often ask these questions on surveys that focus on poverty and income, these questions take up valuable time on a survey, and are outside the scope of most national opinion polls.

Determining eligibility for the Medicaid expansion is also complicated. Prior to the ACA, each state had its own set of rules to determine eligibility. The law was initially intended to expand Medicaid to cover all individuals with incomes of 138 percent of the FPL or less, but the 2012 Supreme Court ruling made the Medicaid expansion optional for states. As a result, determining whether an individual is eligible for Medicaid depends not only on where they fall on the FPL scale, but also on the state in which they reside. The bottom line for national public opinion polls with traditional sample sizes in the range of 1,000 to 1,200 is that it will be impossible to break out the experiences of these various groups with any specificity.

It is also worth noting that some people will think they are eligible for coverage under the ACA when in fact they are not.  For example, undocumented immigrants are not eligible to enroll in Medicaid or participate in the exchanges, but in some parts of the country they make up a significant share of the uninsured.  A recent Kaiser survey of the uninsured in California found that about a fifth of the state’s uninsured self-identified as being undocumented,⁴ and that nearly half this group thought they may be eligible for coverage through Medicaid or the California exchange. Ideally, the undocumented should be excluded from survey estimates of enrollment experiences. However, like income, immigration status is a sensitive topic, and while it is possible to tackle this issue in more targeted projects like the Kaiser California Uninsured Survey, this is another topic that is outside the scope of most national surveys.

When it comes to quantifying the experiences of those who have enrolled so far, the case for using national surveys is even bleaker. The Obama Administration reported that as of the start of November, 106,185 people had enrolled using the exchanges and 396,261 were deemed eligible for Medicaid or the Children’s Health Insurance Program.⁵ This represents a tiny fraction of the overall adult population, and would be impossible to identify in a typical national random sample survey.

Even by the end of 2014, when CBO projects that roughly 7 million people will enroll in health insurance through the exchanges and 9 million will gain coverage through Medicaid or the Children’s Health Insurance Program,⁶ this only represents roughly 2 to 3 percent of the total nonelderly population (including children) and is still too few to expect to reliably break out in a national survey with a typical sample size.

What To Watch For: Impacted Groups Are Under-Represented In Surveys

Another complicating factor in using standard national public opinion polls to measure early ACA impact is that some of the groups most likely to have early experiences with the law, including those with lower incomes and those who speak a language other than English, are populations that are often under-represented in surveys. Weighting survey data to national demographic benchmarks helps account for this under-representation and ensures that the overall survey results are nationally representative, but the fact remains that this phenomenon exacerbates the issue of having too few interviews with individuals in these groups to reliably describe their experiences with the law.

This is even the case with the Kaiser Health Tracking Poll, despite the fact that we take special measures and make extra efforts to maximize inclusion of these groups, such as completing half the survey interviews via cell phones (which younger people and racial and ethnic minorities use in disproportionate numbers) and offering respondents the option of being interviewed in Spanish or English. The tracking poll has a minimum overall sample size of 1,200 and in some months is increased to 1,500. These extra measures all increase the cost of conducting a survey, which does not fit within many polling organizations’ budgets. As a result, many polls reported in the news are based on surveys with smaller overall sample sizes, fewer cell phone interviews, and English-only interviewing. While these methodological details may not have a significant impact on a survey’s ability to represent the overall views of adults nationally, they can make a big difference when it comes to measuring the experiences of some of the groups described above. Surveys conducted via the Internet can be particularly problematic, given the lower level of Internet access and online experience among this population.

What To Watch For: Asking The Right Questions To Determine Actual Experiences Is Complicated

Another major challenge is designing survey questions that will correctly identify those who actually tried to sign up for coverage through the ACA and what their experiences were. While it might be tempting to just ask Americans whether they had tried to sign up for health insurance at Healthcare.Gov, this would certainly provide a misleading result. While much of the news coverage has focused on enrollment through the federal website, there are actually many different ways people can start the enrollment process, many different pathways through the process, and many different potential outcomes. Individuals attempting to sign up online may start not at Healthcare.Gov but at their own state’s exchange website (if they live in a state that is administering its own exchange). Alternatively, they may go to a state Medicaid office, meet with an ACA Navigator or Assister, or call either a national or state-based telephone help line. Some may start the process at the health insurance exchange but end up enrolling in Medicaid. Others may find that they’re eligible to purchase coverage through the exchange but not to receive federal subsidies to help pay for their coverage. Some may go to the website or seek in-person help with the intention of signing up for coverage right away, and others may go to gather information and plan to sign up for coverage later. Many will have no idea that their experience was brought to them by the ACA at all. These are just a few of the factors that complicate any effort to quantify a single “enrollment experience.”

Moving Forward: What Can We Learn About ACA Implementation From National Public Opinion Polls, And What Other Vehicles Might We Look To For Reliable Information?

While not a good vehicle for measuring early enrollment experiences, national public opinion polls can provide other types of useful information for those trying to gauge how ACA implementation is going. For example, polls can measure the public’s perceptions (how well do people think the rollout is going?), understanding and awareness (are they learning more about the law over time?), media exposure (what are they hearing on the news and what is their reaction to the news?), and opinions (does all of this change how they feel about the law overall, or not?). National polls can also be used to gauge the types of discussions people are having with their family and friends about their experiences, giving a sense of what the national dialogue is like. These are the types of questions we have asked over time in the Kaiser Health Tracking Poll, and we’ll continue to do so going forward.

With the Kaiser Health Tracking poll, we’ll also continue to report broadly on the views of the uninsured. While we can’t use this survey to break them down into the Medicaid eligible or the exchange shoppers, or to quantify their enrollment experiences, we can measure things like the extent to which they feel they understand the law, the extent to which they feel the law is helping or hurting them, their awareness of the individual mandate, and whether they intend to seek coverage over the next year or plan to remain uninsured.

For more in-depth data on enrollment experiences, we’ll rely on specially designed projects, like the Foundation’s panel survey of 2,000 randomly selected uninsured Californians.¹³ The baseline survey was conducted last summer, before the ACA open enrollment period, and the next wave will follow up with the same respondents in March, exploring who signed up for coverage, what choices they made, and how they evaluate their experience and the coverage they obtained. Projects like this take a longer time to field than a typical national opinion poll, but allow us to address some of the methodological concerns raised above to provide reliable data describing the experiences of key groups targeted by coverage expansion under the ACA.