A Quick Guide to Checking Data Quality in Medicaid Health Care Claims Data

Accurate data are essential for making informed Medicaid policy decisions and ensuring that analyses of health care utilization and outcomes truly reflect the services provided to beneficiaries. To support this, authorized Medicaid providers submit claims for services rendered to state Medicaid programs. 

States maintain their Medicaid enrollment and claims data for all submitted claims. To have a standardized national dataset of Medicaid claims across states, each state submits their Medicaid enrollment and claims data to the Centers for Medicare & Medicaid Services (CMS). 

Data from each state are combined into the Transformed Medicaid Statistical Information System (T-MSIS). The T-MSIS data are then processed, standardized, and made available to researchers and others outside of CMS via the T-MSIS Analytic Files (TAF). The TAF data are a valuable source for analyzing health care utilization and outcomes within the Medicaid program. However, because the Medicaid data in the TAF have been submitted to CMS separately by each state, the data have some unique quality concerns. 

There are several important factors to consider when evaluating TAF data quality, including completeness, validity, consistency, reliability, and accuracy of the Medicaid data for each state, year, and variable.  It is also important to assess whether issues identified could potentially influence outcome creation and research findings. Below is the process RTI International uses to assess the quality of TAF data for analysis and a few examples of how we have put the process into practice. 

To assess the quality of TAF data for an analysis, RTI takes a three-step process:

1. Using DQ Atlas for Medicaid Data Quality Evaluation

   First, we check the data quality using the publicly available tool from CMS: the DQ Atlas. The Data Quality (DQ) Atlas provides a comprehensive framework to assess various dimensions of data quality—such as completeness, accuracy, and validity—and is a useful starting point to assess the quality of TAF data. RTI uses the DQ Atlas to identify red flags in validity, accuracy, missingness, and volume of data by state and year. 

   However, TAF users cannot rely on the DQ Atlas alone to assess data usability. There are several limitations to the information provided in the DQ Atlas. Specifically, it only presents issues identified across the Medicaid population as a whole and by calendar year and file version. As such, the DQ Atlas assessments lack sufficient granularity for checking specific subpopulations, such as those receiving behavioral health treatment. Examining issues only by year does not flag issues that emerge with reliability and consistency by examining changes in trends over time. Further, examining issues by file version does not identify issues that may arise between preliminary and final versions of the file. Finally, the DQ Atlas is limited to a select group of variables and may not include all variables of interest (for example, county code-—which is instrumental in analyses of rurality—is not included).

   RTI uses TAF data to evaluate programs that span many states and time periods. For example, the evaluation of the Accountable Health Communities Model (AHC) includes a small sample of beneficiaries from 23 states with measurement periods that span across calendar years. We found that some issues flagged in the DQ Atlas as highly concerning for a state (such as low inpatient claim line volume in Oklahoma) did not impact our analysis for our sample, while others did (such as low total expenditures data in Pennsylvania relative to expenditures reported in CMS-64 data). For the Section 1115 Federal Meta-Analysis project, we observed a dramatic reduction in other therapy (OT) claims between the preliminary and final versions of the 2021 TAF data for Idaho that impacted our analysis. Despite the DQ Atlas showing the underlying decrease in the outpatient claims denominator (5.8 million to 3.6 million), its current design requires users to manually compare versions to detect such significant data shifts, as it lacks an automated flagging system for these discrepancies.

2. Conducting Independent Data Quality Analysis on Medicaid Data

   Second, we conduct an independent analysis of TAF using our specific population of interest. To build on what we learn from the DQ Atlas, RTI conducts an independent analysis on the population of interest for a particular research question. To assess the data for the outcomes of interest for our research sample, we check the following:

   Claims volume. RTI reviews the count of overall claims/encounter records (volume) as a trend over time for the sample of interest. This analysis serves to identify whether claims volume issues identified in the DQ Atlas are impacting the sample. 

   Missingness and validity of critical data elements. To review data completeness, we first identify all data elements that are critical to our analysis (e.g., those that could affect our ability to identify the research sample and/or outcomes of interest). RTI then assesses the missingness for each critical data element for the sample and assesses validity by examining values of data elements and whether they are feasible. For example, we examine the values relative to relevant coding systems (e.g., ICD-10 diagnosis codes, procedure codes). We also evaluate a field’s range of values to identify outlying or illogical values or formats. If we identify problematic values, we consider whether there is an appropriate fix to the observed values. For example, as noted in the DQ Atlas, Tennessee has invalid diagnosis codes for over 90% of records in the inpatient file for 2017 to 2019. However, the codes are invalid due to leading zeroes that can be deleted to obtain valid diagnosis codes. 

   Average out of range. To assess consistency, we compute the average value of the study outcomes for our sample in each state and compare it to the overall average across all states. We would consider a state as out of range if its average was ± two standard deviations from the overall average. 

   Trend over time. To assess reliability, we examine trends over time within a state to determine whether there are any anomalous data points that could be due to data issues in a specific year. We anticipate noticeable shifts when states make policy changes—for example, when they expand Medicaid. States with inconsistent trends or large increases that are not associated with either Medicaid expansion or other policy changes generally are placed on a higher level of concern.

3. Benchmarking Medicaid Data Against External Sources for Accuracy

   Finally, we benchmark study outcome values to outside sources. We compare measures to outside sources or alternative calculations of those measures to determine whether our results are similar to expected values. For example, for study outcomes that are Healthcare Effectiveness Data and Information Set (HEDIS) measures, we compare the calculated rates to those reported on the HEDIS website for Medicaid managed care enrollees. We took this approach when conducting Medicaid claims data analyses in the State Innovation Model Round 2 evaluation.   

When an issue is identified, we have choices in how we handle it. First, when possible and as appropriate, we apply a work around to fix or improve the data issue. For example, as noted above, we fixed the diagnosis code formats for Tennessee so that we can continue to use their inpatient data. We can also triangulate multiple data elements to define an outcome. For example, for emergency department visits, we utilize both revenue codes and procedure codes to identify visits to allow flexibility if one of the codes is missing or invalid.

Second, we may proceed to use TAF data with potential quality concerns but add caveats to any reporting and interpretation of the outcome. For example, for the Section 1115 Federal Meta Analysis project, we have noted when lower than expected claims volume in the TAF data may lead to underestimations of the number of outpatient services for a given state.

Third, we may exclude a certain state or year from an analysis if the data are poor while still proceeding with the analysis for other states and/or years. For example, for the AHC evaluation, we excluded Indiana and Pennsylvania from the study sample for the calculation of total expenditures due to poor data.

Finally, as a last resort, we may conclude that it is not possible to conduct an accurate analysis of a certain outcome for a given state and time. For example, for the Section 1115 Federal Meta-Analysis project, we cannot calculate a measure of follow-up with a mental health provider after a mental illness discharge due to a lack of provider data in some states. 

By conducting thoughtful, comprehensive, independent analyses of the Medicaid data for the sample of interest, RTI gains deeper insights into the strengths and weaknesses of the TAF data. This comprehensive approach enhances the accuracy and validity of findings we present to policymakers, ultimately improving the quality of health care analysis within the Medicaid program.