In December 2007, Abt Associates and Vanderbilt University completed an independent evaluation of the Quality Indicator Survey (QIS) methodology.  On June 24, 2009, the Centers for Medicare and Medicaid Services (CMS) issued S&C-09-04 to state survey agencies, announcing the release of the “Evaluation of the Quality Indicator Survey (QIS)” and the posting of the Executive Summary on the Centers for Medicare & Medicaid Services’ (CMS) website as well as outlining an action plan for future QIS evaluation initiatives.  It is unknown why it took CMS 18 months to release the report.  Minnesota QIS surveys were not included in the study.

The evaluation has a number of flaws, the most important being the evaluators only reviewed 20 QIS surveys to make comparisons to traditional surveys.  The evaluation was designed to answer questions about accuracy, documentation, changes in the number and type of deficiencies, and whether the QIS process is more efficient.  CMS believes that improved consistency is inherently embedded into QIS processes, so this was not evaluated.  The study instead assessed whether the QIS also had beneficial effects on other aspects of the survey process, such as improving the accuracy of citations.

Key findings of the report include:

1. Does the QIS lead to increased accuracy?  Based on the relationship between survey findings and a set of care indicators intended to measure the quality of care provided by nursing facilities, they did not find evidence that the QIS was more accurate than the Standard survey.  The results suggested that more survey deficiencies with scope greater than isolated could have been cited for both QIS and Standard surveys.  Ultimately, under both types of surveys, there appears to be a great deal of surveyor discretion and judgment that influences the decision to cite.

2. Does the QIS result in improved documentation of survey deficiencies?  They found essentially no differences in documentation quality associated with the QIS, although interrater reliability concerns limit the strength of this conclusion.

3. How does the time required to complete the QIS compare to the time required for the current survey?  Results indicate that there is nothing inherent in the QIS which indicates that it cannot be resource neutral.  They found that the QIS took considerably longer to complete than Standard surveys in two of the five demonstration states; two states consumed about the same amount of time and one state’s time was open to different interpretations.

4. How does the QIS impact the number and types of deficiencies that are cited?  The results of this evaluation clearly indicate that the QIS cites more deficiencies, at higher levels, and more in these usually under-cited areas.

5. Does the QIS improve surveyor efficiency?  The correlation between time and deficiencies was higher for QIS surveys than for Standard surveys.  Ohio was the only state for which the QIS was associated with an increase in surveyor efficiency.  A number of recommendations for improving the QIS emerged from the field work conducted as part of the summative evaluation and the earlier formative evaluation.  These recommendations focused on ways to improve the accuracy of the QIS.

6. Improve specificity and usability of investigative guidelines.  The care elements that are recommended for investigation in existing interpretative guidelines and critical element pathways should be modified so that they are consistent with the principles that guide reliable and accurate measurement.

7. Provide competency-based training for surveyors to improve consistency.  Provide survey staff with training in the principles of reliable measurement, and document that the trained surveyors can use the investigative protocols to produce consistent and accurate quality conclusions.

8. Evaluate how well the QIS Stage I and unstaged protocols identify problem areas that should be investigated in Stage II.  If the QIS is accurately detecting areas for investigation, then quality measures for facilities that are flagged for an investigation should be different and worse than the measures for facilities that are not flagged.  They did not find these differences in their evaluation of QIS accuracy, suggesting that the question of whether Stage I accurately identifies areas in which there are potential quality problems and which are thus the best targets for Stage II investigations is relevant.

9. Increased structure in Stage II to make decision making more explicit in determining noncompliance, scope, and severity.  Despite the structure of Stage I that ensures that surveyors conduct a more comprehensive survey and utilizes more information from residents and families, the process becomes increasingly subjective in Stage II and during certain facility-level tasks.  Of highest priority is the development of additional CE pathways for the many important care areas where these do not exist.  A second priority is to improve the integration of the CE pathways into the Stage II investigation.

Because the evaluation did not find improved accuracy, CMS had concluded that non-QIS factors, including (a) survey guidance clarification, (b) training of surveyors, and (c) surveyor supervision are prudent approaches to improvement of accuracy.  CMS continues to issue improved surveyor guidance as well as to strengthen surveyor training.  CMS also concluded that future QIS development efforts should concentrate on building upon the QIS strengths relative to consistency improvement, and giving supervisors more tools to assess performance of surveyor teams.

The S&C letter also includes a chart outlining an action plan for design, testing, and evaluation of the QIS system through 2011.  Click here to view the S&C letter, and click here to view the Executive Summary of the QIS evaluation.

Doug Beardsley
952.851.2489
dbeardsl@careproviders.org

Our Sponsors 

Resources