Challenges in ROI Assessments

Efforts to identify, measure, and draw causal connections to benefits prove challenging for several reasons. One key challenge for a number of the grantees was accessing data to allow meaningful analysis of program benefits. While the data sources and challenges varied among Alliances, this challenge was characterized by a few common themes:

• Consumer engagement activities were not designed with evaluation in mind. Alliances were able to articulate the impact they expected the consumer engagement activities to produce. However, they implemented consumer engagement activities before the CE-ROI project, and the programs were not necessarily designed for rigorous evaluation. As a result, many lack the consistent metrics they need. For example, although many medical practices in Humboldt County expressed satisfaction with the Pathways to Health program and felt their patients benefited from participating, the practices did not collect common metrics (e.g., diabetes outcome measures) that would make it possible to compare program results.

• Data on the downstream impact of the program were not readily available. As noted above, when programs are not designed as evaluations, the needed data may not be available or consistent. Furthermore, a number of the interventions were intended to influence behavior for downstream impact—such as avoiding ED use or hospitalizations—for which data can be harder to obtain in a timely or reliable manner. For example, Cincinnati’s consumer education campaign aimed to reduce avoidable ED use,however, updated ED utilization data were not available to include in the analysis. Similarly, Memphis sought to evaluate the impact of a patient coaching program on patients’ ED use and inpatient admissions, but patient-level utilization data were not available.

• Multiple efforts underway in a community make it difficult to attribute impact to a particular activity. Many Alliances found it difficult (if not impossible) to attribute outcomes to the specific intervention or program they sought to evaluate. In Alliances, multiple efforts are underway to achieve a given goal (e.g., reduced ED visits or improved diabetes outcomes). It is difficult to isolate the benefit of a single intervention in the context of multiple efforts. For example, Washington conducted an ROI analysis of its Own Your Health campaign, a consumer engagement communication support for employers. The Alliance found that employees were aware of the concepts covered by the Own Your Health campaign, but they were not able to distinguish between the Own Your Health campaign and a larger campaign being implemented at the same time, called Health Matters.

• Stakeholders can articulate intangible value, but the monetary value may be difficult to quantify. Many stakeholders champion the value of consumer engagement for its impact on the culture of health care, the patient-centered nature of decisions and services, and other intangible benefits. However, even if stakeholders can make a compelling narrative case for the link between these intangible values and downstream quantitative impact, that link can be difficult to measure and demonstrate. For example, South Central Pennsylvania examined the ROI of involving patients as partners with providers in practice quality improvement (QI) efforts. While practices clearly articulated the value of integrating patients into the QI work, they were not able to estimate the resulting downstream benefits. Practice representatives felt that patientpartners helped increase the practice’s reputation for providing patient-centered care. The practice representatives believed improved reputation would lead to an increase in new patients but were unsure of the precise outcome because there was no formal measurement mechanism. Some impact may be found through further measurement and evaluation.