Measuring Progress: Metrics

The NPO was charged with measuring AF4Q progress on two levels—the individual communities’ performance and overall program performance. Over the course of AF4Q, the progress of the 16 Alliances and the program were measured against a set of metrics developed by the Foundation. The NPO, specifically regional support, took a leading role in determining and explaining the criteria supporting those metrics.

The mechanism for measuring community performance was Alliance quality and cost goals. Each of the 16 Alliances established goals that aimed to improve quality and decrease costs in their respective markets. The Alliances worked closely with stakeholders to develop goals that related to specific problems in their communities. The NPO tracked progress toward the goals using data submitted by the Alliances.

Assessing how the overall program was performing—in other words, trying to measure “success”—was slightly more complex. The initial decision on the part of RWJF to “let many flowers bloom” (i.e. Alliances were able to choose community-specific measures) meant that each Alliance’s goals had local relevance and community support, but differed from other communities’ goals. This lack of ability to perform direct comparisons resulted in the development of dashboard measures and their second generation, quality/equality indicators. These internal metrics attempted to capture broad signs of success across the 16 sites. They were monitored by the NPO and were the basis of many conversations between the NPO and Foundation as the program was assessed.

Regional support was the gateway for data flow from the Alliances. At regular intervals throughout the year, each Alliance required to submit a comprehensive report outlining the activities, successes and challenges, related to their goals and the goals of the program. Regional support developed a system for collecting, storing, and tracking the Alliances’ data. The qualitative mechanisms for data flow included summary reports for each of the content areas that provided a snapshot of progress in specific areas, as well as detailed analysis of individual Alliance progress in these areas. The quantitative mechanisms for data flow revolved around a complex Excel database that housed all data fields relevant to the Alliances’ quality and cost goals. This database required substantial upkeep and review, and allowed the NPO to run various analyses around data gaps and trends.