What Types of Performance Measurement and Evaluation are Envisioned?

We envision that all pilots will be required to:

• Select outcome measures and interim progress indicators in the domains of education and employment and, based on pilot-specific objectives, select supplemental measures or indicators in the domains of criminal justice, physical and mental health, and other measures of positive youth development and well-being that will be appropriate and sufficient to gauge the pilot’s effectiveness.
• Maintain reliable administrative data on individual participants, the services they receive, programs in which they participate, and initial outcomes that can be linked across the chosen outcome domains.
• Cooperate with any evaluation carried out or commissioned by the Federal government, including providing timely access to appropriate data and to program personnel and participants. The Federal government currently plans to initiate a systems analysis and process evaluation to document the development, governance, management, partnerships, planning process, and information systems.

Stronger applications would also:

• Establish a baseline to assess progress over time and an appropriate comparison group to evaluate outcomes.
• Demonstrate strong expertise to analyze data and interpret findings in order to inform program implementation and course corrections.

Especially strong applications would also:

• Incorporate strategies and interventions that have already shown promise or effectiveness in other rigorous evaluations.
• Incorporate a rigorous impact evaluation for at least some well-defined component of the pilot (e.g. a particular intervention or service delivery change) as well as a process evaluation.

Because the pilots may be authorized through 2018, potentially incorporating funds from future fiscal years, pilots may be encouraged or expected to use increasingly sophisticated data analysis and evaluation methods to fine-tune their strategies each year. For example, over time a pilot might find more precise ways to analyze how well and for whom an approach is working, such as by disaggregating participation or immediate outcome data by race, age, gender, or other subgroups. The Federal agencies are exploring options that would help pilot jurisdictions to secure strong expertise, necessary funding, and access to administrative data to conduct a high quality evaluation at reasonable cost. Options under consideration include:

• Allowing projects to use a small percentage of their blended funding for data infrastructure, data analytics and evaluation.
• Giving preference to pilot applicants that secure non-Federal funding for evaluation and form partnerships with highly qualified academic researchers to design and conduct evaluations.

To facilitate access to valuable administrative data and lower costs of performance measurement and evaluation, the Federal agencies are considering:

• Requiring all State and local partners in a pilot to demonstrate, through data sharing agreements for example, that pilots will have access to administrative data that can be used to manage the services and activities, and track progress and outcomes over time for the target population, using secure and up-to-date methods for linking data and protecting individual privacy.
• Facilitating access to federally held data, while providing strong privacy protections, in order to measure earnings, employment, and health outcomes.

Key questions for stakeholders:

• What reliable, high-value data sources could be used to improve quality and lower the cost of performance measurement and evaluation?
• What types of partners could help pilot jurisdictions, at a reasonable cost, set up strong data systems and performance management processes, and design rigorous evaluations that help pilot communities learn what works and is cost-effective?
• Are there other approaches to measuring outcomes and incorporating accountability and learning that we should consider?