P3 Report to Congress 2017
The report (S. Rept. 114-74) accompanying S.1695 (titled “Departments of Labor, Health and Human Services, and Education, and Related Agencies Appropriations Act, 2016”), passed by the Senate Committee on Appropriations, requested annual reports containing information about the Performance Partnership Pilots for Disconnected Youth (P3) program. Specifically, the Senate report requested “annual reports containing the following information: a detailed summary of all involved pilot programs, an overview of how pilots were selected, a summary of findings from the various pilots, and recommendations for Congress on how to apply best practices more broadly.”
Because the pilots are still in the early stages of implementation, we have limited information regarding findings from individual pilots or recommendations regarding best practices and opportunities for broader application. This annual report, the first prepared in response to this request, provides detailed information about the P3 program, including descriptions of the individual pilots and how they were selected, and identifies questions and considerations that highlight preliminary lessons from the P3 program. Additional detail is included as appendices. As implementation of the pilots continues and the evaluation findings become available, we will gain a more complete understanding of the P3 program, including the outcomes achieved by individual pilots, and the strengths and limitations of the P3 program model, which we intend to continue sharing with Congress and the public.
P3 offers a unique opportunity to test innovative, cost-effective, and outcome-focused strategies for improving results for disconnected youth, and may also offer broader lessons regarding ways in which various federal grant programs can enable improved outcomes by allowing greater flexibility for grantees. According to the U.S. Census Bureau, more than 5 million 14-to-24- year olds in the United States are not on a clear path to postsecondary education or training and a rewarding career. They are neither working nor in school and, in many cases, face the additional challenges of being homeless, in foster care, or involved in the justice system. Often disconnected from their families and valuable social networks, these young people struggle to make successful transitions to adulthood and to reach the educational and employment milestones critical to escaping a lifetime of poverty. The P3 authority provides States and local and tribal governments greater freedom to innovate in order to serve these youth, providing an opportunity to request waivers of programmatic requirements that appear to impede effective service delivery.
Congress authorized the inclusion of funds from fiscal years 2014, 2015, and 2016, and authorized the selection of up to 10 pilots for each of those years. Federal agencies solicited applications for these three rounds of pilots and received 45 eligible applications across the 3 years. Applications were assessed and scored by peer reviewers using selection criteria published in the notices inviting applications. To date, nine pilots (Round 1) have been authorized, while another seven entities (Rounds 2 and 3) have been designated pilot finalists with which the Federal agencies are currently negotiating performance agreements. Descriptions of the pilots’ and pilot finalists’ projects appear in Appendix A. [Please see the PDF (30 pages) for Appendix A, which provides the Pilot Summaries for each Round's Pilot Sites.]
Implementation of the Fiscal Year 2014 Authority (Round 1)
Enacted on January 17, 2014, the Consolidated Appropriations Act, 2014 (Public Law 113-76) (the 2014 Act) authorized the Departments of Education (ED), Labor (DOL), and Health and Human Services (HHS), along with the Corporation for National and Community Service (CNCS) and the Institute of Museum and Library Services (IMLS) (the Agencies), to enter into performance partnership agreements that give State, local, and tribal governments greater flexibility in using funds awarded under two or more discretionary programs administered by the Agencies. The 2014 Act authorized the Agencies to enter into up to 10 performance partnership agreements.
States, localities, and tribes that entered into P3 agreements were authorized to request waivers for requirements related to certain discretionary funds, including both formula and competitive grant funds, from the Agencies in order to implement strategies to improve outcomes for disconnected youth. The 2014 Act states that “‘[t]o improve outcomes for disconnected youth’ means to increase the rate at which individuals between the ages of 14 and 24 (who are low-income and either homeless, in foster care, involved in the juvenile justice system, unemployed, or not enrolled in or at risk of dropping out of an educational institution) achieve success in meeting educational, employment, or other key goals.” This waiver authority provided the authority to blend discretionary funding.
The 2014 Act states that heads of the Agencies may not only exercise any existing waiver authority but also may waive any statutory, regulatory, or administrative requirement that they are otherwise not authorized to waive, as long as the waiver is in keeping with important safeguards. Some of these safeguards include that waivers must be consistent with the statutory purposes of the respective Federal programs contributing funds to the pilot and necessary to achieve the pilot’s outcomes. In addition, the heads of the Agencies must determine that the waivers and the proposed use of program funds: (1) will not result in denying or restricting individual eligibility for services funded by those programs, and (2) will not adversely affect vulnerable populations that are the recipients of those services. Moreover, the Agencies may not waive requirements related to nondiscrimination, wage and labor standards, or allocation of funds to State and substate levels.
Consultation with the Field
The Agencies have considered public input critical to both the development and the effective implementation of the P3 authority. Anticipating P3 authority, in June 2012, ED published a Request for Information on Disconnected Youth in the Federal Register (77 FR 32959) (RFI) seeking public input regarding the implementation of the potential P3 statutory authority, which was first proposed by in the FY 2013 budget request. The 171 comments ED received in response to the RFI emphasized the need for greater flexibility, highlighted promising initiatives, and offered recommendations for effectively serving disconnected youth and administering P3. The responses informed the Agencies’ thinking about how best to implement the P3 authority when it was enacted into law in FY 2014.
On April 28, 2014, the Agencies issued a consultation paper, Changing the Odds for Disconnected Youth: Initial Design Considerations for Performance Partnership Pilots (PDF, 19 pages), that provided background information about the pilot authority and described the Agencies’ initial thinking about the implementation of the authority.1 The paper also encouraged stakeholders to respond by email to key questions about implementing P3 pilots. The Agencies used their research and analysis for the paper to guide two joint national webinars hosted by HHS on April 21 and 30, 2014, that provided information to the field and solicited feedback about the implementation of the authority.
In addition, on July 31, 2014, ED published a notice in the Federal Register that invited public comment on the application process for P3 (79 FR 44436). ED sought comments on the following three questions:
- What information, in addition to the information required by the Act, should entities be required to submit in their applications?
- What criteria should the Agencies use to evaluate applications?
- What technical assistance would be helpful to entities in preparing their applications?
ED received eight responses to the notice that made various recommendations with respect to the pilot application requirements and process. These recommendations informed the Agencies’ development of the application process. For example, one respondent recommended that applicants provide a logic model, while another recommended that the Agencies give priority to projects that are likely to achieve significant improvements in the outcomes of disconnected youth and projects that will replicate evidence-based approaches.
Round 1 Notice Inviting Applications (FY 2014)
The Office of Management and Budget (OMB) designated ED as the lead Federal agency responsible in FY 2014 for soliciting applications and managing the P3 initiative on behalf of the Agencies.
ED published a notice inviting applications in the Federal Register on November 24, 2014 (79 FR 70033) (Round 1 notice) that described how pilots would be selected for the first P3 competition. The notice included three absolute priorities, which, for the purposes of the first round P3 competition, created separate categories for scoring and considering applications. The Agencies included these absolute priorities to test the theory of action underlying the P3 authority in a variety of settings. These absolute priorities were:
- Absolute Priority 1—Improving Outcomes for Disconnected Youth. Under this priority, we considered applications that proposed pilots that were designed to improve outcomes for disconnected youth who resided in communities that were not included in Absolute Priorities 2 and 3.
- Absolute Priority 2—Improving Outcomes for Disconnected Youth in Rural Communities. Under this priority, we considered applications that proposed to improve outcomes for disconnected youth in one or more rural communities only.
- Absolute Priority 3—Improving Outcomes for Disconnected Youth in Tribal Communities. Under this priority, we considered applications from a partnership that included one or more Indian tribes that proposed to improve outcomes for disconnected youth in one or more Indian tribes.
The Round 1 notice also included three competitive preference priorities that awarded additional points to applications that met certain requirements. The first two priorities awarded additional points to applicants that proposed to implement an independent evaluation of their projects, while the third awarded additional points linked to select local communities. Under Competitive Preference Priority 1, up to 5 additional points were available for applicants that proposed to carry out an independent evaluation that used a quasi-experimental design, while 10 additional points were available under Competitive Preference Priority 2 to applicants that proposed to carry out an independent evaluation that was a randomized controlled trial. Applicants’ evaluation proposals were scored based on the clarity and feasibility of the proposed evaluation design and the applicants’ demonstrated expertise in planning and conducting a study that used a quasi-experimental design or that was a randomized controlled trial. The third competitive preference priority awarded two points to projects that were designed to serve and coordinate with a Federally designated Promise Zone.
Additionally, the Agencies made available to the FY 2014 P3 pilots start-up grants of up to $700,000 each to support ongoing planning, streamlined governance, strengthened data infrastructure, improved coordination, and related activities to help pilots improve outcomes for disconnected youth.
Applicants were given 100 days to submit applications. To promote the opportunity and provide technical assistance to prospective applicants, the Agencies held two webinars that were hosted by DOL. The first webinar focused on the notice inviting applications, and the second provided technical assistance to prospective applicants that were considering proposing independent evaluations. These webinars were viewed collectively by 777 individuals. The Agencies also published answers to frequently asked questions about the notice and the pilot authority at www.youth.gov/P3 and updated the guidance as needed throughout the application period.
ED received 27 eligible applications in response to the Round 1 notice. Using the selection criteria published in the notice, Federal staff from the Agencies, as well as from the U.S. Department of Agriculture and the U.S. Department of Housing and Urban Development, with expertise in areas relevant to P3, such as dropout prevention, intervention, and re-engagement and improving employment outcomes for vulnerable youth and young adult populations, reviewed and scored these 27 applications. The selection criteria addressed: (1) the need for the project (up to 5 points); (2) the need for requested flexibility, including blending of funds and other waivers (up to 10 points); (3) project design (up to 25 points); (4) work plan and project management (up to 10 points); (5) partnership capacity (up to 15 points); (6) data capacity (up to 30 points); and (7) budget and budget narrative (up to 5 points). In addition, reviewers who had expertise in research and evaluation scored Competitive Preference Priorities 1 and 2, the two priorities for independent evaluations.
Following this review, the Agencies reviewed the top-ranked applications and the waivers they sought to determine whether they could be granted under the P3 authority or existing waiver authority, whether the inclusion of a program or particular waiver would be consistent with other significant legal or policy considerations, and whether the waiver was needed. To clarify their waiver requests, the top-ranked applicants were asked to respond in writing to Agency questions and to participate in one or more telephone conference calls. The Agencies selected nine applicants to be finalists for Round 1 pilots. On September 30, 2015, ED awarded $6,154,998 in start-up funds to these nine applicants, but conditioned their access to these funds on their signing a final performance agreement with the Agencies that administered programs included in the project. Abstracts of the nine projects appear in Appendix A.
The Agencies then proceeded to negotiate performance agreements with the nine entities. Section 526(c)(2) of Division H of the 2014 Act specifies the contents of these agreements. The 2014 Act indicated that each agreement must include:
- The length of the agreement, which could extend to September 30, 2018;
- The Federal programs and Federally funded services that are involved in the pilot;
- The Federal discretionary funds that are being used in the pilot and the period of availability for the obligation of such funds;
- The non‐Federal funds that are involved in the pilot, by source (which may include private funds as well as governmental funds) and by amount;
- The State, local, or tribal programs that are involved in the pilot;
- The populations to be served by the pilot;
- The cost‐effective Federal oversight procedures that will be used for the purpose of maintaining the necessary level of accountability for the use of the Federal discretionary funds;
- The cost‐effective State, local, or tribal oversight procedures that will be used for the purpose of maintaining the necessary level of accountability for the use of the Federal discretionary funds;
- The outcome (or outcomes) that the pilot is designed to achieve;
- The appropriate, reliable, and objective outcome‐measurement methodology that will be used to determine whether the pilot is achieving, and has achieved, specified outcomes;
- The statutory, regulatory, or administrative requirements related to Federal mandatory programs that are barriers to achieving improved outcomes of the pilot; and
- Criteria for determining when a pilot is not achieving the specified outcomes that it is designed to achieve and subsequent steps, including: (1) the consequences that will result; and (2) the corrective actions that will be taken in order to increase the likelihood that the pilot will achieve such specified outcomes.
Implementation of the Fiscal Year 2015 Authority (Round 2)
ED continued to be the lead Federal agency responsible in FY 2015 for soliciting applications and managing the P3 initiative on behalf of the Agencies.
Notice of Proposed Priorities
The Consolidated and Further Continuing Appropriations Act, 2015 (Public Law 113-235) (the 2015 Act) extended the P3 authority to provide for up to 10 additional pilots and allow for the inclusion of certain funds from the Department of Justice. To implement this authority and solicit public comment, on October 22, 2015, ED published on behalf of the Agencies a notice of proposed priorities, requirements, definitions, and selection criteria (NPP) in the Federal Register (80 FR 63975). The priorities, requirements, definitions, and selection criteria proposed in this notice were based largely on the priorities, requirements, definitions, and selection criteria that were used in the November 2014 notice inviting applications for Round 1. For example, as in the November 2014 notice, the NPP proposed priorities for projects that serve disconnected youth in rural, tribal, and other communities. One of the lessons learned in Round 1 was that the use of absolute priorities promoted greater diversity in pilot settings than might otherwise have been achieved. In Round 1, the Agencies selected one project that proposed to serve a rural community (Eastern Kentucky Concentrated Employment Programs) and one project submitted by a partnership led by an Indian tribe that proposed to serve disconnected youth in an Indian tribe (Ysleta del Sur Pueblo). However, the Round 1 notice and the NPP documents differed in several other important respects:
- Additional Priorities for High-Need Subpopulations. The NPP also proposed additional priorities that focused on high-need subpopulations of disconnected youth, including priorities for disconnected youth who are unemployed and not enrolled in education, English learners, individuals with disabilities, homeless, in foster care, involved with the justice system, and refugees or other immigrants. In Round 1, the Agencies learned that State, local, and tribal governments may need additional incentives to focus their P3 projects on a variety of high-need subpopulations of youth. Most of the 27 applications that were submitted in Round 1 focused on disconnected youth who were at risk of dropping out of an educational institution. As a result, the Agencies proposed these additional priorities to create a means of providing incentives to applicants to serve the most vulnerable populations of disconnected youth.
- Priority for Work-Based Learning. In addition, the NPP proposed a priority for projects that provide paid work-based learning opportunities, including opportunities that are offered during the summer months, which are integrated with academic and technical instruction. As indicated in the NPP, the Agencies proposed this priority because the employment rate among youth has declined precipitously over the last decade, and addressing the employment needs of disconnected youth is critical to improving their well-being and preparation for lives as productive adults.
- A Single Priority for Evaluation. The NPP proposed to establish a single priority for projects that would support evaluations that use either a randomized controlled trial or a quasi-experimental design, rather than two separate priorities for these two kinds of evaluations as the notice inviting applications for the first Round of P3 had done. Applicants’ proposed evaluation plans would be evaluated based on the quality of the proposed evaluation’s design, appropriateness of the design to best capture key pilot outcomes, the scale of the contribution the evaluation will make to the evidence base, and the applicant’s expertise in planning and conducting comparable studies.
- Streamlined Application Requirements. To reduce burden on applicants and facilitate the review process, the NPP proposed to eliminate or streamline several of the application requirements. During their review of the applications in Round 1, the Agencies determined that some of the information applicants were required to submit could be streamlined or simplified to reduce burden on applicants, as well as to facilitate the timely review of applications by reviewers. Additionally, the NPP proposed to collect some of this information in table form to make clearer to applicants all of the data they must provide in their applications and to simplify how applicants provide these data.
- Simplified Selection Criteria. The NPP also proposed streamlined and simplified selection criteria to reduce burden on applicants, as well as to improve their focus on the factors that the Agencies consider to be most critical to the successful implementation of pilots.
Eleven parties submitted comments in response to the NPP. A key theme of the comments was concern that the FY 2014 selection process for pilots was too complicated. Generally, respondents encouraged the Agencies to streamline further the application requirements and simplify the review process. Some respondents encouraged the Agencies to direct or encourage applicants to focus on youth with the greatest needs and recommended additional priorities (e.g., applicants proposing to serve disconnected youth who are parents, applicants proposing to serve urban communities that have experienced violent protests) and supported additional priorities that were proposed (e.g., immigrant or refugees, English learners, homeless youth).
Other commenters were concerned that introducing new priorities in the competition under the FY 2015 authority would further complicate the process.
Notice of Final Priorities
On behalf of the Agencies, ED published a notice of final priorities, requirements, definitions, and selection criteria (NFP) in the Federal Register on April 28, 2016 (81 FR 25339). In response to public comment, the NFP reduced burden on applicants by removing several application requirements that had been proposed in the NPP. The NFP also revised the priority for disconnected youth who are unemployed and out-of-school to limit it to those youth who face significant barriers to accessing education and employment. Additionally, the NFP revised the priorities for projects designed to improve outcomes for subpopulations of high-need disconnected youth (i.e., youth who are unemployed and out of school, youth who are English Learners (ELs), youth with a disability, homeless youth, youth in foster care, youth involved in the justice system, and youth who are immigrants or refugees) to specify that, in order to meet the priority, a project must serve the particular subpopulation identified in the priority and also be likely to result in significantly better educational or employment outcomes for the subpopulation. Finally, in response to a public comment, the NFP established an additional priority for projects that serve youth who are pregnant or parenting and that are likely to result in significantly better educational or employment outcomes for such youth.
Round 2 Notice Inviting Applications (FY 2015)
On behalf of the Agencies, ED published a notice inviting applications (NIA) in the Federal Register on April 26, 2016, (81 FR 24573) for FY 2015 (Round 2 notice).
The Round 2 notice included four absolute priorities, which, for the purposes of the second round of P3 competition, created separate categories for scoring and considering applications:
- Absolute Priority 1—Improving Outcomes for Disconnected Youth. Under this priority, the Agencies considered applications that were designed to improve the outcomes of disconnected youth who reside in communities that are not included in Absolute Priorities 2, 3, or 4.
- Absolute Priority 2—Improving Outcomes for Disconnected Youth in Rural Communities. Under this priority, the Agencies considered applications that were designed to improve the outcomes of disconnected youth in one or more rural communities only.
- Absolute Priority 3—Improving Outcomes for Disconnected Youth in Tribal Communities. Under this priority, the Agencies considered applications from a partnership that included one or more Indian tribes that were designed to improve the outcomes of disconnected youth in one or more Indian tribes.
- Absolute Priority 4—Improving Outcomes for Disconnected Youth in Communities that Have Recently Experienced Civil Unrest. Consistent with language that Congress added in the Consolidated Appropriations Act, 2016 (Public Law 114-113) (the 2016 Act), under this priority, the Agencies considered applications that were designed to improve outcomes for disconnected youth in one or more communities that have recently experienced civil unrest. The notice indicated that, though the economy has recovered strongly in many places, many communities continue to struggle with high youth unemployment, low graduation rates, and crime. The notice stated that these and other continuing challenges can manifest in different instances of civil unrest, such as large protests or instances of civil disobedience, increases in self-directed or interpersonal violence in concentrated areas, or civic disorder prompted by a public health emergency.
The Round 2 notice also included four competitive preference priorities that allowed applicants to receive extra points for satisfying certain criteria:
- Competitive Preference Priority 1 awarded up to five additional points for projects that proposed to serve those disconnected youth who are neither employed nor enrolled in education and who also face significant barriers to accessing education and employment and that are likely to result in significantly better educational or employment outcomes for such youth.
- Competitive Preference Priority 2 awarded three additional points to projects that proposed to provide all disconnected youth it serves with paid work-based learning opportunities.
- Competitive Preference Priority 3 awarded two additional points to projects that were designed to serve and coordinate with a Federally designated Promise Zone.
- Competitive Preference Priority 4 awarded up to an additional 10 points to applications that proposed to conduct an independent evaluation of the impacts on disconnected youth of its overall program or specific components of its program that is a randomized controlled trial or a quasi-experimental design study.
Additionally, the Agencies announced that they would make available to the FY 2015 P3 pilots start-up grants of up to $350,000 to support ongoing planning, streamlined governance, strengthened data infrastructure, improved coordination, and related activities to help pilots improve outcomes for disconnected youth. The amount of the start-up grants was reduced based on competing priorities for agency funds and on the fact that one of the purposes of P3 is to focus on flexibility rather than new funding for the pilot sites. However, maintaining start-up grants was important based on feedback from the field in responses to the NPP that indicated that the start-up funding plays a critical role in launching and evaluating pilots.
Applicants were given 60 days to submit applications so that pilot finalists could be selected by the end of the fiscal year. To promote the opportunity and provide technical assistance to prospective applicants, the Agencies held a webinar on May 9, 2016, that was hosted by HHS. The webinar was viewed over 400 times during the first broadcast, and was available online for later viewing. The Agencies also published answers to frequently asked questions about the notice and the pilot authority at youth.gov/P3.
A number of applications received in response to the round 2 notice were not eligible because they did not meet all of the requirements in the notice, including the deadline for the submission of applications. To allow applicants additional time to prepare and submit their applications, ED reopened the competition on July 12, 2016, for one week to invite and accept additional applications. If applicants had submitted ineligible applications, they had time to address the issues that made their application ineligible and to submit a revised application.
In response to the notice reopening the competition, ED received three potentially eligible applications. Using the selection criteria published in the first notice, individuals with expertise in areas relevant to P3, such as dropout prevention, intervention, and re-engagement and improving employment outcomes for vulnerable youth and young adult populations, reviewed and scored these three applications. In addition, reviewers who had expertise in research and evaluation scored Competitive Preference Priority 4, the priority for independent evaluations.
The Agencies selected one applicant to be a finalist for designation as a Round 2 pilot, Maricopa County Education Service Agency in Phoenix, Arizona. On September 30, 2016, ED awarded $350,000 in start-up funds to this applicant but conditioned its access to these funds on its meeting certain conditions that would allow the agencies to grant the necessary waivers and its signing a final performance agreement with ED and DOL, the two agencies that administer programs that would be included in the project. An abstract that describes the proposed pilot appears in Appendix A.
Implementation of the Fiscal Year 2016 Authority (Round 3)
Round 3 Notice Inviting Applications (FY 2016)
The 2016 Act reauthorized P3, allowing for the creation of up to 10 additional pilots, as well as allowing for the inclusion of FY 2016 Homeless Assistance Grants administered by the Department of Housing and Urban Development (HUD). On behalf of the Agencies, ED published an NIA in the Federal Register on August 15, 2016, (81 FR 54056) for FY 2016 (Round 3 notice). This notice differed from the Round 2 notice in three respects:
- Homeless Assistance Programs. Consistent with section 242 of Division L of the 2016 Act, the Round 3 notice permitted applicants to include FY 2016 Homeless Assistance Grant programs administered by HUD. These are the Continuum of Care Program and the Emergency Solutions Grants Program.
- Deferred Application Requirements. The Round 2 notice required all applicants to include with their applications a memorandum of understanding or letter of commitment from each partner that described the terms and conditions of its participation in the pilot. All applicants for Round 2 also were required to include in their initial application an assurance that the State, local, or tribal government(s) with authority to grant any needed non-Federal flexibility, including waivers, has approved or will approve such flexibility within 60 days of an applicant’s designation as a pilot finalist; or include an assurance that non-Federal flexibility, including waivers, is not needed in order to successfully implement the pilot. To reduce burden on applicants, the Round 3 notice did not require applicants to include these documents in their applications. Instead, these documents were requested only from top-scoring applicants following the peer review of applications. Top-scoring applicants had 21 days after their notification by ED to submit these documents.
- Longer Application Period. The application period for Round 3 was 75 days, 15 days longer than the application period in the Round 2 notice. The deadline for applications was October 31, 2016.
As in Round 2, HHS hosted a webinar for prospective applicants on August 25, 2016. DOL, ED, and HHS staff participated in delivering the briefing. Eighty people viewed the live webinar, and a recording was viewed almost 600 times.
In response to the third notice inviting applications, ED received 15 potentially eligible applications. (An application is potentially eligible when the applicant meets the basic eligibility criteria but the application has not yet been reviewed for meeting all application requirements.) Using the selection criteria published in the first notice, individuals with expertise in areas relevant to P3, such as dropout prevention, intervention, and re-engagement and improving employment outcomes for vulnerable youth and young adult populations, reviewed and scored these applications. In addition, reviewers who had expertise in research and evaluation scored Competitive Preference Priority 4, the priority for independent evaluations.
National and Site Evaluations
The primary purpose of a demonstration such as P3 is to learn about new program models. To this end, in addition to giving priority to applicants that proposed a rigorous evaluation (described above) certain participating agencies were able to allocate resources to support a national evaluation. The national evaluation will document the extent to which P3 led to increased coordination across agencies and programs in support of greater flexibility for selected communities to improve outcomes for disconnected youth. The national evaluation also will synthesize individual site evaluations and, in doing so, highlight effective and promising practices for improving outcomes for disconnected youth. The national evaluation includes:
- Evaluation Technical Assistance: Rounds 1, 2, and 3 sites and their independent evaluators will receive assistance in designing and conducting P3 site evaluations.
- Systems and Process Analysis: Analysis of P3 implementation and processes and national program and service delivery systems changes that have resulted from P3. Analysis will be based on site visits, interviews, and partner surveys with Round 1 sites, and interviews with Federal staff.
- Collection of Participant Data: Cross-site synthesis of site data and reports to describe Round 1 pilots and participant characteristics and outcomes, based on data collected by sites.
- Synthesis of Local Evaluation Reports: Cross-site aggregation and (to the extent possible) comparison of Round 1 sites based on sites’ final evaluation reports and data.
The final report will be available in 2020, with interim findings available in 2019. Due to the absence of dedicated resources, the national evaluation activity for Rounds 2 and 3 may have to be limited to technical assistance to support evaluations conducted by individual sites. The national evaluation is being managed by the Department of Labor, and conducted under contract by Mathematica Policy Research.
Conclusion: Questions and Considerations
The P3 pilot sites and national evaluations for Round 1 will not have interim findings until 2019 and final findings until 2020. Given this, it is premature to provide recommendations regarding scaling P3 or applying best practices more broadly. Nevertheless, the experiences to date of Federal agencies and feedback from pilot sites and applicants suggest a number of questions and considerations that may warrant attention. As P3 evolves, new lessons may emerge, and the national evaluation will provide a more rigorous assessment of P3 implementation and the success of individual pilots. The below questions and considerations highlight preliminary lessons of P3 that may inform Congress when considering future P3 authorization.
- How can application and appropriations cycles best align? The year-to-year authority to conduct P3 makes it difficult for potential applicants to have sufficient time to plan and prepare applications while still allowing adequate time for Federal agency review. The timing under P3 thus far can present difficulties, as applicants need to reserve funding early in the fiscal year to support a potential pilot but do not know whether they have been selected as a pilot until late in the fiscal year, or even the following fiscal year. Multi-year authority, or an authority that is not dependent on the appropriations cycles, would facilitate applications and allow agencies to better plan and implement the program, including improvements to the application process requested by applicants. Congress may want to consider including such authority in any future P3 authorization.
- How should the application process be structured? A staged application process may better serve applicants, allowing them to submit shorter, preliminary proposals for consideration and feedback, including technical assistance discussed below, and allowing discussion of whether Federal waivers are required for potential projects. A staged process could reduce the burden of applying for P3. Such an application process would not necessarily require a modification of statutory language authorizing P3. However, because such a process would require a longer application period, the staged process would be most feasible if P3 were authorized on a multi-year basis or through an authority that is not dependent on the appropriations cycles.
- What resources and capacity are needed to administer P3? P3 lacks dedicated resources, which has limited funding available to support grantee start-up, technical assistance, and evaluation. Adequate federal staffing to administer P3 remains a challenge, given the complexity of the initiative. A recent Government Accountability Office (GAO) report recommended that agencies determine the resources needed to administer P3 in order to better plan for completing the current rounds of pilots and identifying the capacity and resource implications of scaling P3.
- At what stage is technical assistance most valuable? The pilot selection process has demonstrated that many applicants would benefit from Federal engagement and technical assistance during demonstration planning and prior to application, regarding concept and service model, evaluation design, and identifying current barriers and the need for potential waivers. Technical assistance has been a cornerstone of pilot progress but has occurred during implementation only. It appears that greater technical assistance upfront could reduce the need for ongoing technical assistance after pilot selection, and potentially result in greater efficiency and stronger pilots. Shifting technical assistance in this manner would not necessarily require a modification of the statutory language authorizing P3, but the extent to which Federal agencies could provide greater upfront technical assistance would be limited by capacity and available resources.
- How might agencies support applicants not needing P3 waivers in pursuing program improvements? Some P3 applicants with promising proposals to improve outcomes have been denied because the proposed projects do not require Federal waivers. Some of these projects may benefit from technical assistance, peer learning, and evaluation. Given this, it may make sense to consider how projects that do not require Federal waivers could contribute to P3’s broader agenda for improving outcomes. The extent to which Federal agencies could do this would be limited by capacity and available resources, but would not necessarily require a modification of the statutory language authorizing P3.
1 Available at: http://youth.gov/docs/P3_Consultation_Paper_508.pdf (PDF, 19 pages).