Interventions that explicitly identified program implementation problems or suggested possible issues showed smaller reductions in participants’ externalizing behavior than those that did not mention implementation at all or indicated no problems.

Examples of implementation problems reported by these interventions included low participant attendance, lack of fidelity to a manual or guideline, incomplete or inconsistent service delivery, and lack of capacity to monitor and support implementation.

The best way to know if you have an implementation problem, what it is, what is causing it, and how to address it, is to have a systematic process for monitoring implementation. Without a system in place, program managers must rely on what they hear or happen to observe, which may not fully represent the problem or problems or tell them what is causing it.

Creating a process to identify implementation problems is critical to ensuring strong implementation. Moreover, monitoring implementation is one part of an overall Continuous Quality Improvement (CQI) approach that can ensure what you learn from monitoring is incorporated into staff training and professional development, and program operations and program design.

Assess Feasibility

Current systems. If you already have a system to monitor implementation, examine whether it provides you with the information you need to identify problems. For example, are you able to:
- Track program dosage with attendance records or sign-in sheets?
- Monitor adherence to lesson plans or program guidance with fidelity checklists?
- Assess service delivery quality with observations or participant surveys?
- Identify patterns and areas for improvement? Can you sort or filter the data by staff member, day of the week, or participant characteristics to help pinpoint possible causes?
Fidelity thresholds. Do you have agreed upon thresholds for what constitutes an acceptable level of implementation? For example, what parts of a program are essential to complete? How often do staff need to offer a particular activity to be considered full implementation?
Organizational capacity. If you do not have a way to systematically monitor implementation, do you have organizational capacity to introduce something new? Is your organization’s leadership supportive and committed to integrating new processes into the workflow?

Take Action

Consider any of the following courses of action depending on the results of your feasibility assessment:

Consider adding elements to your implementation monitoring system that may be helpful, such as benchmarks for acceptable levels of implementation based upon past program performance or minimum program requirements. Otherwise, wait until your program has accumulated enough data to set realistic benchmarks.
Use Continuous Quality Improvement or feedback loops to incorporate what you learn into staff training and program operations decisions.
Ensure staff have adequate training and ongoing support so they understand the importance of adhering to program guidelines, and have tools to enhance the quality of services they deliver (also see Recommendation #1).
Monitor data over time to see if there is improvement in the areas you are targeting.
Create a Learning Collaborative or Community of Practice to encourage peer learning, share promising practices, learn about staff perceptions of improvement, and troubleshoot implementation challenges.
Consider working with an external evaluator, or conduct your own internal process evaluation to learn if your program is being implemented as planned and leading to desired results.

Breadcrumb

Monitor Implementation and Address Challenges

Outcome

Externalizing Behavior

Intervention Family

Effective Implementation