Implementation Science, Supports, and Strategies: What Works for What Works

I can’t even read my grocery list in 15 minutes, so I don’t know what this is going to be like.  But I am honored and privileged to be here to talk about this very complex issue of, how do we build, sustain, and scale evidence-based practices to get socially significant results?

And I think a key word for today – and I’m privileged to be here with my panelists; I’m glad Arthur is following me – is in typical service settings.  So, let’s focus a little bit on that, and I think we can learn from a compare-and-contrast perspective.  And so, I’m going to take you through a little metaphor to start out, very briefly.

I think we can perhaps learn something from the Olympics, our successful participation in the London Olympics in 2012, and about a country, that we all know well, whose athletes did not even get to compete.  And that country was Status Quo.

And the Status Quodidians were very surprised when their athletes didn’t actually make it to the Olympic games because they had legislated, mandated, and diligently prepared their athletes.  As a matter of fact, when the rulers heard about the upcoming Olympics, they wanted their Status Quodidians to win big and get great outcomes for their country.

They’re proud of their country, and they were all up for re-election and needed to show that they could achieve great things in very short order.  So, in 2011, they passed decrees that proclaimed the importance of the Olympics and passed laws and regulations requiring teams to win gold metals and use the latest evidence-based technology and equipment.

To compete in the Olympics, you need teams and athletes.  So, Status Quo had to select who would go to the Olympic trials.  But every legislator wanted his or her constituency represented.  After all, that was fair, and they were an egalitarian country.  As a result, they said, “Anyone could be an Olympic athlete, provided they could pay for their own uniforms, their own training, and their own equipment.”

So, they selected themselves without regard to their basic skills, or regard to their adherence to the Olympic ideals, and showed up for training camp and declared to be Olympic athletes.

Then the training started in earnest.  They read manuals and papers with a lot of data and advice about evidence-based rowing, synchronized swimming, and high jumping.  But there were hands-on activities as well because we know about adult learning principles.  You need to be a good team, so they got everybody together at tables, gave each table a set of gumdrops, and uncooked spaghetti, asked them who could build the highest tower and proclaimed then that they had insight into how to be a good team.

I would like to tell you I experienced that moment with some evidence-based training in Thinking for a Change, which is a juvenile justice approach.  So, we left there having had a competition about how to construct these towers and had insight into how to be a good team.

But they also had other important training.  They had a compliance checklist they needed to go through, and compliance takes a lot of time and energy.  But they dutifully passed tests covering the history of the Olympics, safety on the field, privacy in the locker rooms, and how to say, “Hello,” in 15 languages.

And finally they got their equipment, read their manuals, watched videos one more time.  Then they went to the Olympic trials.  Things did not go so well.  It seems other countries had been investing for a long time in getting ready for the games, and at some expense.

The other countries had an infrastructure for sports that included sporting venues and funding for athletes, training, and coaching, and not everyone got to be an Olympic athlete.  They had selection criteria, and a different kind of training that involved actually practicing the sport, having coaches who were experts in the sport, and in coaching.  And they had a lot of timely and actionable data about how good they were, and the opportunity to review that data, re-practice, get more coaching, and get better over time.

So, sadly, the Status Quodidian athletes returned home disappointed and disgraced.  The legislators blamed the athletes and said now that they had tried competitive sports, that competitive sports just didn’t work for Status Quo because they, after all, were unique and different from other countries.

The citizens blamed the legislators for humiliating them.  And the athletes quit sports in frustration and went to work at a well-known coffee chain, where the training, coaching, and support were much better.

So, what does the plight of Status Quo have to do with Investing in What Works?  Well, first of all, Status Quo invested in processes that were insufficient to create success and not based on best evidence.  So, we know from the literature that these issues alone, these approaches alone, by themselves, are insufficient to get implementation.  And this is from our broad review of 800 implementation articles looking at different domains.

So, what do we get for a Return on Investment when we rely primarily on one of these initiatives?  It looks like we get about 5 to 15 percent return on our investment in terms of people even attempting to do the intervention.

That being said, of course, these are necessary things to do.  They’re just not sufficient to get sustained implementation that’s scalable and have differential effectiveness, depending on the “it” that you’re trying to sustain.

And similarly, like the country of Status Quo, our status quo is tough.  We don’t usually typically use what works.  We are, in this room, in an island of excellence.  This is not how human services in this country runs, and I think we all know that by simply reading the headlines that come to us about what’s happening in our service sectors for a variety of reasons.  So, this isn’t about blame; this is about trying to figure out how to make it work with the constraints that we’ve got.

So, then we have this implementation gap.  Sometimes even if we get started, we don’t get better; we get worse, and we lose fidelity.  The program walks out the door with the people that we’ve trained and coached to do the work.  And we don’t know how to scale up yet.  Scaling up is a new thing for us to really think about if we’re talking about reaching societal benefits, even 60 percent of people achieving it.

So, should we fight the status quo?  It’s not clear to me.  I like Bucky Fuller.  You know, “You never change things by fighting the existing reality.  To change something, build a new model that makes the existing model obsolete.”  I’m not sure that I know what that new model is, but I think we’re trying to figure out some of those things.  And I think we have to go at it both ways.  We have to work within the status quo, try to bend it, and we have to build some new ways of looking at it.

So, in simple terms, I think, Status Quodidians did not have a formula for success.  Matter of fact, I think they had a formula for cynicism and disappointment.  And I worry about how we put these requirements out there for evidence-based approaches, and then the success we do or don’t have, and then the general population’s view of what science can or cannot bring to the work that we’re doing.

So, let’s look at this as a possible formula for success and talk about it just a little bit.  What if we have effective and usable interventions times effective implementations methods times enabling context to get the socially significant results.  And it’s no accident that impact those – the multiplication signs are there, because there are lots of evaluators here.  Any number times zero is zero.

So, even if we have something that is 100 percent effective, if in fact we don’t have a good way to implement it effectively, and the context is not enabling, we start to get a diminution of results over time, which is some of what we see in the translation from effectiveness and effective mistrials to real-world impact. So, we’re going to talk a little bit about some of this, very little. 

So, what?  You’ll notice I’ve got up there effective and usable interventions –

That word “usable” is important.  So, not only do communities need to be ready, but the intervention needs to be street ready? Or how street ready is it?  How service ready is it so that we can build in the additional supports that we need?  Do we know what the independent variables are?  Independent variables are the programs and the practices.  And in research, we’re not even very good at specifying what those are in terms of measuring them and measuring the degree to which they’re correlated with outcomes.

So, what is the “it” that we’re trying implement is often not as clear as we would like.  So, we have some criteria around usable interventions that we’d like to propose to you.  And I lack the time to go through them, except to say that practitioners and agencies don’t use experimental rigor, which is important; they use practices and programs that are presented to them in a way that they can use them in real-world settings.

So, we need the rigor, because it has to be worth it to do this work, and it’s got to be usable.  And they need to know how to interview purveyors and developers to see to what they’re going to bring to the table with them, and what it is the community and the agency or the state is going to have to create for itself.

So, what does this mean for building capacity?  And we’re always interested in building capacity, to do what?  So, a lot of these things need to happen. I’m not going to go through them one by one, but I do think we need to match our expectations, and maybe this is something that our federal folks can help us with, and as for our expectations, the speed with which things should be done, and the outcomes we can expect to achieve, given the level of readiness of the intervention, and I would add the level of readiness of the community.

I am very concerned about evidence-based practices going where they’re most wanted rather than most needed.  Because we can actually increase disparities, as we move through this, as the best able to uptake are not always where things are most needed, and yes, everybody gets better.  There’s a good American Psychologist article about everybody gets better, but the gap increases.  So, we have some challenges to do there, to work on there.

How to do this work.  We’re learning more about – just as there is evidence for interventions, there’s evidence about better ways and less better ways to implement.  And who’s going to do this work?  This is where there’s a change.

I will confess, I’m the daughter of a county extension agent.  So, I come by my love for implementation and a move for science to serve us honestly, by many summers traveling around Southern Colorado with my dad, helping the ranchers implement best practices.  So, there’s an infrastructure to do that work.

How will we do the work?  People can’t benefit from interventions that they don’t receive.  We’re not going to go through this except to say, very quickly, on FFT, fidelity might matter.  This is a case where poorly done – not poorly done – but less fidelity to an evidence-based intervention, the results were worse than treatment as usual.  So, it’s no small think to muck around with evidence-based practices and not do them well and provide the supports that are needed.

I think we’re also moving with technical assistance from letting it happen, and helping it happen, and putting the burden on practitioners, teachers, frontline people to say, “We need to create a TA system that helps make things happen, is proactive, and builds capacity.”  I’ll invite you to go to our active implementation hub and learn about the implementation strategies that have evidence behind them.  So, we need to use data about effective implementation processes.

And it’s backwards, so we won’t – I’ll have to go through this.  I only have, like, four minutes left.

Okay, so what does that mean for what works?  We need to build capacity to create implementation teams.  We can’t rely on individual champions.  Who’s on the team that knows the innovation, knows implementation strategies, and knows how to build improvement cycles?  Engage in stage-based work, the right work for the right stages.  Build an effective and sustainable implementation infrastructure.

Brian Bumbarger and his group is here to talk about what are the functions that they’ve created in Pennsylvania – not the form, because we won’t get our state universities to do that – what are the functions, and how can we embed those functions in existing structures, repurpose them to be more effective?  And how do we collect, report, and use data to improve over time?

The environment’s always changing.  We always have to have the sensors out there.  I am leery, leery, leery of saying, “Good, we’ll use an evidence-based program, and then we won’t have to collect outcome and fidelity data.”  Just fundamentally I will say, in my view, humble view, wrong thinking.  Those are the rudders in the water that tell you whether or not you are going to hit the rocky shoals.  You go out in a boat, you don’t says, “Good, my compass worked; I’ll throw it overboard.”

[Laughter]

You keep using the compass.

So, we’re enabling context.  My friend, Patrick McCarthy, systems trump programs.  That’s the sad truth of the matter.  These innovations, multiple or otherwise, have to live in the real world.  And what happens, sadly, is that the innovations get changed to fit the existing system.  And often, the effectiveness of them gets changed.  What we need is to build a structure in which the effective innovation feeds information up to the policy level, up to the regulatory level so that we know how that needs to change in order to support the innovation.

I think one of the things that you can do at the federal level is to be sure that you’re soliciting – I love Mark’s comments about, “We’re a community here.”  So, how do we solicit feedback from those that we have funded about our grant processes?  What worked for you in this grant process?  What didn’t work for you in this grant process?  What could we have done differently?  Where did we get in your way?  What were the barriers?

What were you – there is actual, some fear out there of working with the feds.  You know, like you can’t – fear of sharing data, fear of being wrong.  So, a lot of our work is just about all data are good data.  You know, if we don’t have good information, if we don’t have information, we can’t get started and get better together.

So, it’s this notion of feedback systems from policy to enable practices, but we need practice to inform policy.  And I will tell you, I thought this was, in my naive days, really a technical problem.  We’d set up some linking communication protocols, the information would get back to where it needs.  No.  You are messing with the powerbase.

First of all, you’re making communications transparent and available to everyone, which means my very special personal relationship with Melissa and with Larke and with Shay don’t matter anymore; other people have equal access to them, and that’s messing with the powerbase.

Secondly, some people feel the functions of bureaucracy – I don’t – are to prevent people at the top from hearing the bad news.  And, if in fact, and it gets there, or it has been created, you are a bad person, when, in fact, people at the top need to be able to hear exactly what’s happening at the face-to-face level so that they better understand.  But that only works if you don’t punish people in the middle.

So, what does that mean for us for building capacity?  If you wish to make an improved product, you must already be engaged in making an inferior one.  And we have to get started and get better.  We need to build people’s capacity to effectively use feedback loops at multiple levels to get better.

So, let’s go out on a limb.  Let’s say that we not only need to pay attention to best evidence and effectiveness, but we need to pay equal attention to how we do the work; who’s available to do the work; what infrastructures are going to house it; the best evidence related to how to change the behavior of well-meaning human services professionals, much more challenging than helping families and children change their behavior.

How do we change our behavior? Again, back to Larke’s comments, what can we do?  If we don’t change first, there’s no hope that others will change.  And how do we create hospitable environments at the local and state level?  I think – I wish there had been some state people here.  I’m so glad that Arthur’s here to talk about – because there are things that you can’t do that states can do, that communities can do, and how do we get that vertical slice to work together to produce socially significant outcomes?

That being said, if you want to have a brain festival, I hope some of you can come to the Global Implementation Conference.  Many of us are having to pay our own way there, but we’re coming anyway.  Come see us; we have an active implementation hub that we’ve just launched.  We’re trying to flip the classroom is a big thing.  We’re trying to say, “Come learn about it online so when we coach you, the knowledge transmission’s done via e-learning, and we coach you.”

So, that is being redone. Done. Thank you.  Close.