Assessing Implementation of Multiple Evidence-Based Practices/Evidence-Informed Practices

I’m going to try to come at this from a different perspective and several ways. From what David charged me to do ahead of time, I’ve had concerns about how I was going to do this.  Some of you know a little bit about the work of my colleagues and myself.  Others are gonna end up saying, “What the heck was he talking about?”  Because it’s going to be like taking a drink out of a fire hose in a couple of ways, but let’s see where we end up.

And this didn’t go anywhere.  Oh, here we go.

All right, it seems like we got several questions that I’m supposed to be addressing around, how do we evaluate the implementation of multiple evidenced-based programs influence-wise?  So, one of the questions is going to be, how can we best assess readiness for implementation?  How can we best assess impact of each, and how can we assess the combined effects?

My agenda is to come at this from being a change process researcher.  And one of the things that I’ve run into right off is we had a problem with vocabulary.  The health professionals talk about interventions.  Those of us in change process research distinguish between the change – aka the innovation – and we used the word “interventions” as to be those actions that you take to help get the innovation implemented.

So, you’re gonna see in my slides I’m schizophrenic.  Sometimes I talk about innovation; sometimes I talk about intervention.  But that’s one of the messages right off.  Technical assistance is an intervention to me.  The innovation would be cessation of smoking or something like that.  So, hopefully that’ll help.

The other thing that’s gonna happen in this is, I’m going to talk about it from a Concerns-Based Perspective, which most of you won’t know about.  But if you want to see what some academic will come up with, there it is.  You’ve got boxes, and arrows, and vocabularies, icons, and stuff going everywhere.

What will happen is, we’re really only going to talk about the shaded boxes right now, and I want to use those three constructs to try to talk about how we can do evaluation with a focus on implementation.

In our work, we use some metaphors along the way.  So, I’m gonna use one of my favorite metaphors, which begins with, “We have agencies.”  In this case, it happens to be a school, but it could be a business; it could be a community, so keep that in mind in terms of what I’m talking about.  It’s doing whatever it’s doing.  And then, for some reason, we want differences in outcomes and outputs.  We all know that if we keep doing what you’ve been doing, you’ll probably get what you’ve always got, which says there’s got to be some sort of new practices.

In my metaphor, and having lived in Las Vegas for a couple of years, the metaphor works really well if you think about the Grand Canyon, there tends to be a gap between what we’re doing now and what’s expected in the future in terms of if we’re going to change those outputs and outcomes.  We underestimate how sophisticated and complex our innovations are.  So, we implicitly say, “All you have to do is jump over.”  I don’t think you can do that with most of the things that we’re about.  I think you have to have a bridge called “implementation.”

In our research, we’ve identified three different ways that you can assess how far across the bridge an individual, an organization, and a system is in terms of implementing whatever the new change may be.  So, if we’re gonna do evaluation, it seems like the first step is to be evaluating how far across the bridge our users are.  And then, if we’re gonna do summative, or output, or outcome evaluation, we better take into account whether they are that far across the bridge or not.

So, my illustration for this, to continue this metaphor, would be to suggest that we can use these three ideas, one being Stages of Concern, which deals with the feelings and perceptions that are out there with regard to a change.  And another would be Levels of Use, which is an operationally defined construct that ranges from nonuse, to being a novice, to being an expert with whatever the change may be. And then a third is Innovation Specific.  We discovered a number of decades ago that implementers change and adapt the innovation, and sometimes they mutate it.  But they will claim that they’re using it.  So, we’ve ended up talking about different configurations of an innovation.

Another thing about our work has been – I ended up one time noodling around with different strategies for change, different kinds of interventions and came up with names for them like “Pennsylvania Contingent,” and “Renter Martyr,” and talked about how these would work. And the one that I think applies today, in terms of what we’re about here, is – well, I should point these things out along the way – these constructs can be applied to individuals and groups, and each has got an extensive history of research with it.

Well my metaphor, we’re talking about a multiple adoption design.  So, I tried for a little humor there, a little bit along the way.

[Laughter]

So, all right, now if we use the bridge metaphor, what’s going on is we’ve got multiple bridges. And as David said in his introduction, and as Ajay did, we’ve got all of these things interacting with each other.  So, from an implementation and evaluation point of view, as will be talked about tomorrow by Abe, we’ve got issues of capacity, but we’ve also got the different bridges going on, and we’ve got all these different interactions that have taken place, and all these different outcomes that are going on.  So, part of what happens in a multiple adoption design is we’ve got to run what happens with each initiative individually, but we’ve also got to think a lot more about the interactions between them.

All right, not knowing how my time is going, but moving along.  To illustrate this idea real quickly, using those three evidence-based constructs, I’ve attempted to put together three different innovations that might be going on in the same system.  And if we were to do – these timelines are unrealistic, probably,- but I’ve tried to run an October set of assessments, and then an April set of assessments, just as a for example.

So, for innovation one, when we start off, we’ve got 60 percent of the participants with self-concerns.  Now, there have been some sort of interventions and successful process going on so that by April, that only 30 percent have self-concerns, and 60 percent now have task concerns.  That would be the typical kind of progression I would hope to see in an implementation effort.

The next set, innovation two, as you can now begin to see in the pattern, the only change we really get is more self-concerns.

The third group, dealing with innovation three, is  a more typical kind of progression as well, when you go from not having any – you start with some people are at task concerns, but then you get more with task concerns and fewer with self concerns.  And you end up with some with impact concerns.  And by the way, impact concerns is the goal.  This is where the new approach is implemented and the concerns are around not me; they’re not on how much time it’s taking; but it’s on, how is this affecting our clients, and what can I do to have even more impact on my clients?  Which should be the goal of a change effort.

If you look at the same thing with levels of use, this is dealing with not your feelings and perceptions, but your behaviors. And the way we would start with people, there are three different ways you can be a nonuser in terms of your behavior, including people at the level of zero, they’re not doing anything.  You send out the announcements; they don’t read it.  You require them come to a meeting; they don’t attend.  Or in my case, they’ll end up in the back of the room, knitting or something, if they’re required to be there.  So, behavior wise, they’re not engaged.

What we find, when I first implement it, they’re going to be what we call mechanical use, level three.  There’s a disjointedness.  They’re like me with this PowerPoint, I’m pushing the wrong key.  Deciding at night about what you’ve got to do tomorrow.  You can’t plan ahead very far.  We find that 60 to 70, if not more, percent of first-time implementers will be at Mechanical Use.  Not the time to do a summative evaluation.

So, if you look at our same three cases, in case one, the innovation – the first innovation, we’re starting off in October with nearly all of our target group is nonusers.  And then we watch the progression, by April, we’ve got 65 percent that are now at Mechanical Use, and 10 percent are at higher levels of use.  I’m sorry, that was – yeah, that were on Levels of Use, okay. 

For innovation two, we’ve got nothing going on. Now, in both cases, you don’t need to know what the innovation is.  It’s just sitting there, and we’ve got to construct to talk about it across programs, across initiatives, using the same variables that are evidence-based.

The third innovation was moving along in a nice developmental progression that I would expect, only this is a naive approach.  It would usually take three or four years to have them get that far along.

The next construct, Innovation Configurations, takes a little more explanation.  This is the one where we’re talking about the innovation.  So, this measurement is going to be innovation specific.  And so, we build what we call an “Innovation Configuration Map.”  Now, for you all that have “rubric-itis,” at first you’re gonna see this as a rubric, which in many ways it is.  But for most of you that are doing rubrics, you’ll know that the most highly proficient description will have lots of words.  And as you get less and less proficient, the number of words gets less. But in this particular example here, we want a component out of a configuration map, the number of words stay roughly the same.  This is because of the structure and building of a Configuration Map, where we, first of all, will have the ideal practice at the left, and call it “A.”  And than a string of variations going across that don’t just taper off on the ideal, but they’re doing something else.  So, as you move further to the right, we’re building up descriptions of what they’re doing instead.

So, if you’re not too drowned in the fire hose yet, let me just show you what would happen.  And the bridge is reversed now, unfortunately.  Trust me, there are reasons for this.  But in innovation one, you could see we’ve got heavily loaded to the right.  They’re doing a traditional practice.  Innovation two, all we’ve got is traditional practice.  We’ve got no hint of a new way.  And innovation three, we’ve got developmental progression going.

Our colleague from – timekeeper from Rhode Island has given me the five-minute minus sign.  So, let’s go on to a couple of questions.  If we go back to the basic questions, at this point, that I asked at the beginning, right now, we know a lot about making that adoption decision.  We know a lot about how to facilitate implementing single innovations and programs.  We don’t as much as we need to about sustaining use.  Sustaining what happens after people get across the bridge, I don’t think we know a lot about it at all.  And we know even less about facilitating and evaluating multiple initiatives.

A couple of methodological suggestions for us to be thinking about, I think we have to do implementation assessment of each intervention that’s being done, or innovation, depending on your words.  And a part of that we need to assess implementation for sure.  But the whole set has to be examined differently than looking at each one separately.  I think we should be addressing the change process within it, and actually, I could plug some constructs in that I think we ought to use.

Now for the dangerous part.  To do this presentation, to be a part of this session, suggested – and Paul said it as well, at the beginning – that we ought to challenge our thinking a little bit.  So, there’s my escape route in case I get too far away from where we want to be.  Here’s some things to think about conceptually.

Assessing each implementation bridge provides useful information that can be used by leaders, implementers, researchers, and policy.  If we use some standard constructs, like stages of concern and levels of use, everyone can work off the same set of data.  And if you’ve got – if you’re a leader, and you’ve got people in Mechanical Use, you know your interventions have to be different than if you have people at a higher level of use.

What else did I put up here?  We need to understand more about what is entailed in sustaining?  Yes.  There are important differences between a problem and a dilemma.  And when I’m working with graduate students in leadership, one of the things I try to have them understand is this difference.  A problem, by definition, has a solution.  A dilemma, by definition, does not.

Now, I know in the health sciences, we have to deal more with trying to find specific solutions.  So, with that caveat, let me suggest that the current evidence-based practices and initiatives are designed to address problems.  However, there seems to be a preoccupation with achieving fidelity of implementation of these.  But I think we need to think more about the capacity that results from having these different efforts implemented, rather than assuming we’ve got to have the first time an ongoing fidelity for whatever reason the innovation was implemented.

The evaluation of outcomes needs to be broad spectrum.  The intended outputs of each program need to be thought about, but the consequences intended and unintended needs to be thought about.  Growth in capacity, I think, becomes more important, we have to start thinking about that just the output from the particular innovation.  We need to think – yep, okay.

The last point would be leadership is critical in all of this.  Give me five minutes with your leader, and I’ll tell you how good your change process will go.  And for any of you that were wondering, I did one last thing just for fun, and that was – there is a book, but if you want to work on it in China – Chinese, or in Korean, it’s also published there.  So, thank you.

[Applause]