Evidence-Based Policy Initiatives...Now What?

David Harris: So, welcome good morning everyone. Welcome to this wonderful space. And, we are wondering whether we're going to get a wonderful show from a storm later. This would be a really good place to see a storm, so if it's going to come, hopefully, it's gonna come while we're in here, not while we're out there. So, I'm really excited to be here this morning and pleased to be here. Unfortunately I won't be able to be here for the whole conference, though I'll be able to be back tomorrow and part of today. So, I first want to thank Martha and her team for putting this together. It's a project that was proposed as we were thinking about research projects and funding and human services policy but this was perfect, really exciting. And so, Martha is the lead in children and youth policy as you heard, but also Diana Tyson and Amy Madigan have been working hard on this., I see Sarah Oberlander here as well, from CYP, don't know if there's any other CYP-ers. But there is a whole team who work on this stuff and really been fantastic in pulling this together, and working with Child Trends to make it happen. So, Martha did an excellent job of detailing how we got here.  As you know of the administration came in 2009, and soon after coming in, Peter Orzag had a post on the White House blog that talked about evidence-based policy. And said we were committed as an administration to putting our money where we thought it could have an impact. And, if there are things that we didn't have any reason to believe would have a positive impact, we weren't going to do them. Instead, we are going to use that money for other things. And so that is part of what set in motion the things that Martha talked about.  So, there's an emphasis on evidence-based policy which is a significant shift as many of you will know from how things have operated and from how many things still do operate, sometimes for good reasons. There's a designation identifying key areas for emphasis on evidence, and then once you identify the key areas, you have to figure out what are the evidence-based programs, which programs have enough evidence and therefore should be eligible for national replication. Now, this wasn't a perfect process, it's not a perfect answer. We could have an entire conference and then some, ... people saying, well should it be this or should it be that in home visiting. And that has happened, and will no doubt continue to happen. But I'm confident from having been involved in some of this, that, although not perfect and never could be, it was done incredibly well by quite skilled folks. And, got an answer that, although not perfect, certainly approximates what would be the perfect answer. So, I think we did a really quite excellent job there and I think it's a signature accomplishment of the administration, and will continue to be thought of as such in the years to come. So, why do we need to spend two days, or a day and a half, talking about this? We already did the hard stuff, right? Well, the answer, of course, is we didn't. We just did the set up. And as I thought about this, it reminded me of an experience I had in graduate school that many of you who have been in grad school may have had. So, I was doing my dissertation on racial preferences in neighborhood satisfaction, neighborhood selection, sort of testing white flight kind of things, and to an what extent, it's not race, it's other factors. And, my advisor said to me, when he read a draft, ‘This is good, but I basically crossed off the last three pages.” ‘Why did you cross the last three pages?' ‘Well, that's the policy implications section, and you just sort of tack that part on. You know the policy implications section takes as much thought and effort and attention as that whole first thirty-plus pages where you figured out the analysis, what factors mattered. And, I thought about that when I thought about what we're doing here.  There's a lot of hard work to figure out what are the key programs, what can be replicated, but that's only at most half the work. The really hard work is saying, OK, now we've got these things. Now, how do we actually implement them so the results are similar or match our expectations, hopefully even exceed our expectations. So, that's what I think we're about here, so we've been about in the administration and some of these to varying timelines, different timelines, but we will be about, especially as the grants are funded, especially as people are out in the field trying to do this, and trying to see why it is we're not getting the results that we thought we should get. So, what I wanna do in my time, just briefly, is to share with you four questions. Some of them are compound questions, I suppose, but four questions really (laughing slightly) cause that's how it works. Four questions, I should have made it three in that case, cause Americans like 3s and 5s. Four was a bad idea. So, basically four things to think about as we go through the next day and a half. So, the first thing is what are the opportunities and challenges for promoting evidence-based programs? As you know, the appeal of evidence-based programs is, as I foreshadowed earlier with the Orzag post, it's that we can take our scarce dollars, and we can use it on programs that are going to have significant positive impacts and implement. We don't have to spend time reinventing the wheel, we can just grab this thing and we can go with it and we'll get positive results. Of course, the extent to which this is true depends on a whole bunch of things and that's what we are here to talk about today. So, the first question is to think what are reasonable expectations for these different programs, what kind of impacts might we expect them to have? Second, we've identified what we think are the best available evidence-based models but now how do we reach our goals of improved program quality and accountability? So, despite these evidence-based programs, and, of course, there were evidence- based programs before Orzag posted in June or July of 2009. And, people have been doing things like this and thinking about things like this, but, in many cases we find a gap between what we had in the evidence-based program and what we have in implementation. Now what's going on? What this has led to, of course, is a rise in what some people call implementation science, which is identifying and testing mechanisms that facilitate or inhibit the replication and scale up of evidence-based programs. So, this field is growing rapidly, but there's still a lot of questions. You're going to hear from folks over the next day and a half about some places where there have been successes and places where there have not. But so, one of the key things we want to figure out is what do we need to do to reach our goals?

Third, so, what's actually needed to support successful implementation of evidence-based programs? There's a lot of factors beyond the strength and evidence of the evidence-based practices and the registries and so forth that have popped up. Just to name several: availability of training and TA. So, you say, ‘Here it is, great, Phoenix, you've got the grant, go with it.' Well, someone needs to help them figure out how they actually implement this thing as opposed to just reading up a website saying, ‘I think it was this, I think it was that.' So, to what extent do you have this? What's the capacity? I've just come back from a major American city and one of the things I heard, again and again, was we're great at getting grants, but we have a really hard time carrying these things out because we don't have the capacity to implement them. We've got the capacity to get the grant, which is relatively easy, it's the implementing where it's a huge challenge. And, that can be a challenge in places. We want to think about the community. To what extent is the community ready to do something like this? To what extent are people ready to be told that this program your kids' been in the last five years or kids like yours have been in, we're not doing that anymore because there's no evidence. Instead we're doing this thing over here, that's been shown, among kids who aren't your kids and people who don't live in your community by people you don't know, but, it works better than this thing that you think works in your community. So, to what extent is the community ready for that? And a whole bunch of other factors, costs and so forth. But, the big thing, I think, we've got to think about is the fit between model and community needs. So, I like to think about this, as um, this one factor and its doppelganger. And, which one's the doppelganger depends which day I guess it is, but, fidelity and adaptation. All right? They're siblings, all right? This tremendous tension between these things. Um, the one thing I think we've got to acknowledge, is pure fidelity is impossible. You can't have pure fidelity, all right, in the extreme sense, because this program was implemented in a certain place. But, not just that, at a certain point in time, with certain economic conditions, with certain political environment, and at certain temperature, right? Cause weather matters for various outcomes. Right? there were certain things happening at that point in time, and to think you are actually going to replicate it exactly is just folly. So, you are never going to have pure fidelity. Right? So, the question is how much adaptation is too much? And so, let's take two examples. So, in one example, you have a program for healthy marriage say. This is just hypothetical, healthy marriage program, evidence-based, and there are some evidence-based healthy marriage programs. But a healthy marriage, evidence-based program. It's created, evaluated, in a community, moderate income folks, English speaking. And, someone implements it and says, ‘Well, I don't really have the budget so I'm gonna cut the number of sessions by ninety percent.' K? That's one example, right? That's probably not a good adaptation, right? We don't have any [unintelligible], but, that's probably too much, right? But, then take the other extreme [unintelligible]. Now, let's say you have the exact same program, that program has materials, that program was created, it was tested, evaluated in this English-speaking, moderate income community. And now you're implementing it in a community along the Rio Grande, north of the Rio Grande, where folks don't speak English. And you give them the exact same materials, right? That probably doesn't work, right? That probably is where you need some kind of adaptation. So, I think those are sort of extremes. Giguring out where in there, is the test. And, also trying to figure out, and we've been involved in these conversations in the administration, who decides whether the adaptation (is) significant and on what basis do they make that decision is really tricky. Is it the person who generated the program? Is it the person who's implementing the program? Is it somebody else, and how do they know? How can they tell? And so, one of the other things in here that I'll leave you with, and, this point is something I like to think about, when it comes to evidence-based policies, I like to think, evidence anywhere does not equal evidence everywhere. So, evidence anywhere is not evidence everywhere. So, at some point, if you tested it on certain populations, certain conditions, you may be no better off implementing that in somewhere else then you are implementing some other program that seems good, based on theory but has no evaluation of evidence behind it. So, you gotta figure out how much adaptation is too much, to the point where you think A) it might not work, and B) it's such a different program, you're not implementing an evidence-based program. So, those are some of the kinds of things we'll think about. And, then lastly, I guess I leave you with, as we're out doing this, as we're out implementing these programs, as we're worrying about the twins of adaptation and fidelity, and how to find the right spot in between them, we're, of course, going to find some other things. We're going to find populations, as we already hinted at, for which this doesn't seem quite right. We're gonna find other problems that arise, where you say, well, we don't really have an evidence-based program for this. And, so one of the things we've gotta think about, that certainly comes up in the implementation phase as sort of a tangent off implementing the evidence-based policies, is what do you do when you don't have evidence-based policy? You're in the field, and you say, well, we don't have it. What do you do?  And there's, of course, a number of things you could do, one thing is to say well we're just not gonna worry about that problem, focus on evidence-based policy, we don't have evidence. Sorry folks, we're not dealing with this community, we're not dealing with this problem, cause we're about evidence-based policy.  Of course, the other alternative, another alternative, is to think about theory-based, evidence-informed policy. Right? And, you see this, certainly, in things we're doing, where we've got funds that say, there's evidence-based policy, but then there's also a smaller amount of funds in some of these, where you say, you can use that to sort of test some things that might work. And, then you can, you have to evaluate them, and you want to evaluate them. But, then you might move on even if you don't have evidence in the beginning.  So, I think that's the last sort of thing I leave you with is to think about throughout the day and a half, and I know people are already talking about this, is, what do we do when we don't have evidence-based policy? So, those are just some of the things I'll be thinking about over the next day and a half as a listener. I'm extremely excited to hear from you and learn about this and start our journey, or continue our journey, this early part of the journey, but that's what I had to say. Let me now turn it over to Karen Walker of Child Trends. Karen has been the Project Director on this, and she'll help us understand the agenda for the day. Thanks.