Investing in What Works Project Forum: Welcoming Remarks
Watch and listen as Ajay Chaudry, Deputy Assistant Secretary for Human Services Policy in the Office of the Assistant Secretary for Planning and Evaluation at the U.S. Department of Health and Human Services, gives welcoming remarks and discusses the purpose of the Forum and overarching themes.
Explore videos from all the experts.
Thanks, Sarah. As Sarah said, I oversee the Office of Human Services Policy at ASPE that is sponsoring this project, and the reason we’re here today, so I do want to welcome everybody. And thank you for braving the June drizzle and the D.C. traffic to get here.
As Sarah said, we have important goals for the next two days, mainly to explore the most relevant issues out there for how we expand the use of evidence-based practices in the real world, and how we apply these to working with vulnerable children and youth, and to identify ways that the federal agency serving children and youth can support evidence-based practices and evidence informed interventions.
We’ve been very excited about this event and especially about the enthusiastic response from across HHS and several departments in the federal government that serve children and youth that are well represented here. We’re also impressed by the commitment shown here by our partners in state universities, non-profit organizations, research institutions, and foundations.
Among the people registered for this event are over 50 federal staff and more than 30 partners in the state, local, private and philanthropic sectors. There are federal staff from five different federal agencies and seven different HHS agencies. This indicates to us that we’re focusing on issues that resonate across programs areas. A good range of presenters and outside experts will be lending their insights to the work, and a lot of this work will be occurring in small groups. Most of the learning that will occur in the next day and a half is learning that you will do from each other.
So a little bit about ASPE and the genesis of our work in this area. At ASPE, we have regular opportunities, one of the things that we do is we work across HHS, and we work across federal agencies, and when we talk to the rich array of staff in different agencies, we hear about some of the same concerns about evaluation evidence, and how to use it, and how to have it inform federal work.
And many of the challenges that we hear are common and sort of extend across the federal government. These include differences among agencies in thinking about what it actually means for a program to be considered evidence-based. There are common concerns at the federal level about replication and scale-up, and the struggles that are out there for programs to adopt evidence-based models.
Concerns about how to be as evidence-based as possible when you’re applying a model to a population or to a community context where the model hasn’t been applied before. Struggles that grantees have about implementing evidence practices described in their grant application, when the actual program model is maybe new to them, but it’s what the funding opportunity allows for. And tales of the trial and error process that’s involved in helping support grantees once they’ve won an application adequately through the critical process of implementation. And then just shared concerns about how messy this process is both to do and to understand through the research and evaluation that follows implementation, -because grantees rarely are implementing a single evidence-based model in stable environments.
Staff and agencies across government have been facing these similar issues and having similar conversations. ASPE started this work several years ago - conceived of, as the title says, an “Investing in What Works agenda” through which we could try and move these discussions forward.
I want to especially recognize the spearheading role that Martha Moorehouse has played in developing this agenda. Martha is currently working on assignment at the Corporation for National and Community Service, where she’s actually helping that office apply this thinking in their work, and she’s with us today and tomorrow both in her new role at CNCS, as senior advisor to the Social Innovation Fund - which as I said, working on these issues - and from her continuing work over many years in ASPE to bring not only attention but serious consideration to these issues to our programs and evaluation activities.
Martha’s probably not a stranger to you. I’m probably a newer face to you than Martha is, but do Martha, do you want to just stand up, so everybody can see you and recognize for your role?
We truly wouldn’t be here today if it wasn’t for Martha, so I also wanted to acknowledge our former assistant secretary, Sherry Glied, who in partnership with Martha provided enthusiastic support for this work and encouraged expansion in recent years. This builds on work that preceded my arrival at ASPE, and so I wanted to recognize that.
Over the last year, ASPE has sponsored several meetings as Sarah indicated, and there are already materials from Phase I of this effort on the ASPE website. In July of last year, HHS held an all-day evaluation day to allow evaluators from across the different agencies within HHS to come together and discuss common issues, such as the exciting topics of evaluation methodology, establishing an evidence base, the analysis of costs and benefits in evaluation, best practices for addressing HHS and OMB clearance processes, and integrating evaluation findings into budget, policy and decision making.
I was being a little coy and sort of calling it exciting, but it is the nuts and bolts of what our work - for those of us who work in the evaluation offices of the federal government -do. I mean, our jobs are to identify what is worthy of addressing the goals of our departments, what are worthy investments of federal dollars, and how to make that work in what is a very large and multi-layered federal process, all of which these issues address. So I actually meant it.
So plans are underway to continue this work, and we have heard from other departments that are interested in planning similar events, so if you need a guide book for establishing evaluation day at your department, we are the people to get in touch with.
Just another example of how ASPE has been engaged in this work on the ground on particular issues, so ASPE has been working with ACF, the Administration on Children and Families, and other partners to develop an ongoing systematic review of research evidence on teen pregnancy prevention programs. Some of our partners from the Office of Adolescent Health and ACF are here in the room. Several agencies, including ACF, HRSA, the Health Resources and Services Administration, and CDC joined forces to conduct a similar review of effective home visitation models. This is known by the cool acronym HomeVEE, Home Visiting Evidence of Effectiveness, and I see that the national Project Director for that initiative is here in the room too. And AHRQ, the Agency for Healthcare Research and Quality, - see I’ve been in the federal government for a year and half, and I’m starting to learn the alphabet all anew - did a review of models for treating trauma in children who have been maltreated.
Other topics are ripe for similar reviews, and in the coming years, we expect topics on which such systematic reviews of research evidence to expand. We’re just now talking about well, what are the key next one, three, five areas of our work in human services policy that could be ripe for this work. Our colleagues on the health side have also been engaged in this work for several years and are lining up more and more issues in this area.
What you have before you in the binders is a very fine agenda and materials for engaging in this work. As I understand it, this afternoon starts with a topic we’ve called “Understanding the complex context of implementing and sustaining evidence-based practices in the real world.” There’s a lot of emphasis on the real world here. By this we mean that few efforts funded by our program are ever implemented in a vacuum, right?
We don’t typically fund a single evidence-based program to be implemented in a community that has nothing else related going on. That’s going to affect the effectiveness of that model in that neighborhood. These are complex communities. I’ve been involved in research efforts where the initiative that I’m implementing is also being simultaneously, clients are being offered another initiative in that same neighborhood that often has the same objective, and it’s hard to sort of decipher what’s evidence from our program versus a broader community investment that’s occurring elsewhere. Or, on the flipside, challenges that may be shaping the outcomes.
So I’ve spoken a bit about ASPE’s activities, but I want to recognize that many agencies are active in this area and have terrific work going on. CDC recently released a web-based tool focused on how communities can go about selecting evidence-based programs that match their goals and circumstances. The Office of Adolescent Health and ACF have been working with grantees, as I mentioned, that are doing teen pregnancy prevention, and there’s work on our websites on that.
OMB has been working with a number of agencies around Pay for Success models, and the potential applicability of Social Impact Bonds. The Corporation for National and Community Service’s Social Innovation Fund, is expanding community-based solutions that have evidence of results on priority topics including youth development. And the Departments of Education and Justice are doing related work on children and youth.
ASPE comes at this issue from a federal perspective, as do many of the staff here today, but we need also to consider the state, local, and programmatic participants’ perspective in this endeavor. This really hard work for everyone involved, and you have to get the incentives and requirements right at the federal level, but you also need to build in the supports that allow local programs to work in their communities to make this actually useful in the real world.
And really, the federal government doesn’t operate in the real word. It tries to acknowledge the real world, but we’re usually a few layers away, right? We really want to know for evidence-based practices what happens at the people level, at the front line level what people used to call the interaction between the service delivery of a professional street level bureaucrat - whether that’s a cop or a teacher or a case worker - and the person receiving it.
So we can talk about research a lot, but it only means something if it happens locally, and there are significant challenges to that. We’ve already discussed that in many fields of evidence-based research on effectiveness is scarce. Work on implementation issues is uneven and not always complementary to work on evaluation or program effectiveness.
We also need to acknowledge that really moving this field in the real world means going beyond demonstration efforts for which you can find very high levels of effectiveness and provide incentives to try to do this work at scale within regular funding streams. How do you do this within a regular ongoing funding stream as opposed to a very strongly nurtured model program?
Finally, if we’re serious about this work, we need to build capacity in places that haven’t been ready or had the capacity to do this to date. In any given program area, there will be a few early adopter communities that, as I said, might be incubating a promising model within ideal or very thoughtful circumstances. But how do you do this in the places that maybe in some ways need it the most, that don’t have great grant writers supporting the bringing of new projects to their sites?
So I think addressing these challenges, and what’s needed to reach most of the children and youth living in adverse circumstances, is really a lot of what the next two days is about. So I’m looking forward to the discussion over the next two days. Unfortunately, I won’t be able to be here for the entire meeting, but we have lots of staff in the room, and AIR is leading this work on our behalf. So I look forward to hearing the parts that I miss, and the next steps as we move forward with that. Again, let me welcome you and thank all of you for coming, and I’ll pass the discussion over to Paul Florin, who’s going to be chairing the first panel.
[End of Audio]