Return On Investment for 3Ts: Tools, Training, & Technical Assistance

Let me ask you a question. You all write RFAs [Request for Applications] and RFCs [Request for Comments] and that kind of thing?  No?  Anybody?  Alright, how many have included cost, the word cost, in that?  Oh, I think you’re ready for my presentation.  How many have included the word cost-effectiveness?  Fewer hands and they’re a little shorter.  Cost benefit?  Ooh, one in back.  Three.  Okay, and cost-utility?  Return on investment, anybody mention that?  Okay, that’s interesting.  New terminology.  So the king of evaluation.  ROI – I can’t ever pronounce that right; return on investment – asks several questions and there’s reason why I think return on investment hasn’t yet appeared in a lot of these requests for funding and that is, it asks, is your intervention effective?  Does it work?  It also then usually asks, why?  And it also asks, how much does it cost?  And then it actually should ask, why?  And then, – is it worth it?  Are you now already figuring there’s no way I’m going to respond to this?  And then how does it compare to others in a competitive ecology, and why? 

Okay, now no one’s going to ever include return on investment in your proposals.  Maybe, but something like it has been there already in this area and in some other areas: substance abuse treatment and prevention, some of you are from there but not as many, it’s been there for a long time.  And in HIV prevention, it’s been there and has gone very far, very fast.  So it’s surprising where we’re at.  What I hope to do is offer some models or theories on which we can act.  As Kurt Lewin said, “No theory without action, no action without theory.”  So I’m going to try to offer a few models here.  The first model is the one that most people think of.  We have finally gotten to the point of looking at our outcomes.  Now, if you put a dollar sign or a pound sign or a yen sign or whatever you’re putting on there, then it looks like return on investment to some people and I think that’s the first approximation.  This is beyond contemplation but it’s not really return on investment, in terms of readiness to do return on investment analysis.  Instead, looking at outcomes such as cost to society of different problems and how much we’ve reduced those costs, that’s just part of the picture.  Some folks will go right to the second sort of model, resources, and they look at the resources in for the different activities and that’s all.  And that’s again sort of precontemplation, but it’s not really there.  To look at relationships between the value of the resources invested in the activity, in the program, and the outcomes of that program: monetary and not so monetary.  Not even monetizable.  That’s another stage but that still doesn’t really get at what’s going in the program.  That still doesn’t get at what is it in the practice that’s making a difference, so my way to save the world is to do a full model that looks at resources in, activities conducted, the processes that actually go on inside the client, which is either an organization or a family or a child, and then the actual outcomes: short, proximal, and long-distal term.  So that’s the overall model there. 

Now I think, from what I’ve heard so far, we’re talking a lot about outcomes.  We haven’t talked a lot about resources and, if we talk about resources, we talk about costs.  I’m here to tell you today that costs aren’t resources or at least costs shouldn’t matter.  What should matter is the resources.  Money’s just a way of getting resources: time, energy, expertise, facilities, all those manuals, all the equipment and materials.  So listing those ingredients I think is a crucial thing to do here and, as in my experience, is far more useful than just giving a dollar value.  We could give a dollar value for each evidence-based practice.  Suppose you got a nice list of those.  I don’t think there are many lists of evidence-based practice costs, are there?  Anybody?  So the evidence we’re concerned about is outcomes?  Where’s the cost figure into our evidence-based practices?  Do you sense that there’s a whole set of variables missing and we need to collect data on them? I think we going to look at that and not just compare it to those outcomes. 

Well, I want to move on here.  There’s a lot of different models.  There’s some nomenclature and this could well be the last slide in the presentation because we could spend a lot of time here.  This is fairly common nomenclature, it’s been out there for a long time and any kind of program sponsored by an entity that’s not private enterprise but tries to bring private enterprise like incentives and contingencies to bear.  We’ve got evidence-based practices, been hearing about that, and that’s such a great evolution.  When I started in the mid-seventies working in a program very much like Karen and Dean’s Achievement Place, only it was called Learning House in Palo-Alto, where you could afford to live back then, we had just begun looking at evidence-based practices and I started – I heard from a professor not in my area, not in psychology, but in education, and they said, “Plastics, Brian.  Plastics.  I’m sorry, cost.”  Kind of like the move The Graduate.  I’m dating myself.  Will be re-released.  And that really opened up a whole new set of variables there that I hope you’ll join me someday in looking at, with that further.  Cost studies and cost-analysis is what I get asked to do a whole lot and there’s some great work in this area.  The sort of thing you see done in the Children’s Bureau is going way beyond looking at cost of, the manual that’s being developed there and it’s vital, shows us costs of different activities.  That’s a real step forward.  Combining that, and it’s all set up to do this, combining it with information on outcomes goes beyond just the cost study or, how much does it cost in terms of dollars, to looking at how much does it cost in terms of usually dollars and then, what do we get out of it?  Cost-effectiveness analysis.  Hey, I got three letter acronyms, too.  CEA is the big one there.  And in cost effectiveness analysis, the outcome is not monetary so most people in this field, in most fields, like cost-effectiveness analyses because we don’t have to try to monetize outcomes.  We don’t have to value human life or even improve human life that way.  But, very often, what makes sense to decision makers, especially in a longer-term, is looking at a cost-benefit analysis where the value of the resources invested is measured in the same unit as the value of the outcomes produced.  These don’t have to be dollars or even money.  I’ve done ratings of, was that a really good technique in terms of both how much did it require of your energy and your capacity personally, and then how much did you get out of it, for all sorts of different treatment programs? 

So ratings of resources spent, ratings of outcomes out, and it was the same basic scale.  But usually of course it’s dollars and cents and for cost-benefit analysis there’s several little things you can do.  You can turn this all into outcomes, if you will.  Dependent measures for statistical analyses – we still love that in most of the human sciences; I do – so that we have a cost-effectiveness ratio or a cost-benefit ratio, dividing the benefits by the cost.  For example, five dollars produced for every dollar invested.  How do you get that in human services?  Well, two ways.  You can increase client income and you can also save money.  That last one people like to hear a lot, especially people who allocate money on that little hill a few miles from here.  But in a lot of state legislatures, too, they like the monetary savings.  Now, that’s where a lot of human services really have the biggest payoff.  They save money, free up resources, for use elsewhere.  Does that sound like a good thing?  I mean this is getting a little more human, maybe, and a little less just pecuniary.  There’s a whole lot more there, potentially, but the benefit over cost ratio is one way to approach it and you can do that at the individual level when I do my research in mental health services and when I do research in other areas, I try to collect data from the bottom and then work up.  And that way you can actually treat this like in most of our research models much like outcomes, per se.  You can look at that net benefit but you can go a lot farther.  Yes, you can look at capacity and readiness, but I would actually say we should go a little farther still.  We should actually look at the cost of different activities and we should also examine, if we can, the cost of what the activities do inside of the clients, the organizations, and then what they ultimately do in terms of producing specific outcomes.  This is an example that you can’t read so I’ll go over this really quickly but I try to implement this at the program level.  I’ve started in most of my work to do this for individual treatment programs where…

Okay, well let me just give you a brief idea of what I talk about when I say resources.  Do you think you know what I mean by resources?  These are the ingredients of human services.  What does it take to do a human service?  Or, bring it up to one level, what does it take to allocate funds to different human service programs to decide what is an evidence-based practice?  Those resources involve a lot of different people’s time.  Here’s where the idea of perspectives, which has been introduced before, comes in particularly big, because perspectives are not just that of the provider or the funder but also potentially of the community, the client.  When I work in individual treatment program analyses, I try to figure out how much time it took to get to the treatment program if it’s outpatient, if you will, and how much it took in terms of transportation costs.  Those end up being costs that may be just as big as the provider costs but typically only the provider costs get looked at.  There’s a lot more here in terms of alternate perspectives.  Same thing on outcomes but the major resources that most people spend most time looking at, and that are the biggest in human services, is simply the time of the provider and the staff and the rest because we’re a very labor-intensive activity.  Activities include all the different things we do with those resources in an attempt to achieve those outcomes.  Now let me ask you so far in your understanding of this, where do evidence-based practices go if we’re looking at outcomes of improved quality of life, reduced recidivism, as Karen mentioned?  Where do evidence-based practices go in this model?  Activities, yes.  And the question is, which activities if you want to bump this up from just looking within a program to looking at the levels that you guys operate at – state, federal, city?  How would you apply this?  Well these are your different evidence-based practices.  And how much money flows in to those practices, or should?  Do you know?  This is another level of knowledge analysis that we’re beginning to do, I think, but I don’t think we’ve entirely got it, although it sounds like in Philadelphia you got it.  You’re making these decisions, whether we want to or not.  The question is, can we make them in a way that’s transparent and logical, and that everyone can look at and say, “Oh yeah.  That’s a good idea”?  Usually I want to see a lot of different heads going up and down before I make a decision looking at this.  And the outcomes?  They can be non-monetary or they can be savings in resources, they can be all sorts of things.  Processes are the last thing that most people like to look at because processes are the sort of things that you theorize a lot about.  They’re the things that are supposed to change inside the client.  You can’t see them directly, you can’t measure them, and everyone has different theories about why those evidence-based practices help achieve those outcomes, or don’t.  So I don’t know if you want to look at processes.  Let me just offer you a few more frameworks here before my time’s out.  One thing I do at the level that I think most of you are at is listing different candidate activities.  And this is even above the level of evidence-based practices, A, B, C, and D, but instead tools, training, technical assistance.  And if you would allocate the same amount of money to each of those, or the same amount of time to each of those, that’s great and I know they interact synergistically.  I can’t see just doing just one or the other, it’s the mixture that matters.  Which would you choose to list next, though, if you’re interested in how those tools facilitate your mission and the outcomes that you’re held responsible for?  Would you start looking at the outcomes or the resources?  I think most of you would probably look at outcomes.  That right?  If you’re worried about your budget, you know what you actually look at first, right after activities?  You look at resources, typically, because you’re worried about cost more than anything else.  And here’s some frameworks if I can keep on moving forward.  I’m clicking the wrong buttons I’m afraid.  Within a treatment program these are the kind of resources I get into looking at, from personnel to transportation to facilities and the rest, and this sort of matrix – okay, that’s fancy word for a table – that seems to help a lot.  Where, imagine, at an individual program level we have evidence-based practice, or a community level we have evidence-based practice one through three, and then we have outcomes one through three.  Not necessarily corresponding to those activities.  What is the optimum mix of activities that will help us achieve those outcomes?  And you might wonder, “How do I make that decision?,” or, “I know how to make that decision.”  Do you know how to make it in a way that you can defend?  This is the next stage beyond cost-effectiveness and cost-benefit analysis that I’ve seen in HIV prevention. And that’s operations research where you actually systematically try to figure out what mix of activities helps you achieve the outcomes that you’ve decided you need to achieve within the constraints of the resources you have available.  And I can talk more about that but I think I need to shut up.  Thank you.

[End of Audio]