Measuring Return On Investment in Places with Multiple or Multi-dimensional Programs

Hi, good afternoon.  I’m John Roman.  I’m from the Justice Policy Center, so, I’m coming at all of this from a justice systems perspective.  I think that’s sort of important to point out up front.

When David asked me to give this presentation, “Measuring Return on Investment in Places with Multidimensional or Multiple Interventions,” I said, “No.”

[Laughter]

And I didn’t say no because I was unwilling to do it, as obviously I am.  I didn’t say no because I don’t think it’s important; I think it’s exceptionally important.  I would love to sit here and listen to somebody give a really good talk on this subject.

[Laughter]

So, I went out and did – and talked to some people who are thinking about this, and this is sort of what they came up with.  And so, what I’m thinking about here, in terms of measuring this stuff, is imagine a world where there’s no dashboard, where there’s no braiding, where you can’t unpack this stuff.  Right?  Because we’re all probably working in the same places.  We’re in Garfield Park in Chicago, and Frankford in Philly, and Sandtown-Winchester in Baltimore, and, you know, Barry Farms here in D.C.  And we’re all coming out of different agencies and different perspectives.  And then there are community actors in place there.  There are foundations working there, service organizations.  There’s all this stuff going on.

And so, imagine a world where you can’t just unpack it, unbraid it.  Because if you can, then traditional evaluation has a lot to offer.  And if you can’t, then we have to do something else.  So, that’s what I want to propose is something else.  And it’s gonna sound first like we’re taking a big step backwards in time, to a world before regression.

[Laughter]

But maybe we can come all the way back at the end.

All right, so this is what I want to do.  I want to tell you how we – why we cannot infer causality in the presence of multiple multidimensional interventions, if you cannot unbraid, and what that means.  And then go back to this sort of performance metric world, but maybe in a slightly different way, in terms of inputs, outputs, and outcomes.  And then add cost to the mix and see what we can tease out in terms of Return on Investment for places with multiple programs.

Okay, so what do we mean by “multidimensional multiple programs?”  It’s really this idea that I think government increasingly and philanthropy increasingly see themselves as investors who are investing in an intervention in a place, and they want the maximal returns for that investment.  And they’re looking for multimodal interventions.  Philanthropists are talking a lot about, you know, co-investing.  Different agencies are trying to collaborate, trying to do public-private partnerships, Social Impact Bonds, all this stuff to try and get all kinds of people, who never talked to each other in the past, to talk to each other, work together, and leverage each other’s knowledge.  And you can imagine programs with multiple domains of service.  You can imagine an education component, mental health, child and family services, workforce development.  You can imagine all these things are happening right now in Frankford in Philadelphia.

And then you’ll have many programs, with varied domains of service, serving the same place.  You could have multiple workforce development actors in a single place, doing different kinds of things at different stages of workforce development.  And this is what we want to try and measure returns on.

Okay, so normally at this point, we would return to our scientific method for help.  And it says to us to do these three things: specify a falsifiable hypothesis so that individuals receiving the intervention will have better outcomes than those not receiving the intervention; and then control for all competing explanations everything other than the intervention that could affect the outcome, and we’re going to control for that so we can isolate the effect of the intervention; and then we will interpret whatever we get out of that, I’m trying not to sort of parameter - this month.

Okay, so let’s go to step one.  We specify a falsifiable hypothesis.  And we are immediately in dire trouble. You may have some interventions that are targeting the whole place of Frankford; some who are targeting, you know, teenagers, at risk youth in Frankford; some are targeting the parents; some are targeting their little brother.  So, we already are in a world where there’s no real control, where there aren’t people who don’t get anything.  This has been one of the great insights about doing large-scale evaluations in the last few years. Increasingly, we don’t live in a world where the counter-factual is nothing; it’s just less or a different mix.  And that’s really important.  And then the mix of interventions may vary from place to place; it may actually vary within the same place.  So, different combinations.

So, we go to step two.  We want to control for competing explanations and, you know, in order to control for all competing explanations, we need to interact all the possible combinations of all these different things.  And when you’re talking about a place with more than a couple of active interventions, you know, you’re kind of – you’re stuck with traditional evaluative methods; there isn’t much you can do.

Let’s think about what we can measure at the place.  Right?  What can we learn about the sort of aggregate status of all these people in a place?

So, we might want to say, “We want to improve educational attainment.  We want there to be more economic opportunity that’s proximate to the people in a disadvantaged place.”  We might want their health indicators to go up across a whole bunch of measures: diabetes, obesity, whatever.  We want to improve their housing, and we want to improve the whole infrastructure, and maybe what we care about is digital infrastructure.  Right?

So, we have all these things we want to do.  We’re going to do this mix of services.  We’re going to do pre-K through senior year education.  We’re going to do on-the-job training.  We’re going to fix the community college.  We’re going to do apprenticeships, because not everybody needs to go to college to be successful.  We’re going to do workforce development.  We’re going to do all this stuff.

And it’s – in essence, it’s a system reform targeted at a place.  Right?  And so, what we want to do is to get to this dashboard.  But even if we can’t get to this dashboard, you know, we can ask people – everybody – the providers, the intermediaries, the lead entity, the collaboratives – people call it different stuff in different places, the people who are doing the service provision or coordinating service provision across different actors – and figure out, you know, how many people are working on this problem in this place.  Right?  How many units of service are they delivering?  What are the prices?  What kind of big capital investments are being made here?

And the challenge here – and this is not a trivial challenge – is that this requires the people who are delivering services to know, record, and report what they are doing, and many of them do not and don’t know how.  But we at least need to figure out what’s happening.  And then let’s start looking at the outcomes for the place – right?

Some of this stuff’s really easy.  We can look at employment rate in a place.  We can look at median income in a place.  But some of what we really want to do is really hard to measure.  Have we made a place that’s proximate to disadvantaged people more business friendly?  Is the environment better?  Is it – you know, this goes to the sustainability question – is this a place where businesses are going to want to come now and tomorrow and five years from now and help the place get better?

Are we creating an environment that creates careers versus jobs?  Right?  And this is all the stuff that we want to do.  Because getting people to have these individually better outcomes is the sort of proximal goal, but the distal goal, the long-term goal is to have everybody in the place and make the place healthier.  And these are the things we want to measure.  Now, it requires broad information collection, and it’s subjective and objective measures. But if we just look narrowly at objective measures, we can miss important things.

And we want to look at all the sort of confounders – I don’t have this on the slide.  But we want to look at mobility.  Are people – you know, does the place look better because all the people who are disadvantaged left?  Right?  And that’s true in big parts of this city.

So, all of these things we want to look at.  And so, we say, “Okay, well, is any of this causal?”  And it’s not causal in the sense that we can’t say that this – that this is the mechanism that caused these outcomes to change.  But we can say that the place has become healthier.

And the advantage of this is it actually makes the Return on Investment easier than if you did eight– eight evaluations of eight different interventions, where you had eight different cost benefit analyses that said you probably have, you know, five show an effect, two show nothing, and one show it made things worse.  And how do you put all that stuff together?  And you can do it, but it’s really hard.

Maybe an easier way to do it is to go back to these relatively simple outcome measures and do some much more basic analysis.  Measure stuff consistently, and measure it over time and look at, you know, like monthly data on outcome measures.  So X means as a relatively reasonable place to start, look at changes in outputs and outcomes over time, and look and see where things are getting better and what they’re getting better on, and that will help you to identify the gaps where there are – there is, in this milieu, in this mix of interventions here, you’re lacking a component because you can see that educational attainment is lagging, you know, health indicators.

The other thing that we don’t do enough, we do it more in criminal – in the justice side than maybe some other people do, but we try to identify natural experiments.  And this is where something that affects outcomes changes for some reason other than the outcome.  Right?  So, school performance gets better, but it’s not because we implemented something in the schools to make the schools better.  It’s maybe, you know, something else has happened naturally, a change in law or regulation.

And these are really good opportunities to try and look at causal factors, if you don’t have the ability to unbraid, is to say, you know, “We have this – the federal government made a grant available to hire ten percent more teachers with some set of qualifications that our teachers don’t have.”  And we got the grant because they just gave them out; it wasn’t that we had done something or had some gap or some outcome.  And so, now you have this opportunity where the test is this thing; does this thing have an effect?

Okay, Return on Investment in three minutes.  No problem.  Okay, whew.  So, we think of cost as having two components.  It’s really easier than it sounds.  It has a price component, and it has a quantity component.  What does a unit of something cost?  How much of it did we deliver?  If you’re getting your service providers to report out what they’re doing, it’s relatively easy to figure out what the price of an hour of somebody’s time is, an hour of counseling, an hour of some other intervention. It’s counting how much stuff got delivered that gets really hard.  And if can get them to do that, that gets you a long way down the road.  Think about things like fixed costs, costs that do not change with the number of units delivered. The costs of running a prison don’t change, depending on the number of inmates in it.  And the capital cost of building stuff.

So, you get all this cost data, and it’s relatively easy to get.  Better to be measured bottom-up than top-down.  I’m not gonna spend a lot of time on this, but it’s – you know, one way is to look at what everybody got individually and sum it up.  The other way is just to take the budget and divide by the number of people served.  If you do it bottom-up, you can look at how much variation there is, and that’s really important.  Are a lot of people only getting a little, and a few people getting a lot, so the average is really distorted?  Right?  Or is everybody getting about the same amount?  Is it close to what you want?

So, you count all this stuff up, and then you take your simple outcomes measures that we talked about at the beginning – educational attainment, jobs, whatever, health – and you get some economist to put a dollar value on it.  They’re easy to find; they’re all over the place.  They love to monetize stuff.  They’ll monetize the value of a mother’s love if you ask them.

[Laughter]

And they put it all together, and they create one metric.  And then they look at how that changes over time.  Because in my business, in the justice side – it’s less true probably in many of your businesses – but in the justice side, we are all about harm reduction.  Right?  We have a set of people who are costly to their communities, and we want to make them less costly to their communities.

So, even though we’re still spending much more money on them than the average person, we want to spend less.  And we want to look at how that changes over time.  And looking at over these indicators can help you to figure that out.  And I think that’s it.  I’ve got one minute.  This is my contact information, you can follow me on Twitter.