Discussant Remarks, "Understanding the Complex Context of Implementing and Sustaining EBPs in the Real World"

Well, thank you.  My name is Bill Trochim.  I’m a faculty member at Cornell University and also at the Weill Cornell Medical College.  And I want to thank you all.  I’ve already been – I’ve already received more stimulation in this panel than I can handle in a day.  I’ve got six pages of thoughts and notes that I’m gonna try to make somewhat coherent.  But what a wonderful start to this meeting.

So, here we are in the early part of the 21st century.  And I think a lot of what your talks stimulated me to think about was actually a longer horizon than just where we are this decade. 

[Laughter]

And so what I find myself thinking about is the idea that – when I think of the 20th century, I think about engineering.  And I think about a lot of the things that I think we’re still grappling with in terms of language, including the term “program.”  You think about what a program is; and you think about the idea of practice, in any context of practice you’re thinking about, and somehow those two things don’t quite jive.  That is practice really is not a program, and it’s not even a bunch of programs alone.  If I went to my doctor where he’s practicing medicine, or she’s practicing medicine, and I were to receive nothing but programs, I’d think there was something very fundamentally missing.

So, part of what I think was really interesting about the – what I see you and us struggling with in our conversation has to do with language and a shift in our thinking – from 20th century thinking, which is largely technologically and engineering focused, to whatever it is we’re heading towards, which I think we could probably characterize as a lot more complex and dynamic and, you know, the whole systems thinking kind of thing.  And I would also throw in there, for good measure, the idea of evolutionary.

So, when we are struggling with terms – and some of the terms that I heard mentioned – and we’ll all say them over and over again, and I say them, too, and I think I’m in the midst of this shift in my own thinking – we want to do “planning” with people.  We want to do “coordination” with various communities we’re dealing with.  We want to have better “alignment.”  We want to have our – we want to “assess their needs.”  We want to “assess constraints.”  A lot of that stuff is a lingo that came out of an engineering kind of context.

Now, how do we shift that?  I think part of what I would like to suggest, as a way of thinking about things differently – and I’m not sure this would survive, but I am an evolutionist, so I’ll argue it anyway – is to think about evolutionary theory.  And to think about the idea that what we are really entering – trying to enter into here are complex, dynamic, evolving ecologies.  These are not structures.  These are ecologies.  And why wouldn’t they be?  Because they’re made up of human beings, and we are evolved ourselves, and we are dynamic and continuously evolving.

So, it’s worth keeping in mind.  I want to come back to that notion of evolutionary thinking in a little bit.  No, actually, I think I’ll just continue right on with that.  There’s a couple of things if you think about it.  Maybe the metaphor of what we need to be thinking about is less how we engineer this stuff than how it is we deal when new species enter into new ecosystems.  Right?  Because if you think about it, that’s essentially what we’re trying to do.  I would actually extend this metaphor to the point where I would say that programs are, in effect, like organisms.  We have lots of them; we’re evolving them.

All of this program development and program evaluation is really, in a sense, a way of evolving what might happen naturally and maybe not.  Right?  Communities are out there, engaged in trying to evolve themselves.  That’s the natural selection process, where there will be variations, and they’ll be selecting from those variations those things that have some kind of survival value as they ascertain them.

But we’re in the process of actually what Darwin would have called “artificial selection,” not natural selection.  We’re in the process of trying to create new breeds of things, new interventions that we’re gonna try to introduce.  So, let’s – perhaps if we think about this from a more biological, evolutionary, ecological perspective, we’ll start getting the language to where we need to have it be.  And I would start looking at what happens when we introduce new species into new environments, because we certainly have a lot of history with the introduction of invasive species which take off.  Right?

And then we also have the problem of trying to reintroduce species which are near extinct in one context, and we’re just not able to get them to take in a new environment.  And a lot of those kinds of issues are probably dealing with similar kinds of challenges to what we’re dealing with.

While I’m on the evolutionary topic, I’d also just point out that one of the things that concerns me about evidence-based programs, interventions, or whatever, has to do with where, in the ideal sense, we think we’re heading with this.  I mean if we had a world that was totally evidence-based programs, interventions, or whatever, where would variation come from?  Would we be establishing the programs which are evidence based, and would advancement stop?  That is, to what extent would evolution not occur as we increase our promulgation of evidenced-based programs?  And I think this may not be a concern for many of us who are in the business of doing evidence-based programs, but it may be more of a concern to people out there who are on the receiving end.

Which does bring me to the question of language about perspective and incentives.  And here, what I wanted to raise – I don’t even know how to raise this succinctly, but I’ll try.  If evidence-based practice is something that we are doing, we are engaging in, the people in this room and on this event are engaging in, then who are we doing it to?  Or who are we trying to do it with?  And if we take the perspective that we’re doing something to them, we are trying to engage them.  We are trying to get them to plan; we are trying to get them to coordinate with us.

I guess the question that I have is when I put myself in their position and ask myself the question, “Why would I even want to engage with us?” I think that kind of is a question that I don’t hear answered very often.  What’s in it for them to do any of this?  And what’s the incentive system here?  And if we don’t get the incentive system right, which I think is – I’ve heard a number of you folks really raising the question, that the underlying challenge here is, how do we engage these multiple stakeholders?  And how can we,  how can we – how can we, in effect, get them to see what needs to happen here?  But notice that that’s a really interesting kind of way of framing it.

And if you think about it, I guess maybe another way to think about it would be to say, “What are they incentivized to do now?  What do they want?”  And this gets at the question of needs assessment and so on.  But it’s not sort of the needs assessment that my fathers taught me, that kind of structured needs assessment, going out and surveying and getting the questions back.  It’s really trying to get at, what do they – what is their motivation?  Does it have to feel like a struggling act? What is my motivation here?

But I do think if we don’t understand that, then the “we” – notice how I’m struggling with the “we” and “them” kind of language right away.  There’s something fundamentally wrong with that paradigm.  I don’t think I have the answer to it, so I will just pass on and hope maybe we can talk a little bit about us and them.

I really liked a lot of what I heard, but I was struck by John Roman’s comment, “We live in a world where the counterfactual is not nothing; it’s less or a different mix.”  And I thought that was really profound.  And I guess part of the question – several – probably all of you were looking at, how do we engage multiple stakeholders and get – you know, we all recognize that that’s going to be critical here.

I guess one of the things that your comment, John, started getting me thinking about is the idea of the old-fashioned quasi=experimental thinking, the plausible alternative explanations for a causal inference.  And I was thinking, “You know, in research we tend to assume that it’s our job as researchers to figure out what the plausible alternative explanations are, but in fact, they – the people who are in the world out there that we’re trying to engage with, they know a hell of a lot more than we do about that.  And wouldn’t it be interesting for us and them to work together on positing those plausible alternative explanations?”

So, if we can pose data in the form of a dashboard, or any of the kinds of data that you folks are thinking of feeding back to them, or that we as evaluators would feed back, if we can pose data, why should we be the ones to interpret it?  How can we get the interpretations of people who are on the ground and know what the heck’s happening?  We can’t, I don’t think, do that just by asking them.  Because I don’t think they’re going to have much of a frame of reference.

I think we really have to – we really have to work with them to make sure that they understand the nature of what’s being presented.  But my guess is, like most people, they are going to be fascinated by the puzzle of what’s happening in their own lives as much as we’re fascinated as researchers and policy makers.  And we need to find – some of the more practical methods that I’d like us to work on are ways of engaging with all of the stakeholders in identifying those plausible alternative explanations about all of this braided stuff that we’re doing in these communities, which we are inevitably struggling to do.  I think that that may be something worth methodologically pursuing, and I’d like to encourage that.  So, thank you.

[End of Audio]