Replicating Evidence-Based Programs in Boys and Girls Clubs

Ed Mishrell: Good afternoon. So I started working in a Boys and Girls Club over thirty years ago in South Philadelphia as part of a field placement for graduate school, and I would describe myself as somebody said this morning as deeply invested and absolutely convinced that it works without a lot of scientific evidence. So I'm probably gonna come at this from a little different direction. What I was, want to do is share a couple of models about how we've developed a program and implemented it and some of the, some of the challenges that  we have  in doing that, and I'm not sure how many people are familiar with, that familiar with the Boys and Girls club. We have eleven hundred separate organizations all over the country. They're all their own 501C3 corporation. There's four thousand clubs. Some of the organizations are huge, ten to fifteen million dollar budgets, others are a hundred and fifty to two hundred thousand dollar budgets. Some are in urban areas, some are in cities, some are on Native American lands. There's  a club on every military base around the world. So we have a big diversity of communities and diversity of need. Four million kids, one point four billion that sounds like a lot when you see it all at once, but when you break it up in four thousand places it's not very very much. Fifty thousand adult staff, about a quarter of them are full time, which means we have a very transient workforce because about three quarters of who's working with kids and clubs are are part time. So that gives you just sort of a context  of the of the of the environment. And I want to talk about two different programs and how we're approaching them differently, and the first one is a program called Project Learn, and Project Learn was based on some research and thinking by Dr. Reginald Clark about what kids do outside of school that makes them successful in school, and his research led him to believe that kids that spend twenty five to thirty hours a week outside of school in what he called high yield activities did better in school. So that seemed like a good model for us, so we worked with him and developed a program that we wanted to test, and we were able to get some funding, and we did a quasi experimental design, and we adopted his work to how we thought it worked in a club. So that we have, we often have kids for fifteen to twenty hours so if we can program that time so that that's that meets his criteria for high yield we felt that we could have a have a difference, so the program ended up having these kind of components. One was homework help and tutoring, that's kind of intuitive if kids do their homework, if they get help when they need help that that's going to do better, and that was part of his research. The other part was high yield learning activities. Now in his case it was conversations with adults, it was playing cognitive games, it was reading, it was writing, doing the kinds of things that would support learning. We felt like we could take almost everything that happens in a Boys and Girls Club and turn it into a high yield activity if we were if we were really smart about it because twenty five to thirty hours was going to  take them beyond the club . Parent involvement became something that we've built into it so we could educate parents. We had to have a collaboration with schools so that the homework help and tutoring the kids are getting was going to be consistent with what they needed and because kids don't naturally want to do their homework and necessarily do things we build in a system of incentives, which wasn't necessarily paying them, it was more about recognition and goal setting, and that if kids did certain things in the club for a period of time that there was recognition with that. So we'll talk about what we what we did to replicate that. But  just the first thing that came to mind when Karen asked me to talk was all the challenges of replicating it and that very complicated messy system that that I described. So the first one is staff turnover, the second is organizational capacity. We have you know clubs of all different shapes and sizes and sophistication. Third one is different local funder requirements. So we may say this is how the program should work when they go to get it funded locally the funder has a better idea and builds that into what they're going to fund. Community difference was talked about extensively this morning. Partnerships with schools, some of our organizations have school districts that are great partners, other school districts are very focused internally what happens in the school and not thinking beyond their walls, and so that makes it very difficult to replicate the program. And then just uneven uneven resources. We are not passing through money to clubs. They have to raise their own own money, so they have a lot of autonomy in how they how they how they apply that money. So here's the solutions we came up with, and what we didn't do was we didn't try to create a system where we certified or monitored every organization and club that did it. We felt like that was not something that was going to be possible. It would require a huge investment in resources on our part. We felt like that we needed to have a strategy that enabled as many of those four thousand clubs to use this because the evaluation came out really well. The kids after a couple of years were doing better than they were and doing better than the group that they were being compared to. So what we said is there's some core concepts here. This idea of twenty five thirty hours a week of high yield activities is really what's central to making this work. How you end up getting kids to do that is less is less important as long as you as long as you hold fast to to those components of the program. We did all the usual stuff, we did manuals, we did we repli—we did whole binders full of high yield activities that the clubs had done, we created a homework help program, it's the one thing we did sort of try to to to make a little bit more even across clubs so that clubs were, kids were participating in homework more regularly, and there were some incentives for that. You know lots of ongoing training, because of turnover training is a never ending thing. We've been doing this for over ten years, and it's probably one of the most significant trainings we do every year because many of our clubs have new people. We created sample proposals, and this is actually a really sneaky way to replicate something because we created the proposal that they could go for funding. They just had to fill in the local information. But in the proposal it described how the program should be run, so the local funder would hold them, would be the group that monitored it. So don't tell any clubs that that's the strategy if you talk to anybody though. And in some cases we were able to provide pass through funding where we were able to raise money to pass through to clubs, and in that case we did do more monitoring and and making sure that they they were successful. I would say that there's there's over twenty, twenty five hundred plus clubs are running Project Learn right now. So that's a huge huge impact, about a half million kids participated in it last year, so that's a lot a lot of kids. On a scale of adapting from like absolutely fidelity to adaption over here, we're probably much closer to the here than over here. What what's interesting though is the idea of the high yield activity has now become part of the culture of clubs. That if you talk to people in Boys and Girls Clubs as part of the language and the way they think about whether it's a basketball league or an art program, how do we make this high yield? How do we make this about more than just painting or sculpturing or whatever the program is? Now contrast that with a partnership to we're just at the very beginning and through our work with Child Trends and Karen last summer we became very interested in how we can prevent the summer learning loss of kids. Cause we have kind of an advantage that we have kids for a full day in the summer, and that we thought we could really do something here, but we didn't have a program beyond Project Learn, which can be applied in the summer, but there's parts of it that don't work. So we've had discussions with Earl Phalen, the founder of Summer Advantage, and they have a very specific program that's been evaluated in with great results, and we are we are we are looking for funding to to to kind of pilot this in clubs next summer. And it's a different it's a different approach. So the first thing we did, our missions about what we're doing are pretty compatible, but you see in the Summer Advantage where it talks about mastery of basic skills, reading, writing, and math, that's not something that's sort of core to a club. So that's that's the piece that we need to to to partner on. In terms of replicating it, we're really thinking about this very different. Our roles in organizations is kind of figure out the partnership and how this might work, create some funding to test it make sure it works in our environment because Summer Advantage has been going into schools and and doing it, and what we came up with is that Summer Advantage will actually hire the staff that run their part of the program, and the core program is math, reading, and science, but a big part of the program is an enrichment program where it's more recreational focused, there's a field trip once a week. That's what clubs do in the summertime so that we feel like we can we can take advantage of the part of the program that's very structured that Summer Advantage has done and that we can work with the clubs to make sure that the rest of the day at the club is is is focused around that. Running out of time so real quick, the Summer Advantage program is five hours. The need in the community for most parents is they want to drop their kids off at seven thirty and pick them up at six o'clock. Five hours isn't gonna work for a lot of people, so a lot of kids would not be able to participate. So clubs run a full day program. We feel like we can embed the Summer Advantage program into a big a part of a part of that day, and the kids can still be part of the rest of the rest of the program at the at the club. So that it kind of meets, it addresses the summer learning, and it also addresses the needs of parents and doesn't kind of leave the kids stranded for the rest of the rest of the day. The last thing I wanted to say is that these are very different approaches. With Project Learn it was you know a lot of adaption, you've got a core concept about what works you want to get that out to a lot of people. With Summer Advantage we're we're saying that you know what what what Summer Advantage has has evaluated that works is really this core curriculum that's very structured. Not something that a lot of clubs are really are really good at, so our true partnership we're bringing in the expertise to run the program, but enabling the kids in the Boys and Girls Club that need that the most to have opportunity to that. I'll say the the you know the one of the things with Project Learn as I mentioned is that it's become part of the culture of clubs, is there's sort of this ongoing innovation that happens. So even though we evaluated the program a while ago there are some things that people discover when you have two thousand plus entities doing it that really work. It's hard to go back and evaluate all of those, so I think you know we try to make our best judgment about is that a really good idea or not and then try to share that with with with others. I thank you for the opportunity. My time is up.