Evidence Matters, Part 2: Improving Community Outcomes

Episode Topic: Improving Community Outcomes 

Listen in to a conversation among researchers, practitioners, and community partners on how they engaged in deep learning from the unexpected results of a multi-year randomized controlled trial study on preventing homelessness.

Featured Speakers:

• Carrie Cihak, King County Evidence & Impact Officer
• Vincent Quan, Co-Executive Director, J-PAL
• Kimberly Dodds, Homelessness Prevention Program Manager, King County
• David Phillips, Research Professor of Economics, Wilson Sheehan Lab for Economic Opportunities
• Tanya Robertson, Program Coordinator, Youth and Family Homelessness Prevention Initiative

Read this episode's recap over on the University of Notre Dame's open online learning community platform, ThinkND: https://go.nd.edu/97eb64

This podcast is a part of the ThinkND Series titled Evidence Matters.

Thanks for listening! The ThinkND Podcast is brought to you by ThinkND, the University of Notre Dame's online learning community. We connect you with videos, podcasts, articles, courses, and other resources to inspire minds and spark conversations on topics that matter to you — everything from faith and politics, to science, technology, and your career.

• Learn more about ThinkND and register for upcoming live events at think.nd.edu.
• Join our LinkedIn community for updates, episode clips, and more.

Transcript

Overview of J-PAL and Youth and Family Homelessness Prevention Initiative (YFHPI)

1 0:06

my name is Carrie Cheehok. I'm King County's evidence and impact officer, and I'm passionate about advancing racial equity through bringing together teams like you have here today. we've got King County staff, our research partners, and our community partners all here together. And I'm really excited to be here with our guest host, Vincent Kwan. Hi, Vincent.

2 0:31

Hi there. Thanks for having me today.

1 0:34

Yeah, I'm so excited to have you. Vincent is co director at J PAL North America. Some of you probably know about J PAL, many of you probably don't. Vincent, do you want to tell us a little bit about J PAL?

2 0:47

That sounds great. Thanks, Kerry. So J PAL stands for the Abdul Latif Jameel Poverty Action Lab. We are a research and policy center based within MIT, and our focus is conducting high quality rigorous research to identify the most effective ways to reduce poverty. we've done a lot of work with King County, and we're really thrilled to be here today.

Introducing the Panelists

1 1:09

Yeah, I'm excited to have you, Vincent. Our relationship goes back, quite a ways, actually. I'm feeling like this is a little bit of a reunion for us. It's really great. the project we're talking about today, the Youth and Family Homelessness Prevention Initiative, you'll hear us talk about it as UFIPI, Youth and Family Homelessness Prevention Initiative. So that project was actually the first partnership between J PAL and LEO. LEO is not, I'm not talking about my astrological sign. I'm talking about our friends at Notre Dame in the Lab for Economic Opportunities. this project was the first partnership between King County, Leo, and J Cow. Super exciting. that was, starting back in 2015. I am a Results for America local government fellow, and J PAL had organized a convening at MIT, that's where I first got connected with J PAL, and I met David. Hi, David! David Phillips, who's going to be talking with us today. Hey! and it's really exciting, that initial partnership has blossomed into a range of activities and projects David's been working with us on, transit and helping us understand how we can better deliver mobility to people with low incomes. We've got another project with J PAL on delivering mobility to people with disabilities. we've got, work going on in the climate and energy space to help low income homeowners. In the clean energy transition. So there's so much going on now, including this webinar. it's super exciting where should we start vincent? Should we tell people a little bit about evidence matters Evidence Matters is this webinar series. We're super happy to have you all here, and we really built this so that we could, three purposes, so we want to share our knowledge that we're building from our learning partnerships, and use that knowledge, really across the country and help to advance more better together. More equitable outcomes in our communities. We also want to celebrate some of our successes and share some of our challenges. You'll be hearing about that today. And we want to build connection with all of you who are tuning in from around the country, so I'll drop my email into the chat, once we're through with this intro, and I encourage you to be in contact with me you want to share some of the work that you're doing or have some reflections. I'd love to hear from you. what we're going to do today is, lead a conversation with, as I said, we've got our research partners here, David Phillips, our community partner, Tanya Robertson. and one of our King County staff, Kim Dodds, here. And this is absolutely one of my favorite projects we've worked on in King County. I'm so proud of this team. I'm so proud of the work that they've done. And let me tell you a little bit why. So first of all, this project, represents partnership and commitment among the people on the screen. to make a good, evidence building project work, you need the research expertise, you need the program expertise. You need the commitment to community and the community expertise. It's so valuable. And so we've got all three of those types of partners here The other thing I love about this project is that we got a really unexpected result We had every reason to believe the work we were doing was going to have a big impact for our community And the result we got back Was a null finding meaning the research didn't show that it had an impact And that's really tough, right? it was unexpected, and I'm so proud of the team because they worked through that together. they learned from it, and they really invested in that deep learning. the partners, Are supporting each other, and they're using that finding to make change that improves things for the families we serve. the three types of partners here, David, Tanya, and Kim. our real partners in this work are the families we serve, and we need to hold them central in this work. even though it can be disappointing and scary to enter into a research project like this, where you don't know what the results are going to end up being, And in this case, we had a really unexpected result. Thank goodness we know that, right? because it gave us the opportunity to really dig in and figure out what wasn't working right and then correct it and make it better so that we can serve our families better. I'm really excited to be here. we're going to reserve time at the end of our discussion vincent's going to lead a discussion with our partners here and we're going to have time at the end to Have some of your questions answered by our panel. So I really encourage you to put your questions in the chat and we will get to as many of them as we can let me just give a brief introduction of the folks who we have here today. So let me start with tanya Hi, tanya. How are you?

4 6:36

I'm

1 6:37

good.

4 6:37

How are

1 6:37

Good. It's great to see you. Thanks for joining us. So Tanya's at Therapeutic Health Services and they provide a range of supports for families and kids who are at risk of homelessness. They're really one of our strongest, best community partners and award winning too. They won some of our first Best Starts for Kids awards. They were recognized by the Inspector General of the United States of America for their COVID response work. They've really done amazing work here in our region that has had national impact. So thanks Tanya.

4 7:12

Thank you for having me

1 7:13

Kim Dodds. is our fearless, leader of the Youth and Family Homelessness Prevention Initiative at King County. She's in our Department of, Community and Human Services. she's leading strategy and implementation of this program, working with all of our community partners, who have connection with families in King County. And I just, I'm so appreciative of Kim for digging into this work, using the data and evidence to help deliver better results. Thanks, Tammy! Thanks for being here, Kim. And David Phillips. Hi, David.

7:52

Okay.

1 7:53

I see David almost every week. I'm happy to see him twice a week. David and I worked together for many years now. David's a research faculty at the Lab for Economic Opportunities at Notre Dame. this is the first project we worked on together, and he really, jumped in and stuck with us all the way through. it's really wonderful to see him here today talking about this. So with that, Vincent, I think I'm just going to turn it over to you and you can take it from here.

Understanding the YFHPI Program

2 8:20

That sounds great, Kerry. Thanks so much for that great introduction. And thanks again for having me on the session today to facilitate this great discussion about null results. So I'd like today to feel very much like a conversation among all of you. like Kerry mentioned, this research project and partnership that you all have formed together is really inspiring, you all bring different and very complementary perspectives to this type of work, including. how high quality research and sometimes unexpected findings can be utilized to improve outcomes in our communities. I have prepared a number of discussion questions for all of you, but I'd love for it to feel conversational, so feel free to jump in and respond to each of your fellow co panelists as Kerry previewed at the beginning, today's session is going to focus on null results. How do we make sense of null results in rigorous research, and how do we interpret null results and potentially act upon null results to better serve our communities? before diving into that discussion, though, I'd love to hear a little bit more about the idea behind the YFIPI program specifically. Kim, I'll hand it over to you to talk about what the YFIPI program is intended to do, how did the program work, and what were some of its goals?

4 9:33

Thanks, Vincent! The AVIVI program was designed under Best Starts for Kids, so again, that premise of every kid should have their best start to ensure that families or youth that were housed could stay housed, that their vulnerabilities, would not result in housing loss for them, that we could put together a program that would hold up people that were facing imminent risk of losing their housing, and hopefully get them into a place with great case management, long term housing stability. So it really was our goal and the desire to test a theory in this does this work? How effective is this? And I think the reason for that, Vincent, honestly, was. There is no great national evidence, and you hear a lot of, anti theory actually. You can't actually tell who you prevented homelessness for. You don't actually know that what you gave them is why they didn't become homeless. Maybe fair enough, but we wanted to challenge that notion. We wanted to say, yes, there isn't a perfect study out there that says if you do A, B, and C, you will stay housed, right? People are complex. The challenges they face are complex. So we were really interested in this. Is case management really a critical component of ensuring housing stability? Rent assistance alone doesn't require that you contract with, retain, train a lot of highly educated people to do excellent case management. Essentially, rental assistance alone could be cost effective. we wanted to test that theory because if I have a thought partner with me, if I have somebody highly trained like Tonya working alongside me, we're virtually married to each other throughout this process. I'm going to do better. I'm going to be more stably housed. If I have that partner. I need a lot more than that. I need access to employment. I need a connection to local resources. So that was really the premise of why we wanted to do this is, Hey, let's figure this out. we want to be wise with the investment that the community has given us. And if the goal is housing stability, if one is more effective than the other, let's test it.

Researcher and Community Perspectives

2 11:51

That's really great, Kim, and it's valuable hearing that perspective from the government side about why pursuing rigorous evaluation is important to help tease out what is the impact of the program and is it actually achieving its intended outcomes. David, I want to hand it over to you now. from your perspective as a researcher, what excited you about this project? And in what ways did you try to show up as a researcher in this government community driven research partnership?

12:18

Sure. in terms of what was exciting about it, the model of UFIP seemed innovative and we were excited about other people and other communities learning about so Homelessness prevention programs exist in almost every community. I know there's some folks outside of king county here, right? There's those folks there's almost certainly homelessness prevention programming in your community Often those programs focus heavily on emergency financial assistance and there's a little bit more research on the effectiveness of that A family on the verge of eviction right? that can be a really complicated situation with a lot of things going on. Legal, negotiating with landlord, trying to figure out how to generate more income. It's just a bunch of stuff going on. And so there's a lot of reason to think that having a human being next to me. walking side by side with me can make a really big difference. And so I think we were excited looking at this saying, what about this sort of holistic response and homelessness prevention? How much does that add beyond, some of these one time financial assistance programs that are a bit more common? So for us, that was exciting. on the partnership side of it, I think it was really neat to step into a place where we were partnering both with local government and with a couple of dozen, local nonprofit organizations that were implementing this program, that were working in lots of different communities throughout King County and be able to work with their expertise and benefit from what they knew about, the folks they were working with. And so that we could try to work together to design an evaluation. doing rigorous research and high quality social service programs and having those happen together, rather than fight against each other. It takes some work. And so being with a group of partners who were excited about that, who were going to give us honest feedback when we came with an initial research design mutual sense of everybody's expertise and being able to work together to get to designing a study that would ask a question that everybody thought was important. It would do that in a way that was manageable for everybody. I think was something that was exciting for us from the research side.

2 14:03

That's great, David, and thanks so much for talking about the importance of the strength of the partnership that you all formed with the county Tanya, I'd love to hear your perspective as the one who was on the front lines working with community partners and families about what the research experience was like. What were some of those concerns and how did building this strong research partnership alleviate some of the questions and concerns that you had about engaging on this project?

4 14:31

my initial concern was because it was a randomized study, I felt like maybe our most vulnerable families wouldn't get selected, and so I was concerned about the data. If you're. selecting households that aren't most vulnerable. question in my mind was what kind of data would that produce? after that, I thought, this is a big opportunity to gain insight and knowledge so that we can work together, in changing whatever needed to be changed. initially, I was like, In my community, it's like a lottery, and that's puts a bad taste in people's mouth because they never think they're going to get selected. so that was my initial response, but building the relationship and partnership, I just learned it was an opportunity to learn and to grow. Vincent, I wanted to add something to that, it was really cool, and David alluded to it a little bit, is that in that early design stage, we did bring in community partners, and that Tonya is talking about, really came out loud and clear. Hey, if I have a family walking in, they have vulnerabilities spilling out of their pores, right? Like they are the most vulnerable. It would be such a disservice to them if they got randomized intervention assistance alone. because of their feedback, because of their concerns, David, King County, we all strategize and said, what if we built a research study that took those highest level, most vulnerable households out of the study? Let's take them out. And you know what? That alone gave that pause and that rest to our partners to say, okay, it doesn't feel unethical now. That feels a little bit doable. David, did you want to add anything to that part of it?

16:21

No, that was great. It was a lot of back and forth in that early process. And so I think part of what was great as a researcher was being with a group of folks the thing I worry about in research partnerships is when I work with a partner and they're concerned about something and it doesn't come out. And so I think one thing that was great about this partnership. folks, voiced, concerns early on. And we were able to say, okay, how can we work together? How can we build a research design that still answers the question that we think is the most important question while doing it in a way that everybody participating is okay, yeah, this is something that we feel comfortable with how we're going about doing it. We think we're treating participants with respect. we think we're dealing with these tricky ethical situations in a responsible way. there really were, situations that I'm blind to as somebody who's a little bit more external. you need the partnership to make sure that I don't walk into, a landmine that I'm not aware is there.

2 17:07

I really love that. And I think this is why this is such a strong case study of how this really, powerful research partnership where there's respect in all different directions and valuing of expertise from different perspectives ultimately makes for a much stronger research project that can also meet different goals So thanks for sharing that additional perspective. I'd like to shift gears a little and talk about the unexpected findings. the results came back as null which is not what anybody had anticipated. I want to hear about how you all as a group collectively worked through making meaning of those results. David, let me hand it to you first to talk about what were some of the key findings from the study.

17:50

what the study did is compared two groups of people. Everybody was receiving some homelessness prevention services. some were offered, just emergency financial assistance and other were offered this, combination that Kim described of emergency financial assistance and intensive case management we followed up with those two groups through existing records. About a year later, we looked for things like, housing stability surprisingly, we didn't see lower eviction filing rates, for people who had case management on top of the financial assistance. we actually saw higher rates of use of homelessness services, like emergency shelter and street outreach for that group. that was a bit surprising. we didn't see a whole lot of change in some further downstream outcomes from housing stability, things that might be good measures of quality of life, Employment levels and things like that. And yeah, and so overall, I think Our expectations perhaps going in is that we would see, lower use of homelessness services, higher rates of employment, for the case management group. what we saw in the end was a little different than that.

2 18:42

David, given that these findings were quite unexpected, it must have been hard to think about how to relate the findings to Kim, Tanya and the rest of the case management team. How did you go about doing that? And what were some of the key principles that you had in mind?

18:57

key principles that we had in mind. I think the first part, is partnership. it's really important that there was this strong partnership, We had gone through some difficult questions about how to design the study in the first place. we had all bought into the question that we were asking and how we were doing that as being the right way about asking it. And so I think that builds trust in the results for all of us. just the fact of having partnerships regularly seeing each other weekly over multiple years, builds a level of trust, of being able to go in and say, okay, here's some news I have for you that is a little different than what I expected or you expected. And so I think that baseline of partnership is really important. A lot of the pottery that we tried to take from the research team, the academic side of things was to start with small and humble questions of just saying, okay, this is this thing I'm seeing this surprises me. What do you make of this? for something like Tonya, who's really close to the program or Kim, who sees the program in its entirety, I can tell you some things that are happening that we can see in the data. I'm not necessarily the best positioned person to say why we're seeing those things. going to our partners and saying, what are things that you might think are going on that would explain what's going on? How do we understand this is then a group effort rather than researchers should be involved, but it can be a little bit too much research driven. And so in partnership let's try to understand and interpret these results, having the team try to do that together was something that we tried to do.

2 20:09

That's great. it really speaks to the importance of the strong partnership that you all developed to communicate with nuance what these findings were and help interpret the meaning of those neural results and unexpected findings as well. Kim, I'd love to hear your perspective as the person who was responsible for the program itself. These unexpected findings must have been a lot to take in. Tell us about your initial reaction and how you started working through that.

4 20:38

I think my initial reaction was disbelief, shock. How is it possible that a partner walking alongside you Especially partners that we've invested so much time and, energy into making sure that they had training and, really important modalities that would work, to support families. how is this possible? This makes no sense. And I think that's the exciting part of this that when something happens that makes no sense, Like David said, we started sitting around going, so why might this have happened? Why could this be occurring? in those pieces of trying to understand we really gained perhaps the biggest learning. I think the null result in many ways was the best result for us. If we had this overwhelming, oh my goodness, 95 percent of people did better if they had case management and 42 percent wound up in the homeless system if they only got rent assistance you're whew, okay, job done, right? Like keep marching forward. When you get a null result like what we got, It was really, Oh, now we have work to do. now we need to figure some stuff out. Tonya plays a huge role in that partnership with David and the other providers, everybody being given a voice, so going to, our agencies and saying, the results are surprising, let's engage in some conversations. Why do you think this could be? Bringing all those voices together, we could challenge some thoughts and start to look at ways that we could do things differently. And, create success out of a null result, which is exactly what we've done.

2 22:26

Kim, I'd love to jump on that point where you mentioned sharing some of the key takeaways of the null results with other folks What was that communication process like? I imagine that for folks that you shared these findings with, they may have initially had that same shock and reaction that you initially experienced. So what was it like? to socialize some of the key findings with them and help them process what this meant.

4 22:53

think it was. I, Tonya will be able to speak to how did it feel to be the one that was sitting there and hearing that case management might not be proven to be as effective. there was fear for all of us. Imagine the ripple effect it could have in the communities across the country. why would we pay for these expensive interventions if rent assistance alone is enough? we wanted to immediately get on top of why we think this is what happened. We did have a couple of thoughts as we were working through this process of, struggling with this. I wonder what the impact will be of the struggle we're facing throughout the process. it was a long journey. It was about two years of data collection. now we need to test this theory because. If this is the reason the results were there, we have some work to do.

Program Improvements and Case Management

2 23:40

That's great, and I want to share some of our reflections, even at J PAL, having supported a number of different impact evaluations that have also produced unexpected null results. It often is difficult, whether or not you're a researcher, or the government, or the non profit implementing partner, to grapple with some of these initial results, because they can feel quite shocking and unexpected, once You are able to process those key takeaways. You're able to dig a bit deeper and ask the question of why might not it be working as effectively as expected, and then it does enable you to shift gears a bit, to pivot the program, and tinker with different models and components to more effectively serve the communities We are hoping to support. And so I think these types of null results can feel shocking and unexpected, but they're not uncommon. this is not something unique to King County, but I appreciate you all sharing that perspective of what it was like to process those unexpected findings and to make sense of them. I want to talk about how you all were able to leverage these results to improve outcomes and improve the design and implementation of the program to serve folks within the community more effectively. Kim, talk about what kind of changes you instituted as a program manager based on what you learned, and how do you think the findings helped you improve the way you worked with families?

4 25:07

what I alluded to just prior was we had some concerns throughout the study. One of the things that we were battling was extremely high case manager turnover. We had no wage standards for the program. The program didn't fund a lot in administrative, resources to the agencies that were doing the work. And so we had case managers having pretty low salaries and we had. We have one to three case managers turnover every couple months. So then there's retraining, reteaching the study, the efficacy of how they're implementing the randomized control trial. All of those things are a concern, If you've got this turnover and you're trying to train people how to hold fidelity to the model of the research standard, it's a challenge and a concern. people coming and going responsible for the data you're learning from in the program maybe leaving their position, we had concerns about the quality of the data based on the turnover as well. And so we decided we would address that. Those concerns in the next iteration and say we're only going to know how effective case management is if we can effectively keep case managers. So we implemented an iteration 2. 0 minimum wage standard that required the case managers make good wages. We also held a cap on the caseload. So before agencies would decide on their own, how many, households a case manager might have. So we had some at 13 households. We had some that had 40 households on their caseload, and we decided we'd create a standard of 15. And then to control for the data, we decided as program managers, we would Consistently, which we started once a week, every week, and go through the data with them together and make sure that, they're entering data notes that the data is high quality, that there's not data errors going on throughout the process. And then we meet with them every other week, we've been doing this for a year and a half. And so we have some control over when they're exiting and ensuring that they're exiting. And again, that data quality is good. So we've solved for some of the bigger challenges. But I'm not going to say that wasn't without an impact, Tanya probably is thinking, oh yeah, there was an impact for all these changes.

2 27:29

Listen, Tanya, I'd love to hear your perspective from the perspective of being on the front lines providing case management, how do you feel like these research results and the subsequent changes that were implemented to the program impacted what your day to day looked like?

4 27:44

I'll start with the impact. When I got the results, I looked at myself as a case manager. What am I not doing correctly? What can I do better? So I self reflected because I'm an overachiever and my goal is to keep people stably housed. Then I looked at the world around us. A lot of my clients being laid off, not having jobs, the rent increase. Then I start looking at other systems. When you talk about the homeless system, I start having, building relationships with them to find out, wow, I exited this client. They were stably housed. Now they're showing up in your system. What's going on? It also had me have the courage to go back to clients and say, Hey, when I exited you, you were stably housed. Now I'm. getting a report that you're homeless. I start getting feedback that made me understand why we got those results. some clients are desperate. When you're facing homelessness. and you have five kids on one income and they've raised the rent$500 or$400. some of the clients would say, my past clients oh, I wasn't really homeless. I just said that because I needed move in assistance. for me it was like, aha. I'm understanding now why we got those results. it gave me the courage to dig a little deeper. When I would partner with agencies and talk to them, they were like, Oh yeah, we saw them one time. Oh, my fault. I didn't close them out. We didn't give them any services. But they're still showing up as active. So for me, it just had me, have the courage to talk to past clients, talk to other agencies and figure out, this person's on your caseload, but they say you never gave them services. And they're like, Oh yeah, that's just an error. We should have closed it. So it gave me a more wider view of the whole process in general. and some of the changes they made, which I've been in the nonprofit field for over 20 years. I never got into this field to make money. I got in this field to serve. Increasing our wages was fine for me, but more importantly, was reducing the caseload. When I'm working with 31 families, I can't be as intimate and as engaging as I can to see what else they need besides rental assistance. Once they reduced our caseloads, because I work in the home with clients. once they reduced the caseload, I was able to, find out, this person is suffering from, trauma Let me get them connected to a mental health organization. It helped me, make the family, more whole than just giving them rental assistance. I was able to spend the time needed to really help them stabilize. Not only their housing, but their mental wellness. and their children's wellness. after we got those findings, the changes that were made, were positive toward the program and it's helped me as a case manager identify the needs that the clients need.

Lessons Learned from the Project

2 30:58

That's really great. Thank you for sharing that perspective, Tanya. Kim, I know you mentioned earlier that in some ways the null result actually ended up being the best result. And I think some of the perspectives that you all are sharing in terms of. really trying to investigate further why the unexpected results showed up and also some of the key programmatic changes that happened as a result. And Tanya, your perspective about how case management looks different for you now and how caseloads look different for you, I think, really speak to some of the concrete changes that happen as a result of the null and unexpected findings. it's a powerful story about how rigorous research results, even if they don't initially say that the program is super positive, or make a huge impact, can still lead to very positive changes for the communities that we are aiming to serve. And David, I'd love to hear your perspective about what other lessons did you learn from the project, and how has that impacted the way that you work with community partners and also governments on research? what's motivated you to stay engaged rather than just wrapping up the study saying that it's a one and done and moving on?

The Importance of Partnerships

32:10

Yeah, the biggest thing to me, I think that it underlines about how we've done things in the past and how we emphasize continuing to do it, where I work at Leo, is just emphasizing connecting with the right partner. And so I, Kim and Tanya, by listening to what they've been talking about today, I just underline what they've been saying and how they've worked with this program and their clients and worked with these results and that reaction of saying. why are we in this? We're in this because of the clients and the best thing that we can do with these results is improve services as much as possible. The partner who takes that perspective is exactly the partner that we want to work with. And so I think for us it underlined, this has been such a fantastic partnership. And so when we, look out to make other partnerships in the future, it's that's underlined three times of yes, it's a big key to make sure that we've got the right folks in the room. cause it makes such a huge difference. So that's the biggest thing I would say in terms of our side, how we engaged as researchers, I'd say, I think there's a lot of academics actually out there who want to do both things, right? They want to do high quality academic research and have it disseminated to other communities and have other places learn from a place like King County. I want to publish in academic journals and also do right by their partners learn from results and do more than just publish things in journals that, only economists like me read, want things to be used locally. I think a lot of academics operate in an environment where that's difficult to do. The incentives they face in terms of, tenure promotion and publication don't always give strong support for saying, Hey, let's dig in on questions that are. about how this is used locally. I think I'm fortunate to operate within an organization that says, our goals are both of these things when Leo takes on projects, we want to, both generate highest quality research and work with partners on having that used locally. Everyone behind me says yeah, do you invest time in this? I think goes a long way. And so I think I'm fortunate in that sense. and I think a lot of academics are similarly oriented that way, that they want to do both of those things.

2 34:01

That's great. Thank you, David. one thing that I have been thinking about, especially when it comes to null results and unexpected findings is that it can be difficult to grapple with them early on. I want to ask all three of you a question, knowing what you know now about the results. if you could go back in time, do you think that you would do it all over again, and why or why not?

4 34:29

I'll go first, because I think it starts with, the program and the interest in studying something. Disappointing results. Disbelief in the results, right? no, it can't be that. understanding it created an opportunity and the partnership's key. Honestly, I don't know that I would be saying today, Oh yeah, I would do this again in a heartbeat. It was a lot of work, but that partnership working with David, with Leo, with J PAL partners, with Tonya and the partners, it was such a great collaboration that, yes, I would do it again, I believe that null result pushed us to say and why, and that and why is where the real value might have been. We didn't know that to begin with, but we sure know that now. So that would be my feedback. I'd jump into it in a heartbeat, as long as I had, like David alluded to, the right people in the room, the right levels of curiosity, the right partners. And I would definitely, continue to do this work just because if you have never faced homelessness and you're a single parent, you've got three kids, you don't know what that feels like. And so I would always do this kind of work to help prevent some child from being on the street. And to go to school from a car or to go to school from a hotel, I've dealt with them all. I would continue to do this work. There's no doubt.

36:10

Yeah. And absolutely for me too. And I think for me, a lot of what I would say has already been said. I think the partnership and what we learned from actually doing the study, related to what Tanya just said, as a researcher to be invited into the space of both the county as it's operating the program or colleagues as they're working with their clients to be able to, hear their stories to be able to think, critically about, okay, how can this be as the best thing that it possibly is. And just to be able to help myself learn from Kim, from Tanya, from all of their colleagues. I think it, for me, an incredible experience when it's done in the right partnership. And so it's, yeah, absolutely. Do it again.

1 36:44

Benson, I'll chime in here too. I'm really glad that the team responded that way. when we, undertake a study, we would love to see really positive results that affirm the work that we're doing. And the most important results that we get are the ones that are unexpected because that's what we need to know, right? We want to make sure that we're doing the best for the people that we serve and The reason we undertake this kind of research is to uncover those areas where we need to make improvement and do better. And again, I'm just so proud of the team and I hope everyone, here with us, can feel the passion that they have for, really delivering, and improving outcomes for people in our community.

2 37:34

Great. Thank you, Kerry. And so I'd love to ask you all what's next in terms of the research partnership. You all have formed such a strong relationship and gone through a really important research project together. What comes next? What are some of the important research questions that you all are tackling and are excited about?

4 37:52

Yeah, David, what's

37:53

next? Yeah, so so at our lab, we're doing a bunch of different work. So we're working some with. King County on a number of different projects, both related to homelessness prevention and other topics. So I think continuing to learn about, generate evidence about what's making a big difference, for people in situations of poverty and using that information to make those programs better. that's what we're up to. I think specifically with homelessness prevention, we're working with organizations all around the country, including in King County to learn about other aspects of homelessness prevention. Aspects related to, lots of different parts of it. Like there are lots of questions that came out of this project, like not just about case management, but implicit in our conversation is it's not just, should we do something like case management or should we do financial assistance, but who should we do it for? Like who should be prioritized for these different forms of, of emergency assistance. So we're working with. One other partner in King County right now on a study related to that. we're working with, organizations around the country to think about questions of like, how, like these questions of how intensive should these be, how much time of case management, how big of caseloads should be, how much financial assistance is enough, right? But those are other questions that we're working on with other partners as well. yeah, lots of stuff going on, lots of things to learn. Any one study, we learn a whole lot. That's really important. It also. Raises new questions that are important for us to keep building on in the future. So yeah, working on a bunch of stuff, both in King County and elsewhere, related to this.

4 39:11

And so I want to add that was an added benefit of this partnership early on is that David introduced us to other people that were doing similar studies and we've been connected and meeting, across the U S together over the course of the last several years. And that is a value add that I didn't even know would come from participating in this process, the relationships I've built, the learnings I've obtained, as a program manager from partners across the country doing this work. That was, that was great, and I hope that continues. I hope that this builds. interest in others, gathering together and learning together, but, that was sure a benefit that I've appreciated as well.

39:53

Yeah. Kim and a lot of our partners in San Jose, Chicago, Houston, all over the country, right around the time the enrollment in the AFIPI study ended was right when COVID hit. And so then a lot, yeah, a lot of our partners were transitioning to scaling up massively to do COVID era assistance. And yeah, this, That, that was one piece of the immediate follow up, oh, can we connect these folks together? And they learn from each other about how to just even implement this, new thing that everybody was trying to do all of a sudden.

2 40:21

That's fantastic. Thank you. And I know that we want to open up the session to get some questions from the audience, and I see that there are a number of questions here in the meeting chat. Kerry, how should we go about, go about this. Should we just look at the questions and voice them out loud or what's the best way?

1 40:39

Yeah, why don't we see if some folks would be willing to Come off mic and voice the questions themselves. I think I see that. One of the first questions was from jonathan oskins. Jonathan, would you pop off mic and ask your question?

3 40:56

Sure. Yeah jonathan oskins, los angeles county, department of health services, housing for health And the homelessness prevention unit and, out here, I worked in direct homeless services for a very long time, on the streets and such, and I'm new to this type of work and I would like to be effective. I've seen the studies, prior to these regarding how, regarding case management. So I was wondering whether among all the other great questions here, if there's a standardized. intake form assessment survey used. I assume there was, but I didn't know if it had changed over the course of the program or if it was looked, there, it was being looked at to be updated. Best determine which case management needs might be useful. obviously as a case manager, I try and talk to my folks, gain rapport, trust, see their needs, which can change over time. But in the very beginning, identifying some of those things can be important, I think, but you're the expert. So I wanted to ask.

41:55

And do you want to take that? You want me to? Okay. Yeah. So the program didn't have a standardized, intake survey. It's one that's, that, that was, locally specific. So it's, I think the one in King County is not used in other communities, although the one that this program used had some features in common with, for instance, I believe a prior eviction prevention program that the city of Seattle had run. and so that, yeah. So the program, people were both eligibility for the program itself. And whether somebody was eligible for the study, which Tanya alluded to earlier, depended on that sort of risk scoring tool. So the very highest risk people on that all got case management people at a middle risk range. We're also in the program, but then we're in the study. and that assessment persisted for a while, though, actually I don't know if Kim or Tanya may know whether that's changed between the time of the study and now.

4 42:45

The one thing we haven't mentioned, though, that I just did want to mention is, People had to opt into the study. So we haven't talked about that. So the highest intensity households were based on that assessment that David just spoke to. They weren't eligible to be, considered for participation. And then they were informed about the study, all about the study, what it meant to be randomized in this, and they could opt out at that time. So if they said, Oh no, I want case management, I need case management. They were never randomized at all. So I think it is important to say that there was also choice, and say you had been randomized to receive branch assistance alone. If you made a different decision, you could pull out of the program at any given time, out of the study, and then just be enrolled into the case management program. So there were other options as well. I just didn't want to leave that out.

1 43:42

And then, maybe Regina, you could pop off and, pop onto the mic and ask your question of Tanya.

5 43:50

Hi, can you guys hear me okay?

1 43:52

Yeah.

5 43:54

Great. let me just, I was curious, Tanya, as a case manager, like on the front line, Did you ever feel or was there ever like a vibe from the other case managers? Was there anything that you were like, maybe some type of, not a red flag, maybe like a yellow flag that you were like, thinking like, oh, like this might affect the results, like maybe this isn't working, like prior to get those results, was there anything like the case managers As like a coalition kind of felt in that. I don't know if that makes sense

1 44:27

So trying to manipulate the way you worked with people so that It would influence how the study results came out. Is that kind of what you're asking regina?

5 44:37

No, not like purposeful. I just mean okay, like you're working with someone like in the past. I also was a case manager and they were doing some type of study and a few of us that were like We weren't necessarily hurt. We didn't necessarily have that much of a voice with the upper ups in this and we would have like moments with certain clients that we were like, Oh, I don't know if this is going to work for this client like this might affect the study like this situation, It's hard to, but we couldn't, yeah, like we felt oh, this is a weird thing that like maybe should be in the study. Maybe shouldn't be like, I don't know if there's a good way to describe that. I'm just curious if the vibe you were like, oh, maybe there is going to be like negative results or maybe there will be no results because of this experience that I had with the client. And the answer might just be no. And you're like, Regina, what the heck are you asking?

4 45:31

No, I think it was a very valid question. And I get what you're saying. just for me and the group that I worked with prior to, the new, 2. 0, I took the research out of my brain and I just did the work, right? My initial reaction was. Made me feel uncomfortable just because I felt like what if a family really needed this service, but they were placed here. So that was my only concern. But as far as the research part of it, I took the research out of my brain and just implemented what they asked me to do and then was excited to hear the results and then disappointed, but then okay, research is a tool that helps us grow and change. And that's exactly what we did. But I do understand your question. oh, and I also want to say I didn't really have a lot of collaboration because we had such a high turnover rate. So we really couldn't sit down and discuss or figure out stuff because every other month people were leaving And with this new cohort, The turnover rate has not been like it was in the past So we're able to connect and voice and talk and meet and do the things case managers are supposed to do To build relationships so that we can what help our families because they're the most important thing.

1 46:55

That's great. Thank

5 46:56

you

1 46:57

Yeah. Thanks, Tonya. Thanks, Regina. We've got a couple of questions from both Denise. Hi, Denise and Melissa, about standards. And so Kim, I think this one might go to you. It's about how did you set, standards for training and wages? How does the agencies respond to that? And how did you find additional funding to provide for like higher wages? Fair

4 47:25

questions. I know we're running short, so I'm going to answer in brevity. Yeah, how did we find more money? There was no more money. So we knew from day one how much money we would have each year in this levy. So it meant that there would be fewer case managers. leading in one area meant that there would be a change in the other area. there was a little resistance from a couple of the agencies because our minimum standard would put a couple of the case managers at higher wages than other fellow case managers. And we just said, Hey, we did this randomized control trial. We had this study. We are testing a theory that, the case managers that are leaving consistently may have been a part of the null result. And so we're going to do what it takes to have low to no case manager turnover and raising the wages and has created that. We have virtually, as Tonya said, we really don't have any turnover. People come and they stay, they've been there the entire iteration of 2. 0. So that is fantastic. The training, it's nice because we started with high intensity training at the beginning and we were so used to having to consistently train in UPIPI. Again, case managers aren't leaving, so when we ramped up at the beginning, we've been able to slowly taper off a little bit of some of the training, which has been wonderful. there was a third part of that, Carrie, do you remember the third part of that?

1 48:54

Paying wages, how did agencies react? yeah, that's about it. I think you covered it all.

4 49:00

The agencies agreed over time, and guess what? The agencies have benefited because turnover costs them too. And so the fact that we're not losing case managers by raising wages, and lowering caseload, everybody thought if you lower the caseload, you're going to serve fewer people. No, because we're constantly assessing and monitoring when they have achieved at certain levels, it's time to exit them. So we haven't actually found that to be true either. So a lot of the conjecture and concerns actually, have eroded over time.

1 49:31

Yeah. And we're scheduled till 1. 15, so we've got a little bit of time. Tanya, did you want to add anything on to Kim's response?

4 49:43

just as a case manager, I think, as I stated before, the lower case low helped me because I could really work intensively with the clients and, explore other needs that they need. Thank you, Tanya. Rent is only one thing. There's mental wellness that needs to be happening as well, right? There's other things that these families need. Childcare. Sometimes I spend all of my case management with a family, just helping them find jobs. We spend the whole time filling out applications, getting their resumes together. it just gave me a lot more time to be very intentional when I'm working with them.

1 50:26

Yeah. You can't care for others unless you're cared for yourself, right? No. Melissa, do you want, you have a question about, maybe I'll just, I'll read it here. Were results by agency studied to see if some were implementing programming more effectively? Kim, do you want to talk a little bit about, so we're looking at the whole program in this study, but. How do you look at agency by agency and share learnings between agencies and like, how does that all work?

4 51:00

Let's say that was a part of our work. we have performance measurement evaluators that do that work continuously. So we had that vantage point where we would see results based on agency, but the overall report. David, I don't remember it breaking down by agency specifically to give us different results. Along the journey we had that, so we could work with agencies that maybe, the data just showed you're not exiling anybody. So in order to, really help us with the results of this study, we need you to do some of the key data points. data work that has to be done in order to produce a result for this study. David, I can't remember, did your final report break out by agency?

51:43

Yeah, I think in the end we didn't want to identify particular agencies and say here's what I'm doing, but we did, one thing, some things we were able to do was, for instance, We could see that there was some differences across agencies, for instance, in terms of like on average, what the program looks like. And like one of the things we were able to do, for instance, was just to look at, okay, for some agencies that look like they're able to spend more time with clients on average, what do outcomes look like there versus an agency where they're on average, maybe that bigger caseloads, they're able to spend less time on average with somebody. And so one of the things we saw, for instance, was that. a really thin version of reaction to our results might be case management bad, right? But if case management is bad, then what would you expect if people get more case management they would look worse? But we didn't see like agencies that were spending more time with clients. Those clients weren't doing worse. They looked actually better on some dimensions. so I think that's a little bit outside of the original randomized control trial, but it's one of the things that we could use where like Kim and others, Have these, ideas that they see from watching the program of saying, maybe there are things about. the intensity of case management about caseloads that's important. then there's something we can test with the data to see. Oh, no, actually, yeah, it does look like things are like it's a benign thing when people have, more time with their case managers. Not a bad thing, right? And so I think there were ways like that we could go back and forth between the data and between what people who are closer to the program than me could see. And in some places those supported each other like with looking at caseloads.

1 53:08

What I love about these real research practice partnerships is that we're bringing insights from the programming side and feeding questions that we have to our research partner like David who can then work with all the data that's there from the study to answer questions that we didn't even know we had when we started, right? And that's been so valuable in all the work we've done with David and A key thing that I look for in a research partner is someone who can really work with us to answer those kinds of questions

4 53:42

I want to add a little bit to that too.

1 53:44

Sure

4 53:45

There's a plus to that in another way too You know as we're sitting around grappling with some of the results of this and I might throw out I think it could be this David would say, actually, if you dig into the data in this area, which I didn't understand the, some of the complexities of the data, it was, outside of my realm. But he could say, actually, interesting thought, Kim, but no, the data doesn't actually support that. So it also allowed us to focus inward on so what are the areas we really should look at? And, so again, it was that partnership and then the ability to banter and dialogue together in that great relationship where David could come back to me and say, nice question. Maybe not so much. The data kind of supports something else.

54:31

Yeah, the thing about that was really great from an academics point of view is the thing that academics do sometimes that we do that's not great is we'll run a study, we have these sort of headline numbers, we say, we know that these numbers are accurate. And then we go into storytelling mode of here's why they happened, right? And that storytelling is sometimes not super grounded in, in, in reality. And so one of the great things about the partnership is then just like Kim was saying that, the data can Create some boundaries on what are reasonable explanations for her. Their expertise as our partners also does the same thing for me, where it says, okay, I have these things that I know to be true, but then my, also my storytelling around that has to be. You know disciplined by the reality that Ken and Tonya and others see When they're operating more closely to the program than I am So that's the thing I like about is that there's yeah, we both benefit from each other's expertise

4 55:18

And I just wanted to add. just a little bit that I love research I think it helps us. look at other things how we can evolve and change and help our families But I also want to say that until the housing system itself changes, meaning the increase in rent when a client's salary doesn't increase, right? I talked to a landlord that raised the rent 1, 000 and she said I can because you guys didn't vote not to put a cap on it. And I thought, my goodness, 1, 000? as a case manager in the homes, dealing with my clients, there's a bigger picture as well, when our clients are facing homelessness. so I just wanted to add that as well.

Concluding Thoughts

1 56:19

Yeah, there's such huge macro factors driving inequality in our country, right? And we're trying to tackle them kind of little piece by little piece, right? And, we need to do all we can do as best we can do as quickly as we can do it Which is what a lot of this work is about so we're we are coming up on time I did hi. Hi quinn. It's great to see you and results from for america folks here You're responsible for this whole partnership. I'll just tell you that and quinn was asking a question about You know just the resistance by leadership to the potential consequences of negative results and I think We've really you know a couple of things. So that's a really important consideration. We are really fortunate in king county To have a lot of data driven leadership here leadership who is really encouraging learning and Exploration and Really focused on, the outcomes that we're trying to create with our communities. And, that's from King County executive Dow Constantine to our entire county council, to all of our, department leaders. we really appreciate that support and, it take, it does take a lot of patience and working through these questions and taking them seriously. You have to invest in that kind of setup and make sure everybody's on board which is not easy, but the results are worth it So we have to invest that time. So maybe vincent you and I can start to wrap things up here i'd just be really curious to hear Your reactions to this conversation. What were your takeaways from our time with our panelists here?

2 58:20

Yes, this is a really, oh, sorry, there's a bit of an echo. This is a really fantastic conversation. So thank you Tanya, David, and Kim for sharing your perspective, your insights, and some of the key takeaways. I think a couple of things out at me. The first thing is the importance of the partnership itself, and that seems really foundational to the research project, the interpretation of the results. communication of those results and really acting upon that results. And it became very evident from this conversation how much you all invested in making that strong partnership a priority and how critical that strong partnership was to making this research project a success and processing some of these really unexpected results. I think the other key takeaway is how unexpected results and null results can actually be a real blessing in disguise. And you mentioned earlier at the beginning of the conversation, Kim, that the null result in some ways may have been the best result. And although it is difficult to grapple with findings that can be really initially disappointing, where you have invested a lot of time and energy, and everybody wants They're a program to work and they want the impact evaluation to validate that this approach that you're taking is a really effective strategy. If we ultimately are hoping to help folks within our community, we have to spend the time and energy to really figure out what ultimately is making a difference. And the null results can really help us change course and move away from a model that might be less effective to make real changes that. are on the ground making an impact and being able to hear how you have been able to take these null results, make real meaningful changes to the program design. Tanya, here in your perspective about how that's actually impacted the caseload and the types of services that you provide as a case manager. It's a really powerful and inspiring example of how we can leverage unexpected findings from rigorous research, null results from rigorous research, to ultimately better serve the communities that we all collectively are trying to help. And so I think those are two of my key takeaways. Carrie, I'd love to hear your perspective on some of the key takeaways from today's conversation.

1 1:00:40

Yeah. it really underscores for me, something that, that I do really believe in that is that, a study isn't about a single result. And often we like think about evaluation as, Oh, we're going to do this study and it's going to give us a number that's going to give us the answer. And then we're done. And we really, I think, have to approach this work around evaluation and evidence building as continuous learning. And that's what I love about what this team did, right? we didn't just approach it as a study and we got an answer and then we said, Oh, gosh, this isn't working. We're done here. We're walking away. We really used it as the vehicle to dig in, learn more, make change, make things better. And and we're not done, right? We're still learning. We're still continuing on. We're still digging in deeper. We're still There's so many great questions in the chat we have yet to answer right and we're all part of this kind of learning community across the united states kim I really appreciated you lifting that point up that you know This work helped us and david helped us to connect With others doing this work so that we can You know, we're all trying to do we all face the same problems in our communities all around the country We don't have to do this by ourselves We can learn from each other. And so that's what this evidence matters series is all about. It's um, learning from each other so I really appreciate everyone being here I said at the outset we had a goal to You Double our attendance and we more than busted through that So thank you all for being here and sharing with us if you had a question. We didn't get to wanted more contact information You can feel free to email me my emails in the chat. I can answer questions get you connected to other people We would love to hear from you, and we really appreciate everyone for jumping in. Thank you to our panelists, David, Tanya, Kim, love you. Vincent. Thank you having

4 1:02:52

me. Vincent. Thank

1 1:02:53

you for having me. Yeah. Vincent. Vincent, you're fabulous co-host. We'll be back. Thanks, Vincent. Ask you to join us again.

2 1:02:59

Happy to be here anytime. Thanks,

1 1:03:01

Carrie. All right. Thanks everybody.