The ThinkND Podcast
The ThinkND Podcast
Evidence Matters, Part 11: 10 Lessons for Building Government and Research Partnerships That Matter
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Episode Topic: 10 Lessons for Building Government and Research Partnerships That Matter
Bridge the gap between social service leadership and rigorous scientific research to move people out of poverty permanently. Join Notre Dame’s LEO and JPAL North America for a discussion on building high-impact partnerships. Learn how data-driven collaboration transforms lives and empowers government leaders to create lasting, equitable impact.
Featured Speakers:
- Vincent Quan, J-PAL North America
- Louise Geraghty, J-PAL North America
- Carrie Cihak, King County, Washington
- Patrick Turner, Notre Dame University
Read this episode's recap over on the University of Notre Dame's open online learning community platform, ThinkND: https://go.nd.edu/501377.
This podcast is a part of the ThinkND Series titled Evidence Matters.
Thanks for listening! The ThinkND Podcast is brought to you by ThinkND, the University of Notre Dame's online learning community. We connect you with videos, podcasts, articles, courses, and other resources to inspire minds and spark conversations on topics that matter to you — everything from faith and politics, to science, technology, and your career.
- Learn more about ThinkND and register for upcoming live events at think.nd.edu.
- Join our LinkedIn community for updates, episode clips, and more.
Welcome to Evidence Matters
Speaker 2Good afternoon again, everyone, and a big welcome to our Evidence Matters series. Uh, my name is Regina Giesecke. I am the dissemination associate here at the Wilson Sheehan Lab for Economic Opportunities, otherwise known as LEO at Notre Dame. At LEO, we match top researchers with passionate leaders in the social service sector to conduct impact evaluations that identify innovative, effective, and scalable programs that move people out of poverty permanently. The stakes are high, as we know. In the US today, over forty million people live in poverty, and while roughly one trillion dollars is spent annually to fight poverty, less than one percent is spent on evidence-based programs. So LEO seeks to bridge that gap by overlaying rigorous research, including randomized control trials, onto existing interventions and programs, hoping to isolate what truly works to move people out of poverty. So myself and my colleagues are super excited to have you all here today, and I wanna turn it over now to Vincent Kwan to introduce himself and keep us moving.
Speaker 3All right. Thanks very much, Regina. I'm Vincent Kwan. I am the acting executive director of J-PAL North America. I'm very excited to see both familiar faces and new faces in the room today. For those of you who don't know, J-PAL stands for the Abdul Latif Jameel Poverty Action Lab, and we are a research and policy center that's based within MIT. And our whole focus is simple yet ambitious, which is to reduce poverty through the use of rigorous scientific research to understand what policies and programs measurably improve the lives of people that are experiencing poverty. And very excited for the conversation that we have today. And with that, I will hand it over to Kari to introduce herself.
Speaker 4Hi, everybody. I'm Kari Chihock, and I've spent my career advancing equitable community outcomes in the public sector. Um, many of you, know of me through my most recent role in King County as the evidence and impact officer. And I recently left King County, and I'm now, working on my own. And, uh, I was struggling a little bit to figure out how I was gonna introduce myself given that. But, um, what I realized is, you know, my passion for the role of the public sector has not changed. Um, that's a big part of this Evidence Matters series. And I'm s- still just as passionate about bringing governments together with researchers and communities to support them in creating lasting, equitable impact. So I'm really excited to bring my experience in local government here to our Evidence Matters webinar series. and I'll turn it back to you, Vincent.
Speaker 3All right. Thank you very much, Kari. So I know some of you have joined previous Evidence Matters, panels before or series before, but we thought it'd be helpful to briefly introduce what Evidence Matters is all about. So in short, Evidence Matters is a collaborative learning series that brings together folks from government, researchers, and the community on what it takes to promote rigorous learning to advance equitable, evidence-based outcomes. It started off as a partnership in twenty twenty-one between LEO and King County. They brought J-PAL North America into the fold a couple of years ago, and each year we've been able to host different learning series focused on a range of topics that pertain to evidence-based policymaking and ultimately why evidence matters. And so, Regina, do you wanna share a bit more about what LEO is focused on with Evidence Matters?
Speaker 2Yeah. I think, um, a major part of our, our work in this is, motivator is to, to share knowledge. Um, we don't want academic research to sit on a shelf, um, but to be able to get out and used in, in communities that, that need information about what works to, to move people out of poverty. so One of the major exciting pieces of, of Evidence Matters is building this community and, and sharing knowledge with each other. not sort of gatekeeping what works and making sure that those across the country, can, can learn from, from our learnings and learnings we've developed with our partners.
Speaker 3I think piggybacking on that point, I think all of us know that evidence-based policymaking, even though the term sounds simple, is actually quite messy and quite complicated. So Evidence Matters really is a forum for us to dig into the messy in-between real-world steps of building research partnerships that ultimately hold up and change practice. And we know that often when we focus just on the individual study, we end up paying attention exclusively to the top-line result from a study. But this forum really is an opportunity for us to dig into all those nitty-gritty process details that we know are equally instrumental in getting valuable research into the field and ultimately to, to inform policy.
Speaker 4Yeah. Yeah. I totally, totally relate to that, Vincent. And as well, you know, I think often those of us who are doing this work in government sometimes are kind of doing it in isolation. Um, it can feel like we don't have a- another partner in our government to do the work. So part of what we're also trying to do with Evidence Matters is build that network, connecting, government, you know, government staff in different places who are doing similar work, and also connecting you all with researchers, practitioners, and other people so we're not doing it alone, that we really feel a sense of community. And so we're really gonna hope to draw you out, uh, in this conversation here today.
1Absolutely.
Speaker 2So, um, Vincent will moderate a conversation, amongst our panelists today, Carrie, Patrick, and Louise, on 10 lessons learned about how governments can foster research partnerships that work, that have impact. so I'd love to turn it now to, um, Louise and then Patrick to introduce themselves, and then I'll hand it over to Vincent to, to start our conversation. So Louise, if you wouldn't mind.
Speaker 5thank you so much. my name is Louise Gerrity. I'm the government partnerships lead at J-PAL North America. and my work focuses on connecting, um, our network of affiliated academic researchers with state and local governments all across the US, to generate new evidence, do new research projects, and then, to find opportunities to use the evidence, from, from research that we've generated. so I think we'll get more into the specifics about what that entails, how we approach that at J-PAL North America, but really happy to, to be here with all of you today.
1Great.
Speaker 6And, uh, my name is Patrick Turner. I'm an assistant professor of economics at the University of Notre Dame. For the last eight years, I've been working at LEO, uh, building partnerships with state and local governments, with local nonprofit providers to understand what works in supporting the economic livelihood of low-income adults in the United States. I'm so excited to dig in and kinda share my experience of, collaborating between, uh, academics and, and, uh, practitioners out in the field.
Speaker 3Great. Thank you, Louise. Thank you, Patrick. Very excited for today's conversation. And so for today, we really will be focused on ten lessons learned about how governments can foster research partnerships that produce real-world impact. And the focus today will be for government staff who are seeking high-quality partnerships with researchers. But with that said, we also think that many of the researchers in the audience will gain from hearing directly from governments about their perspectives, their incentives, their challenges, and what opportunities they see to form these types of, partnerships. And so we really encourage a lively discussion today and also encourage folks in the audience to share questions directly in the Zoom chat, um, and we will try to get to as many of these as possible. We wanna acknowledge that this conversation actually stemmed from a different session that Carrie and Louise had hosted at Results for America's, uh, Government Results Network. And we thought it was such a lively conversation and so relevant to many folks who are working in government that we decided it would be a perfect conversation topic for, uh, for Evidence Matters as well. And Carrie, do you wanna share a bit more about, the Results Network for RFA?
Speaker 4Yeah. really encourage anyone on the call here who, is with, with working for a government to join the Results Network. It's a community and a resource hub that's specific for people in government to support you in using data and evidence to tackle your community's, um, biggest challenges. So it's a great place to connect with others who are doing the work. And, we've got a link in the chat for you to sign up.
Speaker 3Fantastic. Thank you, Carrie. And plugging again, RFA. They are a great, uh, resource and, and great organization to check out. Okay. So we wanna keep this as interactive as possible. So what we are going to do now is actually offer for folks in the audience to spend a few minutes reading through the ten lessons that we've compiled and then think about which one of these lessons you'd like to most focus our time together on. And in a few minutes, we will ask folks to fill out a poll ranking the top lesson you wanna focus on, and we'll make sure that the panelists today share some key takeaways related to that, the topics. So it seems like folks are really interested in hearing more about how to design research projects that ultimately center a government's policy objectives, really helping to inform an actionable policy decision. So Carrie, maybe I can start with you. As a person who's worked in government for a long time playing an evaluation role, how do you think about designing and partnering on research projects that ultimately answer the question that the government is interested in answering? And then Patrick and Louise, afterwards, I'd love to hear your perspectives as folks that work for academic research centers about how you design research studies to meet the policy objectives of your government partners. But Carrie, why don't we start with you?
Speaker 4Yeah, happy to. So I do think, this is one of the most important lessons on the list. if you're not really staying true, to answering a question that your government needs information on to move decision-making forward, you can spend a lot of time in a partnership, that's not producing useful results for you. And, you know, it's, it's really easy to get kind of moved off track. we have lots and lots of questions as government staff about how our, you know, work is effective or not, and researchers have even more questions than we do. And so we really want to try to, focus in on the kind of key, key piece of, key question about a program that you're, you're implementing that is going to drive a next step in decision-making. Now, a lot of times, I think we s- we kind of s- start or become interested in doing a research project as a government without having that question really well formulated, and that's okay. there are resources out there to help with that. I will put a plug in for J-PAL, and some of the technical assistance and funding they've actually provided, that really helps governments before they get into a research partnership to really think through what's the objective? What am I trying to answer? What is my theory of change? You wanna know what outcome you're trying to affect, w- what your intervention is, and then how are you getting from A to, A to B in moving from what you're doing to the results that you're seeking. And that's really gonna help you formulate what that research question is. And you can then do that also in partnership with a researcher. You just wanna make sure that As you're going through those conversations, you're always coming back to, you know, taking a step back, get up on the balcony, try to make sure that the question you're tackling is really related to, uh, what, what you're trying to achieve.
Speaker 3That, that's really helpful, Carrie. Thank you for sharing that perspective. Patrick, love to hear from you. As a person who's worked closely with governments on these types of research projects, how do you think about ensuring that a research project is actually informing concrete policy decisions? And what do you like to see from government partners to make sure that you also are understanding what their policy objectives are?
Speaker 6Yeah, I think that's great. Uh, I mean, a lot of the times when we're starting kind of the early stage process of, uh, working with the new research partners, you know, they have a program that they're running in mind, and they wanna know what the impact of that particular program is. It's helpful. I, I liked how Carrie was talking about how, you know, having questions that where the answer provides like an actionable step for the organization, right? Is it, you know, what would, what would it mean for this program to demonstrate success? And then targeting the research question towards that. you know, as a, as a researcher, I'm interested in kinda answering novel questions that might extend beyond the particular life of the organization. What can we say more broadly about how a particular program works or how, uh, individuals interact with that type of a program on a broader sense? but where there's, like, really good alignment between the research objectives and the agenda that the organization has, what answer would provide them with good actionable next steps, and then how that aligns with what we do or do not know from kind of the academic literature. That provides a really good, launching-off point for, for doing that. oftentimes, the partners I work with, you know, they have an idea of what an outcome that they might care about is. And, the researcher, right, if it's a good partnership, should have a good sense of, like, what's the best way of trying to measure that, either through administrative data, through survey data. And so it really is this co-creative process of trying to find overlap on where there's research interest on the question and then working towards how can we best, overlay research evidence on top of that.
Speaker 3That's great. Thank you, Patrick. Uh, I think your point around co-creation speaks to, I think, the importance of transparency whenever government and researchers are working with one another, being transparent about what the incentives are, being transparent about what some of the challenges are, and overall, what the various policy and research objectives are. Louise, I wanna hand it over to you. You're a person who thinks a lot about how to match the incentives between academic researchers within the J-PAL network to those within our government partner network. So how does J-PAL think about merging these partnerships and ensuring that- Both parties that are coming to the table, whether or not you're a researcher or a government partner, are really approaching it o- on equal footing with transparency and with respect, to one another
Speaker 5so I think that we design, our opportunities for governments to engage
Lesson One Policy Drives Research
Speaker 5that really start-- use the government priorities as a starting point for designing a research project. so, uh, some of the opportunities that we have, I think still available now are through our learning labs for government, which offer a combination of, capacity building training, around evidence and evaluation for government staff members, and then eventually matchmaking with researchers within our network, who share similar research priorities to what our government partners are, are looking to conduct evaluations on. so, our staff provide a combination of technical supports, including the theory of change, which Kari mentioned, uh, as well as a number of other principles related to designing, um, an impact evaluation. and governments are able to use that to kind of further develop their research questions and the logistical plan for implementation. we are-- uh, and then we eventually look, look at our network, think about who is kind of interested in similar questions to what this government is, is interested in doing, and, um, and, and find folks who are, are able to, to work together is, is the ultimate goal. we're also able to do that on more of, of an ad hoc basis. Um, we'll occasionally get requests, from governments, um, who've worked with us before in this way, um, on, on other topics, um, and we're able to provide some matchmaking support, through, other, other mechanisms as well.
Speaker 3Great. Thank you, Louise. Kari, one thing that you have shared with me before is how folks in government that are working on these research projects should really consider themselves a core and necessary part of the research team, and that folks within government bring a perspective that is so foundational and instrumental to the success of a research project. Can you share a bit more about what you mean when you say that?
Speaker 4Yeah, absolutely. I think a lot of times as government staff, we look to researchers as the experts in the room. And researchers certainly do bring, uh, certain kinds of expertise. Uh, but I really want to encourage, you know, everyone who is here from government to think of themselves as researchers too. It's part of human nature to be problem solvers, and that's really what research is all about. And you hold a ton of expertise that is really critical To any research project being successful. So you have really deep program expertise, you understand decision-making and other processes and how information gets used within your organization. You know the pitfalls that you have to avoid and what's feasible and what's not. You wouldn't be in your position in government if you didn't really know how to kind of work the organization to take information and, and turn it into action. And the most important thing that you're bringing is that you really know what the most important questions are. And, um, that's kind of the most important part of the research process, which is what, what we started, started out talking about. So again, really think of yourself as a researcher. You-- This is-- You know, to be a successful research partnership, you can't just come to a research organization or a researcher's and say, "Here's the question I have," expect them to go away for six months and come back with an answer. It doesn't work that way at all. You have to be in the game. You have to feel like you own the research, because you do, and that you're really part of the team.
Speaker 3Definitely wanna double-click on that, Carrie. And Patrick and Louise, would love to hear any concrete examples you have of research partnerships you both have led with government, where having the government partner really being an equal player and truly a researcher on the team helped shape some critical research design considerations you had or, or fundamentally how you decided to tackle the question.
Speaker 6Yeah. I, I would just, tackle kind of like two pieces there. I w- Carrie, exactly right. One other piece that I think the, the partner has expertise in that the researchers don't is the populations that they're serving, right? They're, they're working closely in line with these, folks. And so whenever we're designing a research study, uh, we take very seriously what the, uh... how that, uh, experience, how, how the, uh, study participants will experience their interaction with the research process. And, and we really heavily rely on our partners for helping us understand, you know, con- how consent forms, how, uh, any sort of randomization process or other data collection activities will be experienced by partners, uh, by the study participants when we're designing these things. a, a couple, uh, tangible examples. So, this isn't with a government partner, but with one of my nonprofit partners. you know, they were working in a, a community in North Nashville, Tennessee. Uh, the primary study participant in that population is a Black woman, and the two researchers on the team that were helping out with this partnership, uh, are, are kind of white men. And so, you know, we heavily relied on our research partners for designing consent forms, for how we were going to describe the study Uh, et cetera. And, and what are the relevant questions that we care about for this population? we relied on them when thinking about what questions to ask in the follow-up survey that would measure what we were hoping to, to, to get at with the research study, um, because of their intimate knowledge of what the program was. another example with one of our research partners, um, you know, we were in early stages of designing a randomized control trial of a subsidized employment program, and one of our partners in the Larimore County Workforce Center had been running the program already for a year and a half. And so we heavily relied on, uh, his programmatic experience of that program for thinking about, um, how could we design baseline survey questions to try to tease out mechanisms to try to understand who might benefit most from this program, and how could we measure that, uh, as a part of this research study. Uh, and then, you know, ex-post, when we go to analyze the results, we can go back and rely on their experience on the ground for trying to understand a more nuanced perspective when we're looking at our results afterwards.
Speaker 4Vincent, I'd love to jump on that point that, Patrick made about having diversity on your team. it's really important that you have, uh, folks on your team who have knowledge, and connection and lived experience with relation to the community that you are working to serve. That, perspective is invaluable, and we can talk a little bit more about that probably if we tackle some of the, some of the, the, the point about, community connections. But I just wanted to underscore that, that point about having a diverse team.
Speaker 3Yes, Carrie, definitely let's put a pin on that too. I'm seeing a comment here from Mary that very much relates to the piece about community perspectives and making sure that they are represented in a research project if the aim is ultimately to improve lives of people experiencing poverty. Louise, do you have any other examples that you would like to share from the J-PAL repository?
Speaker 5Sure. so I'm thinking of, a project related to housing stability where a, a county government and the research team were interested in evaluating the impact of eviction prevention workshops. and there were a number of questions that came up related to the overall program design and how to best reach the, the community, which, the government partner as well as their nonprofit partners were able to answer really effectively. The research team, of course, never would have, would have been able to do that just by themselves. and then I think also, some key pieces related to timing of the workshops, which again, kind of a program design question, but also related to data collection. Sort of what are, what are the best opportunities for data collection? how can we reach the community? kind of what are, what are gonna be the most, what do we think the most effective channels will be, and, and the most effective approaches, to be able to get the, the data we need to answer our research questions.
Speaker 3Those are really helpful examples. Thank you, Louise. And I think the, the key point here really is that government leaders and government staff are equal partners in this research process, and your perspectives about the program, your insights into what questions are most important to answer are really foundational to the success of any research project. So we certainly encourage folks to be really forthcoming and to lean into your own expertise and to be really confident in your, your own expertise. And we could not emphasize that, that more. So switching gears, I know the second topic that was upvoted was the one about null results or negative results. And one thing that we have often heard from our government and nonprofit partners is that it can be really scary to participate in an evaluation if the evaluation results end up showing that there's a null result or a negative result. I think every person who works on a program, whether or not you're in government or nonprofit or philanthropy, you really believe in that program, and you're working on that program because you really believe it will make a difference in the lives of people that you are trying to serve. So this fear of null and negative results is very, very understandable. Carrie, maybe starting off with you as a person who actually worked on many evaluations from the government perspective, how have you thought about null results, and wha- what are some of your responses to those types of fears that people have?
Speaker 4Yes. Yeah. Those fears are totally understandable. And I also wanna encourage everybody to think of null results as a gift. They are the most important results that we can have because the last thing that we wanna do as a government or a service provider is be providing a service that isn't working or even worse, might be producing harm. And so typically, you know, a lot of the fear comes from thinking that you're evaluating a program or an intervention as a whole, and when that null result comes in, that means that program goes away. And we really have to get beyond that zero one thinking. What we need to do when we get a null result is really dig in and understand, okay, what's not working? If we went into a well-designed research project, we had a really strong theory of change that we were testing. And, you know, if we've done work in the community and we've really kind of tested out that theory of change, we should have some belief that that theory of change i- is valid. And so there may be some other, Thing that's happening, that is, creating a null result that we can easily fix within that program and within that s- existing theory of change. So I'll give you a quick example. in a large northwestern county where I used to work, we, we were running an, a homelessness eviction, uh, preve-- a homelessness prevention program. And what we really wanted to test was whether case management in addition to rapid financial assessment would help prevent people from falling into homelessness. And we wanted to test that case management piece because it's very expensive to provide, it's very difficult to implement. And so if it's not having an effect, then we didn't want to invest in that. so we, we did an, an RCT on this, and we actually found, that the case management was not having an impact, and in fact, it was doing some harm in some cases. Now, what was wonderful about, getting that result is we had a research team. It was a research team at LEO, and I believe that J-PAL had funded part of this project. we, we really dug into trying to understand what was going on, and it-- and we were able to uncover the factors that were leading to that null result. So one factor was that we had a lot of turnover in case managers, and so families were not experiencing consistent provision of services from those folks. In addition, we had set up the processes in such a way that the case management was really kind of standing as like a, a gatekeeper to people accessing the financial assistance. And so when we solved those issues, we were able to j- a-address the underlying concerns and then go back and, and, uh, create a program that was actually very effective in, in keeping people out of homelessness. So it's not a zero one, you know, this program works, this program doesn't. It's about really unpacking, doing the deep learning to understand what's going on with how you're implementing something and then making the necessary changes so that it's actually working for people.
Speaker 3Yeah, I think that's exactly right. I think at J-PAL, we often encourage folks not to frame it around does it work or does it not work, but rather focus on what can we learn about how the program is working, and really treating in evaluation, whether or not it's an RCT or another form of impact evaluation, as a step in the learning journey to understand How is this program actually working in, in practice? And so Patrick and Louise, I think with that, I'd love to turn it over to you to share a bit more about what you think government partners can learn in the process of doing an evaluation. To Carrie's point earlier, sometimes, you know, we can so focus on that final top-line result and over-anchor on a null result or a negative result or a positive result. But we actually know that some of the best learnings are from the process of partnering on an evaluation. So I'd love to have the two of you share a bit about what you think governments can learn during that process of an impact evaluation.
Speaker 6Yeah, I would just add, you know, along the... You know, as, as a process of setting up a research evaluation, sometimes we need to track particular things in data. And so some of my best research partnerships with governments, during the process of standing up the RCT, they've
Embracing Null Results
Speaker 6redesigned or rebuilt or added fields to their already great data systems to allow us to track things along the way. And so I think that's one benefit that comes with research, is thinking about, how do they, track data about study, uh, study participants or their, their program participants more generally. And that allows them to kind of have better, richer data going forward. one of the things, lesson seven, right, is this nurturing continuous and rigorous learning. Uh, one of the things I've seen along the way is, you know, maybe we're trying to measure something a year or two years after somebody participated in the program. But as a process of the RCT, we kind of set up a real-time tracking of how program implementation is going. Um, maybe it's a emergency rental assistance program, and we can track, okay, from when somebody calls in, how quickly are they getting their emergency rental assistance? Uh, and if that's not being delivered in the, uh, the way in which the program was intended, we can catch those kind of learnings earlier. How can we better implement what's already going on so we're testing the best version, uh, of the program?
1That's great. And Louise, anything to add there?
Speaker 5Yeah. so, uh, really also wanna appreciate Anne's comment here in the chat about what's working for whom and under what circumstances. and would echo all of Carrie's points about null results being an opportunity to revisit your theory of change. if you didn't get the outcome that you were expecting, after putting in, you know, all this sort of strategic work, why, why is that? could it be that, you need to think about targeting a slightly different population, make changes to the intervention to really achieve the, the outcome that you were hoping for? I think this is also an opportunity to continue engage governments, as part of the research team, in thinking about, you know, if, if we've-- if we have some ideas about what needs to be adjusted to the theory of change how can we then think about testing some of those ideas, in, in kind of subsequent research, which hopefully you're able to do, um, in establishing a good research partnership. so I'm thinking, Vincent, of, of the messaging around, J-PAL North America's evaluation of the healthcare hotspotting program, uh, in, in Camden, New Jersey, which was led by our scientific director, Amy Finkelstein. they didn't get the results that they were expecting, um, as, as a result of their intervention, and have gone on to do some follow-up work to kind of unpack, you know, what, what are some of the, the reasons for that.
Speaker 3That's really helpful. I think one other thing that I'll mention is the importance and value of long-term research partnerships to be able to dig into these questions further. I know that King County and LEO have had a very long-standing partnership where it's not just a one-and-done evaluation, but really a continuous learning journey where they can continue tackling some of these questions together. For example, if you found null results the first time around, why was that the case? Or is this intervention working more for a specific population than another population? Or if you're thinking about expanding a given program or adding another programmatic component, how might that ultimately affect the results that we see? And we do think that these types of long-term partnerships are so valuable also in part because it takes a lot of time to set these types of research partnerships up. There's a lot of rapport building and alignment of incentives that needs to happen on the front end. And when you find a good match, that's something that we actively encourage folks to really nurture and can continue cultivating so it can be a long-term research partnership to answer mutually beneficial, uh, questions. Yes, and thank you, Lena, for sharing some of those, uh, additional resources there. Okay. on that point, I know, Carrie, one of your key lessons is that timing is not everything, and that especially with these types of research partnerships, it can be very challenging to actually adhere to a specific timeline. Policy windows might change, policy challenges might change. Research can often take longer than expected. Navigating data use agreements can take a very long time. Tell us a little bit more about what you mean when you say timing is not everything.
Speaker 4Yeah. undertaking, a real research project that's a partnership does involve a large investment of time, and even when you start with a seemingly quite simple question, you should expect it's gonna take at least twice as long as you think it will. That's just the nature of it, and that's why some of these other points about learning along the way- fostering long-term partnerships are, are really, really important. I saw Courtney's comment in the chat about how often, you know, elected leadership or others are really looking to do research to sort of say, "This thing works. It's solving homelessness," or, "It's moving people out of poverty." I understand the desire for that and, you know, to invest the time into the kinds of research partnerships we're talking about here. We really have to have, deeper aims than that. If we're looking to enter into a research project just to kind of prove what we're doing is working, and we wanna do that by a next legislative body vote, we're probably not going to have a very successful research partnership. so I think we really have to, be cognizant of that, understand that, you know, the goal in these kinds of partnerships is really to do that deep learning that's going to create change. It's not about proving something existing already works.
Speaker 3Yes. That's very helpful framing, and I, I think something that we definitely would second at J-PAL as well. I, I know that timing can sometimes be considered a barrier. Another barrier that we also hear from our government partners is related to procurement, contracting, securing the necessary resources to get an evaluation off the ground. Louise and Kari, so the next question is for the both of you. Kari, as a person who used to work within government, how have you been able to address some of those challenges related to contracting and procurement or securing resources to get an evaluation off the ground? And then Louise, I'd love to also have you share more about the role that organizations like J-PAL can play in funding different research partnerships. So Kari, let's start off with you and then hand it to you. Yeah.
Speaker 4This is a very boring and very critically important question. so, you know, our typical government contracting processes aren't really set up for the kinds of partnerships that we're talking about here today, where we're really trying to... You know, often when w- often a government contracting process is about the government stepping back, understanding exactly what the scope of work is, releasing an RFP, doing a competitive process. We own all the intellectual property, et cetera and so forth. In the kinds of research partnerships we're talking about here, we often don't really know what the scope of the project is until we're in some significant conversations with researchers. And Finding the right fit between, you know, your question as government and who the research partner you want to work with is not something that's very conducive to doing a competit- a typical competitive procurement process. And that's why we really, you know, in King County, we really relied on J-PAL, who is essentially, you know, you have a very large network of researchers, and you're essentially kind of screening researchers and creating that competitive process on our behalf and then recommending matches for us. I think it's really, you know, the, the typical barriers that I found in, you know, trying to develop the, the financial or the contracting mechanisms, a big one is around intellectual property. This is the thing that researchers care about most, is their intellectual property. As a government, we typically wanna own the intellectual property related to a project. I think we need to think about this in a different way as government. It's part of what we are trying to purchase from a researcher in a project like this, is their independence. And so I think we can make the case within government that us not actually owning the intellectual property is also in the government's interests. But that's a barrier that can be hard to get through, in our processes. So one of the things that I have found is that it's often helpful if you can, first of all, start by getting matched through, you know, an organization like J-PAL to a researcher, and then, if you can develop a scope of work that you can go out for grant funding for. Governments are very used to developing contracts for grant funding that adopt the conditions of the grant. And in these kinds of research projects, that usually includes all of the IP provisions and other things like that. So that's been a really helpful way I found to get started. Also, don't try to reinvent the
Timing And Policy Windows
Speaker 4wheel. a lot of governments have, have started these kinds of projects. We've figured out how to get through the intellectual property questions, the indemnification questions, the data-sharing questions. There are examples out there that you can lean on. Um, so, you know, reach out to your colleagues. You can reach out to me, other colleagues you know through, the Results Network at RFA for some guidance on, on working through that. It is a big barrier, and it does take some persistence to work through it.
Speaker 3Very helpful, Carrie. Uh, Louise, so over to you to talk about how some of these projects get funded.
Speaker 5Sure. so, uh, I can talk a little bit about our model and the way that we approach funding research projects. so we raise philanthropic funding, to be able to support, the, the work that the researchers in our network do and that our government partners are working on together. so, we run, uh, uh, RFPs through our research funding initiatives, which our researchers, kind of lead the, the charge on developing the applications for in partnership, um, with the government or a nonprofit, or another agency. uh, and that's how, how we find the resources to support that. so it's-- I think it's not our expectation that, a government is, uh, paying money for these evaluation services and more that we will, you know, find the philanthropic, money separately, to, to be able to support the, the work.
Speaker 3Very helpful. Thank you, Louise. and Patrick, I, I think that LEO uses a similar model as well, where you partner with governments but then are able to also raise your own funding to support those research projects. Is there anything else you want to share on that point?
Speaker 6Yeah. I mean, I think I would just piggyback a little bit on what Carrie was saying around intellectual property, right? Like, the university pays me and pays academic researchers more generally to do this type of research, and so we're not looking to fund our own time. The benefit that we get, the incentives that we have then are to go and publish these, uh, these studies in academic journals. And so, it's kind of this nice exchange where we can provide research services at no cost to a government agency or a partner, uh, more broadly. Uh, and, you know, kind of the exchange that we get then is the possibility of, uh, of publishing these results. And so getting that data-sharing agreement right is really important from our perspective. LEO, more generally, when we go to provide research, you know, th- we have to pay our RAs and, and oftentimes there's data collection. As a lab, we commit to covering all of the research costs that are related to this. We go to nice funders like J-PAL and others then to kind of backfill. But, you know, we, we don't expect our research partners at LEO to, to, to pay for any of this research. We do it. uh, you know, we cover the research costs and, and they're covering program costs typically. One thing I might add, Vincent, is that, even though often, there's not an exchange of funds from the government to the researcher for these kinds of projects, so it doesn't necessarily re-require a contract, it's always a good idea to set out some kind of like memorandum of expectations or understanding that help guide, guide the project. So, I think that's a, always a good practice.
Speaker 3Yes. Thank you very much for sharing that, Carrie. Okay, one more question before we hand it over to the audience. I do wanna circle back on the point related to community engagement. So as government, you ultimately are accountable and should be responsive to the needs of the community. But at the same time, we know that even between government and community members, there can sometimes be mistrust and not necessarily even the strongest relationship. So Carrie, curious to hear from your perspective as a person who has worked within government before, what steps have you taken to make sure that any research project is ultimately responsive to the needs of the, the community?
Speaker 4Yeah. this is, this is a, an important question, consideration, and one that I feel strongly about in terms of, you know, the durable relationship between the community is between the community and, and the government. And if you're, if you're starting a research project, I'm definitely encouraging you to have community engagement and participation. You cannot devolve that responsibility to the research partner. Research partners are often not trained or don't have the relationships, to undertake that. And again, you know, I really feel strongly that that's our responsibility as government to remain in the lead of those relationships. some things that you can think about are, you know, all the stages of the program design and research process from designing outcomes to the program and research design, implementation, dissemination of results. All of those different stages have opportunities where you can engage members of the community. Try to think about what those opportunities are. Of course, you have to balance that with what's practical and possible. it's not likely to be like a big public meeting, right? This is a place where some creativity, i- is, is very helpful. So think about, are you working with a group of service providers who you could bring in at key points? I think Mary put an example in the, in the chat earlier about, you know, bringing together, some community groups, to help inform different stages of the process. Can you start with some deep data dives, um, with communities? do you wanna add a qualitative researcher onto your team to do things like survey- surveys? And throughout this process, it's really important that you intentionally think about the equity considerations. as Ann said, you know, what's working for whom under what circumstances? You will miss very important opportunities for ensuring your work achieves the outcome you intend if you do not think about equity from the outside-- outset. So who are the populations who are most in need? And who are the ones who are least likely to benefit unless you really think through how t- how to design, the work in ways that is easily accessible for them? And then how are you going to begin to engage with those communities? And very importantly, what are you willing to do differently to incorporate their input into what, what you're doing? So there's way more here than we have time to dive into. We could do an entire Evidence Matters session on this. But thinking about translation, and please not through AI, it gets things wrong all the time. outreach through trusted partners, offering compensation to community members, things like childcare, food, those kinds of things. Those are all, all things that you can do, to help really get, important community input into your work.
Speaker 3Thanks, Carrie. I think piggybacking on that point, it is so important also for the government partner to transparently communicate the needs, perspectives, and potential challenges of working on a project involving a given community. Because sometimes researchers, they might come into a new community context and sure, on paper, the research design looks good, but there may be certain political considerations that would make this project completely not feasible. Or there could be another research question that is so much more important for this community to answer. And I think circling back to the point earlier about how you as a member of government are an equal member of the research team and are a researcher yourself, we really encourage you to be very forthcoming and to really lean into your role as an equal partner in this research project, especially as a representative of the community. Because researchers, however well-intentioned they are or however much research they've done, they often don't, don't know and they don't have that experience, and they don't have those, those insights. So definitely want to emphasize that point further. Okay, um, so I know that I've been doing a lot of talking, and we wanted to now open it up to see if there are any questions from folks in the audience. I know some folks have already shared some in the Zoom chat, but also encourage folks to go off of mute and share any questions they might have, either ones that relate to the different topics we've talked about or any other lessons learned that weren't covered in the ten lessons.
Speaker 4Hey, Vincent, there was a question early on in the chat about whether researchers can initiate projects and then have the government partner with them, and I thought it would be good to address that from ROC. it's definitely possible, and, I know of many cases where researchers
Community Engagement And Equity
Speaker 4have approached government and, uh, ended up forming really long-term partnerships on particular projects. I'm thinking particularly of the summer youth employment work in Boston, for example. it's a caution though. So I think if you are a researcher and you have a really well-defined question and you're expecting to come to government and quickly get access to data, I want to discourage you from that belief. your government, you know, government staff are really very busy people, and it's not easy to curate datasets and do all that's necessary, to release them to a researcher. So I don't wanna discourage you from initiating ideas and initiating partnerships on your own. I just want you to be prepared to put in the work, that's the trust-building and the relationship-building that's really necessary, to get a partnership off the ground. And so that involves not just bringing your idea, but really doing some deep listening and understanding what the government, staff's needs are in that area, and then finding a place of mutual benefit for you to work together.
Speaker 3I think building on that point, Carrie, one of the reasons why we really emphasize long-term partnership building is precisely that, where so much of the determination of what is a feasible research project, what is a valuable research question, I think emerges from those long-term partnerships and those ongoing conversations. So I think our suggestion to researchers is when you are thinking about initiating a research project with a given government, really think about it as a long-term partnership, not just as a one-and-done RCT where you fly in, take all of the data, do the experiment, and fly out. But rather as a long-term partnership where you can co-create the research questions and co-create the research projects with the aim of ultimately answering questions that will be valuable both for the government partner and also for the researcher. So it's, uh, another double-click on long-term research partnerships. Carrie, Louise-- Um, sorry, Patrick, Louise, um, anything else that you would add to that point?
Speaker 6I mean, just, you know, there are large fixed costs to start up this type of work, and so anytime you can like you said, have that long-term partnership where you're answering multiple questions and you dig in, you know, you find something you didn't expect, and you can dig in. You know, those longer-term relationships, uh, are very, very valuable, from a,
1from an academics perspective too.
Speaker 5and I would maybe just echo Carrie's points that, I think I have seen instances where researchers have a very specific set of research questions or priorities, and are looking for partners to, to work with. and so I've certainly seen success in that. Um, I think it does generally sort of take longer, um, to, to find those partners. Um, and you might have some more sort of, you know, kind of like fa-failed attempts, um, at, at trying to, to answer a, a particular research question, um, before, before you get the, the, the result that you're looking for.
Speaker 3Great. I see a question here. Do you have any advice for tapping into the philanthropic and community networks of the nonprofit partner agency to raise resources to, sorry, uh, to fund the research side of the project when the default for foundations and donors is to prioritize funding direct service over evidence-based evaluations? So I think, there definitely are a large number of funders that are very interested in building evidence. J-PAL is one of those funders as well, so certainly encourage, folks to, to reach out, if they're interested in potential funding opportunities. I think I would encourage folks to look for different foundations, whether or not it's groups like Arnold Ventures or the Robert Wood Johnson Foundation or the Alfred P. Sloan Foundation, that really care about building evidence on a given topic area. Some foundations are very interested in specific methodologies. For example, Arnold Ventures focuses on high-quality, quasi-experimental, and randomized control trials. Some other funders will focus more on descriptive research. So definitely think about the type of research and evaluation you're hoping to pursue, and work backwards from there to figure out which funder is best suited for that specific type of research design. Patrick, Louise, Carrie, anything that you would add to that point?
Speaker 4I'd just add that often, researchers have a connection to funding sources that we don't have as government, and vice versa. So, you know, you might, be thinking about, you know, finding a researcher who is interested in pursuing grant funding with you. As Vincent says, there's lots of foundations out there that do wanna fund research.
Audience Q&A
Speaker 4Robert Wood Johnson Foundation, the Ballmer Group. There are many foundations out there, and even often the direct service provider the, the foundations that fund direct service, they're interested in understanding the impact of their work. And so, they may be interested in adding a component that focuses on the research.
Speaker 3And one final piece of advice there is definitely look into local foundations as well. There are many funders that are focused on a specific geography that are also interested in building evidence around what is effective within that particular community context. Okay. Any other questions from the audience? I have many other questions, but wanna make sure that other folks are getting their thoughts in. Or Carrie, Louise, and Patrick, any questions that the three of you have for one another that
1you're hoping to discuss? Okay. I
Speaker 3will, uh, take the prerogative to ask another question then. So Patrick, this is mainly for you. We touched on this a bit earlier about academic incentives. You've talked a bit about how publication and pursuing novel research ideas is something that is very motivating for an academic researcher. Can you share more about what an incentive-- what the incentives are for researchers to work on a research project, and perhaps what are some common misconceptions that you've seen governments and nonprofits have when working with academic researchers?
Speaker 6well, I mean, the first misconception, and this comes from, from my family too, is that all I do is teach. Uh, but, you know, for most academics, kind of the primary thing that we do is we generate, new evidence, new knowledge in our sphere of influence. I'm an economist, and so I, you know, the primary goal of my job as an assistant professor is to publish in good economics journals, and that is ultimately what leads to my own career progression. so, you know, when I'm trying to evaluate a project, you know, I got into this work because I wanted to use my skills to support, you know, nonprofits, state and local governments in understanding what works in their communities to help people out. in order to translate that into my career as an academic economist, I need to publish well. And so when I'm evaluating projects, you know, some of the things I'll do is say, you know, "If we're evaluating a program, has that program been studied before? Or is there something new that we can learn about this particular type of a-" of a program. And so while it might be, you know, when we're evaluating at LEO whether a project is a good project, their question might be a really great question for that organization. It might provide them with actionable information. We're also trying to evaluate, does that research question provide us with more generalizable knowledge outside of a large county in Northwest US or outside of a community, uh, whatever community we're studying that particular question. And so that's where the overlap is really good, is if, if it provides a good answer to their question and it lets us say something more generalizable beyond that local community. you know, when, when we're starting out projects, a smaller nar-- you know, we've talked about longer term research project, longer term partnerships. A smaller narrower question may not be relevant for my career, but you can imagine the second question that we could get to after a first research project might be something more impactful, um, might provide a more, um, you know, a better journal publication for an e-economist. oftentimes our lab will take on those types of projects because we see down the road the second or the third, third project, if this goes well, might be really, allow us to answer some new questions that we don't have answers to.
Speaker 3Very helpful. Thank you, Patrick. Uh, Louise, do you have anything that you'd like to add to that? I know you think a lot about academic incentives as a person who has to facilitate so many of these research partnerships.
Speaker 5definitely. So, uh, I would, what, what's coming to mind for me is closely related to, lesson number three, that you are a necessary member of the research team. and it's sort of like assuming the opposite, that the, the researcher who's coming in is fully staffed up. Why would I, you know, as a person working in government, you know, need to ask questions and sort of contribute to this, when I think, you know, exactly the opposite is true. Um, you know, you, you're the expert, you know your community, and, and the researchers that you're working with are, are looking to you as, as such. uh, and so, um, I think encouraging, folks to ask questions of researchers. I think another, uh, challenge we, we often, often encounter are sort of, kind of making assumptions, about, the specifics of a research design, um, and the impact that that may have on, on a community. so kind of depending on, um, in the context of a randomized evaluation, what the RCT design is, does that mean that I'm not gonna be able to offer my community members this particular service, that may or may not specifically be the, um, kind of the, the program that's being randomized? these are all questions that I would encourage everybody to ask. and your researchers have an incentive to find a solution and to work with you to find ultimately, um, a, a research design that is operational and feasible within your context. and that, you know, will-- can, can actually be implemented. That's, that's the goal.
Speaker 3Very, very helpful. Yes, definitely encourage folks within government, if you're working with a researcher, ask a lot of questions. big questions, small questions, and everything in between. I think being able to lay it out all on the table really is foundational to being able to move forward with a, with a valuable research project.
Speaker 6I would just add on top of that too, right? there is one model of this world where there's an academic sitting in an office wanting to test a particular theory. that is not the model that we use at LEO. Like, you know, from our perspective and to, to Louise's point about, you know, the partners being a key part of the research team is we think the best, most innovative solutions are developed by people who are out in the field doing this important work. And so, I'm not gonna come up with the right program to solve a particular problem in a local community. That's really where we leverage on the, the, the, um, expertise and the knowledge of the people who are working out in the field.
Speaker 3Fantastic. I know that we are coming up on top. So Kerry, I just saw you unmute. Oh,
Speaker 4I was just gonna say that that's exactly why we have loved partnering with LEO and, and you, Patrick, uh, 'cause you bring that, that respect for the, you know, the government expertise and, and for the community. So thank you.
Speaker 3Fantastic. So I know that we are coming up on time. This has been a really lively and energizing conversation. I wanna hand it back to Regina and Kerry to share any final reflections that they have for today's Evidence Matters session before we wrap up.
Speaker 2Yeah. Thank you, Vincent, and thanks everyone for participating. I-- I mean, it's been said so many times, but for me, the real big takeaway was, all sides of, of a project recognizing the agency and the expertise that, that each other is bringing. and I think Kerry at one point said something about not proving that something, works, but creating change and kind of having the humility on all sides to, to work together in that way and just, and hearing how, how all of you have done that has been really heartening.
Speaker 4Yeah. And I'm-- I was really appreciative of actually kind of demonstrating that kind of partnership here live in our webinar. So you've got someone representing, you know, government expertise. You've got Patrick here, representing the research perspective, and Vincent and Louise, you're kind of, you know, acting as like a policy intermediary. And so, you know, having all of those perspectives available is what really creates a well-rounded project, and particularly when we're all actively working to bring in the perspective of community. so I app- I appreciate having this well-rounded conversation here.
Speaker 3And I think bigger picture, the whole field of evidence-based policymaking really is an ecosystem of so many different types of partners, whether or not you're an academic researcher or a nonprofit leader or a government staffer. And so everybody has their role to play within the evidence-based policymaking movement, and just really encouraged again by this conversation and all of the folks that continue to be committed to using evidence to drive positive change. Okay. And so if folks are interested in a follow-up, uh, conversation, whether or not you want to discuss these topics further, or if you're interested in a specific research partnership with an academic researcher, we've included some contact information here that we can leave on the screen for a few minutes for folks to jot down.
1Okay.
Speaker 3And I think for folks who need to hop off, you are also free to go. Thank you so much for joining us today, and please join us for our next Evidence Matters, session, which we'll be announcing in the next couple of months. So thank you again.
Speaker 4Thanks, everybody. It was great to
1see so many friends in the audience