Episode 20, Jun 02, 2021

How to Use Data and Rigorous Evaluation to Help Achieve Your Nonprofit’s Goals

Regularly assessing the impact of your nonprofit’s initiatives and even its mission is more important than ever to all stakeholders including your donors.

In this episode, Bob DiMeo and Devon Francis speak with Annie Duflo, executive director at Innovations for Poverty Action. Together, they explore the steps involved in using effective data that fits your organizational needs, analyzing it to identify the core issues and investment opportunities, and making more informed decisions that help you achieve your nonprofit’s mission.

Annie discusses:

• How data-driven decisions can help your nonprofit achieve significantly higher returns on investment
• Ways to ensure that your data is not under-utilized
• How IPA conducted extensive research on microcredit to study the impact on poverty
• Tips to help you adapt your initiatives and programs to a larger scale
• And more!

[00:00:00] Welcome to Nonprofit Investment Stewards with Bob DiMeo and Devon Francis from Fiducient Advisors. Bob and Devon are passionate about helping nonprofit organizations prosper. Whether you oversee endowment, foundation or retirement plan investments, this podcast exists to help stewards improve performance, reduce costs and discover strategies that enable your charitable organization to prosper and advance its mission.

[00:00:26] Now onto the show. Hello, and welcome back to the Nonprofit Investment Stewards podcast. I'm Bob DiMeo, as always joined by co-host Devon Francis. Now our podcast has a variety of listeners from nonprofits, CFOs and executive directors to board members, and many more. Today's episode hits on a topic that is increasingly important to all these stakeholders, including your donors.

[00:00:54] And that is, how can we tell if a nonprofit’s initiatives or even its [00:01:00] mission are effective. We've got an amazing guest who will help us understand how data and very rigorous evaluation should play a vital role if you're a nonprofit leader. But before that, Devon always going to be with you. How are things?

[00:01:16] Fantastic. Thanks, Bob. I am so excited today to hear from our guest, Annie Duflo, who is the executive director at Innovations for Poverty Action. IPA is a Washington based nonprofit that evaluates programs serving the poor and Annie is responsible for leading the strategic direction of IPA. She has a broad wealth of experience implementing and managing randomized evaluations and using data to make better decisions.

[00:01:44] She has a long list of achievements. She holds a master's of public administration and an international development degree from Harvard University and IPA research covers a lot of ground from agriculture and financial inclusion to education, health, [00:02:00] and a lot of other topics.

[00:02:01] So Annie, we are so eager to glean your insights on this fascinating subject. So, welcome to the show. Thank you. Thanks for having me. Annie, it's so good to be with you. And before we jump into the importance of using data to improve decisions and such, I'll just share how I became familiar with you. I know it was earlier this year.

[00:02:20] It was a cold Saturday morning back in January in Chicago. And I'm reading the wall street journal featuring an article about you and IPA. And I have to tell you, I was so impressed with the analytical rigor that's referenced in terms of the work that you're doing and so on. I jumped on your website and learned more, made a tiny donation just because I was so moved by the work that you're doing.

[00:02:44] And then ultimately as you know, reached out and we've gotten to know each other a little bit that way. So, before we jump into the things, perhaps you can share a little bit about your background, how you landed at IPA, and really the role the organization plays in fighting poverty. [00:03:00] Great. Happy to. So, I was raised with two values that I think are fundamental to IPA’s work.

[00:03:08] My father is a mathematician and my mother is a pediatrician who also was volunteering for a nonprofit organization that helps children victims of war. And that really instilled in me both strong belief in science and rigor, but also the desire to have a social impact. And since college, I have volunteered and then worked for a couple of nonprofit organizations, both in France and then in India, where I lived for six years, which is when I really decided to work in international development more specifically.

[00:03:44] And it's also in India that I was introduced to IPA. At the time, I was leading a research center focused on micro finance, the center for microfinance research. And our model was very similar to that of IPA in that we would partner with [00:04:00] researchers, practitioners, decision-makers to study microfinance.

[00:04:05] And that's when I first worked with Dean Karlan, an economist who founded IPA in 2002. So, micro credit at the time was a very popular approach. Now I would go to conferences and hear about how this was a silver bullet to poverty alleviation and, and how all those things would follow, like women empowerment, education. At the same time though, there were local governments and local media who were accusing microcredit organizations to push people and farmers into over-indebtedness and suicide.

[00:04:41] I remember that one of our partner organizations almost had to close their doors because of that. So, what do you believe? Right? There is always one anecdote or another to support one story or the other, and that's really where IPA comes in.

[00:04:59] Our [00:05:00] mission is to answer that questions rigorously and to discover what works to reduce poverty. And we do that by partnering with researchers, practitioners, decision-makers very much the same way that the R and D unit of pharmaceutical company does for medicines. Right. So, I give you the example of microcredit, but we work across a range of issues like agriculture, health, education, or even conflict.

[00:05:31] It's so interesting, Annie. So, you and IPA conduct extensive research all with the goal of ultimately fighting poverty. So, before we delve into specifics, can you talk broadly about the role that good data and analysis can and should play in the world of nonprofits? Right.

[00:07:45] And I imagine that it must be hard for a lot of nonprofits to really take a step back and, and ask those questions and be willing to deal with the answers. If, you know, for instance, if they're, if [00:08:00] the impact is not, you know, meeting the goals. So, from your perspective, do you feel that data is under-utilized by charitable organizations?

[00:08:11] That's a good question. I would rephrase by asking whether they're using the right data in the right amount and whether they do use it. And what we often see with the organizations that we have worked with is that

And at IPA, we have created a unit that provides advisory services to organizations to help them develop the right fit monitoring and evaluation strategy. And we call it the right fit evidence you need for, for that reason, because this is really about developing the right fit data strategy that that fits an [00:09:00] organization's needs.

[00:09:01] Annie, in that Wall Street Journal article, you talked about dispelling misperceptions. I think the example was reducing bacteria related deaths with a pretty simple solve. And you were quoted as saying, sometimes people need a short nudge rather than a long lecture. Expand on what this means. And again, how it might relate to using data to advance a mission.

[00:09:24] I was referring to behavioral approaches to influencing people's behavior. And I can start with myself as an example. If I didn't have an automatic transfer from my bank account to my savings account, I certainly wouldn't save as much as I do now. And we all have these things, I'm sure you do as well, that, you know, we know we should do, and yet we don't do them.

[00:09:49] Like go to the gym or eat a salad and lecturing me on the value of savings or healthy eating probably won't do the trick. Right. And [00:10:00] people are no different, except that they have a lot more things to think about because a lot of things like automatic transfers are not baked into their day-to-day life.

[00:10:11] So going back to the savings example that I started with, there has in fact been several studies of the impact of conventional, conventional financial education programs. And the results are pretty consistent. These conventional financial education approaches don't really change knowledge, nor do they change behaviors.

[00:10:33] And that's a really important finding because every year, several millions of dollars are being spent by nonprofit organizations, as well as governments on the financial education programs. On the other hand, there has been other approaches that, that we've tested at IPA that did lead to a change in behavior.

[00:10:59] For [00:11:00] example, sending people text messages to remind them to save did lead to increases in savings or in the Dominican Republic, we tested an alternative approach to financial education training, which was a rule of thumbs training. The idea was to give entrepreneurs simple tips, like, you know, separate your salary from the rest of your business, as opposed to teaching them underlying accounting principles.

[00:11:29] And this approach did lead to improvements in business practices. So, building on my quote and not just often work better than, than lectures, but also the design and the content of lectures and trainings really matters. And not, not all, but a lot of the interventions that we evaluated at IPA do include such behavioral components.

[00:11:52] Cost sectors. And to give you a very, a relevant example right now, we're [00:12:00] working in, in Bangladesh to test different approaches to aggravate people to wear masks consistently. So, for example, using trusted leaders to share messages or reminding people on the street to wear their masks.

[00:12:15] And those have been shown to increase consistent mask putting three-fold. And as we speak, these approaches are being adopted by some organizations in India. That's terrific. You're talking about the, pardon me Devon, talking about the effectiveness and not necessarily sticking with you know, what has been done for some time, even if it isn't necessarily effective. Really enjoy hearing about that.

[00:12:43] So Annie, IPA has, you already alluded to your work in the micro finance and micro credit space and your organization has conducted some fascinating research on microcredit. So, can you share what you learn about unintended [00:13:00] consequences in this area? And if you have examples of other areas with examples of unintended consequences. So, it's not so much about unintended consequences but about results that not everyone expected, I guess.

[00:13:18] So remember, I mentioned the two very opposite perspectives on, on microcredit, right? Either it's a silver bullet to poverty or it pushes people into suicide. Well, it turns out that none of these stories was, was true. There has been six legal studies of microcredit programs by IPA and other researchers.

[00:13:41] And those were all randomized evaluations, which is the main methodology like you mentioned in your introduction, that IPA uses to evaluate the impact of programs. That's the same method that's being used to assess the effectiveness of medicines. So, all of these studies were pretty [00:14:00] consistent and showed that the traditional microcredit product did help some businesses.

[00:14:03] But overall, it didn't lead to increased incomes across the board. It's a useful financial tool. It provided people freedom of choice on how to earn and on how to spend their money, which is certainly a great thing. But again, it was not a silver bullet to poverty alleviation, and it didn't lead to other things like women empowerment and things like that.

[00:14:31] So, certainly this is a useful financial tool that private investors and banks should, should invest in, but not necessarily where philanthropic dollars should be invested. Now, I started by saying those were not, these results were not expected by, by many. In some sense, maybe it wasn't that surprising, right?

[00:14:51] It's like my credit card. Right. It's very helpful to me, it's a useful tool, but it's certainly, you know, it hasn't made me an entrepreneur in a day. [00:15:00] Right? On the other hand, microcredit wasn't harm-free either. It was not, it was not pushing people into suicide or over indebtedness. Annie, one area of focus, an important area of focus for IPA is social protection.

[00:15:16] And the research section of your website poses this question, can extensive support help households work themselves out of extreme poverty? And it touches on points and I'll do air quotes here, points like asset transfers on everything from goats and chickens to savings and consumption. And love to have you tell us more about your findings and whether training and support can actually help solve poverty.

[00:15:46] So let's start by finding ourselves what it means to be an extreme poor. The world bank definition is that it means living on the $1.99 a day, which for you and me requires a stretch of the [00:16:00] imagination. Yet, it's not just about the money, right. Being extreme poor means not having enough savings to weather any sort of shock, like a health shock, which means that you're not able to take any risks.

[00:16:15] Therefore probably not start a business. It often means going to pay lots of hungers, not being able to keep a job as a result, low self-esteem. In other words, it's a vicious circle. And as a result, it's hard to know where to start. So, at IPA, we have evaluated in seven different countries, a bundled approach that addresses several of these problems at once.

[00:16:44] And it's called The Ultra Poor Graduation Approach. It's an approach that was initially designed and the holdout by BRAC, the largest nonprofit in Bangladesh and the world, I think. And in fact, they, they designed this approach initially because [00:17:00] they realize that their microcredit product was not reaching the poorest of the poor, probably because it was too much of a risk for them to take.

[00:17:09] So this ultra poor graduation approach consists in giving people a productive asset, like a cow or a goat or chicken. And together with that, some training on how to use this asset. Weekly household visits to, to provide them support over 18 months. It also involves the consumption stipends to prevent periods of hanger and access to a savings account or a health stipend.

[00:17:38] And what we find is that this approach led to all sorts of improvements, increased income, increased consumption, increased assets, increased food security, better health, and also more positive outlooks on life. And the, the program also, the effects of the program [00:18:00] lasted beyond the 18 months that, that the visits lasted.

[00:18:03] So that that's an important finding. And even though the program is, is quite expensive, when you compare cost and benefits, it was overall quite a cost-effective program. We actually calculated returns on investment. And from that, they range from 133% to 433% depending on the context and the cost. And this approach has now been adopted in more than 20 countries as, as a result, I would say it's one of the most effective approaches that, that we have evaluated, not to say that this is a silver bullet, because I don't think there is any silver bullet, but it's certainly something that we recommend to organizations.

[00:18:48] We have a lot of investment minded people. And I think their ears probably perked up when they heard 133 to 433 return on investment. That's a [00:19:00] pretty good track record. And it must impress donors too. It's, it's, it's wonderful. So, Annie, IPA aims to help solve global poverty and clearly your use of data to drive decisions and investments is very impressive.

[00:19:16] I think I can speak for both Bob and myself when I say that the work that you do is just staggering. What advice do you have for other non-profit stakeholders? So, in any type of charitable organization about the use of data and making better decisions.

So, I mentioned that at IPA we've [00:21:00] started this new unit called the right fit evidence unit that advises organization on how to develop them, monitoring and evaluation framework.

[00:21:09] We developed a framework which we call the CART framework. It stands for credible, actionable, responsible, and transportable. There were other frameworks out there, but it's a useful one to really guide your decisions in terms of what data to use and collect. And finally, my last advice would be that

People are sometimes surprised that I say that because that's kind of the business that we're in, but I think it's really important to first look at without there is existing evidence out there. And like I [00:22:00] said earlier, if there is enough of it, there is no need to measure impact again, in that case, focus on monitoring your intervention to make sure that it's implemented as planned to achieve the desired outcome.

[00:22:12] Annie, in terms of a charitable organization attempting to amplify their mission and impact, we know that scale matters. What are some of the opportunities and maybe some of the pitfalls when it comes to scaling up initiatives and, and what role can data play in that example? So, we started with my own experience.

[00:22:35] Some years back, I spent time in Ghana and worked with the ministry of education there to explore how they could adapt and scale a targeted education approach that was shown to be effective in India. And I had, when I lived in India, I had worked with the nonprofit organization that developed this approach pattern.

[00:22:59] And I'm, [00:23:00] have a great admiration for this, for this organization. So, I was really excited about exploring, adopting this approach in Ghana. And all sorts of questions came up about how this could be adapted in a different context, not just geographical, but also cultural and institutional, because in this case, it had to be done to the government.

[00:23:22] And there were questions around whether this will work at all in this new context. So indeed when, when we scale something, right, that, that worked initially, often, that means that the organization, or even just the people implementing the program will change. The context will change as well. The all sources will change as well because you may have to do it with lower unit costs.

[00:23:49] So the original intervention may not end up having the same impact when scaled, because the intervention actually will change when we all do [00:24:00] them. So, I think that there are three pieces of advice here. The first is to think about scale from the beginning. So even  .

[00:24:19] The second piece of advice is, because the program will have to be adapted somewhat to the new scale and context, it's really important to be very clear about the key components that made the program successful in the first place so that they don't get lost in the adaptation. Right. And at the same time, it's of course, very important to understand the new context and what will be different about it, right.

[00:24:45] Will the people implementing it be different, will the cultural response be different, et cetera? And finally, it's really important, I think, to have a research agenda and the learning agenda to doubt the process. [00:25:00] At IPA, we have what we call a pass to scale research agenda, where we identify promising interventions that were rigorously evaluated and shown to be effective.

[00:25:12] And then we work to scale them up together with partners and to study their impact as they get scaled. Wow. So, we've covered a lot of ground today. Is there anything that you haven't said that you would leave behind as an offering to nonprofit leaders, maybe the board or even the investment committee of a nonprofit organization that might be useful as they attempt to kind of move through the world as good stewards?

[00:25:41] So I'll get back to some of the things that I mentioned earlier that I'd like to leave you with.

That's terrific, Annie.

[00:27:14] The term that came to mind, as you were describing that you called it learning, in my mind, I had discovery and really the same. You're just saying, Hey, let's, let's learn, let's discover and ultimately improve and grow. So that's terrific. So, Annie, we always like to learn a little bit more about the person and so Devon and I would like to ask you, you obviously are so committed to your profession and it's, it's demanding and such, but.

[00:27:38] Tell us about one thing you enjoy doing outside of your professional and volunteer activities? Well, I have a two-year-old baby, toddler, I guess. So that, that keeps me quite occupied and very happy. And in my spare time, apart from that, I like to play the piano. Excellent. Well, I'm sure you don't [00:28:00] have that much spare time, but that's certainly a valuable way to spend it. I make sure to play a little bit almost every day. Oh, great. Good for you. Wonderful. Practice matters.

[00:28:13] So Annie, it's been so great to have you on the show. We really appreciate you coming on. If listeners want to learn more about Innovations for Poverty Action, where should they go to find out some info? Our website, I guess is the best place. www.poverty-action.org, poverty-action.org. Perfect. Well, thank you so much for coming on.

[00:28:36] It's been excellent to speak with you. Thank you so much for having me. It's a pleasure. Thank you, Annie. I echo Devon's thoughts. Our listeners will greatly appreciate this and thanks so much to our listeners as I digest what Annie has to say about the use of good data. I'd almost define IPA as being healthy skeptics and candidly, that's the role that Devon and myself and frankly, [00:29:00] everyone at Fiducient Advisors attempts to play in helping our clients oversee endowment, foundation, even retirement plan investments for nonprofits.

[00:29:09] So if you or your board feels there's room for improvement or you're just not sure, you absolutely should reach out to me or Devon via LinkedIn. And you definitely want to check out the many resources at fiducientadvisors.com. So, to all you good stewards, thanks for investing time to help your nonprofits prosper.

[00:29:29] We'll connect with you soon on the next episode. Thank you for listening to the Nonprofit Investment Stewards podcast. Click the subscribe button below to be notified of new episodes and visit fiducientadvisors.com for more information. The information covered and posted represents the views and opinions of the guests and does not necessarily represent the views or opinions of Fiducient Advisors.

[00:29:52] Content is made available for informational and educational purposes only and does not represent a specific recommendation. Always [00:30:00] seek the advice of qualified professionals familiar with your unique circumstances.