SDI Update – March 2015

Editor’s note: Coefficient Giving was formerly a part of GiveWell under the name “GiveWell Labs,” and later operated as Open Philanthropy. This post originally appeared on the GiveWell Blog; uses of “we” and “our” in the below post may refer to GiveWell. Some content may be outdated; you can see our latest writing here.

This page gives an update on the Service Delivery Indicators Program, which is being carried out by the World Bank in partnership with the African Economic Research Consortium and the African Development Bank. Good Ventures contributed to this project based on a joint assessment with GiveWell.

Summary

In March 2014, Good Ventures made a $500,000 grant to the Service Delivery Indicators (SDI) program. Since then, SDI has continued to make progress on its surveys. Key updates are:

• SDI survey results have received some attention and use. Its data has been used by blogs, news agencies, governments, and the World Bank (more).
• SDI’s expenses are greater than it originally predicted, with surveys costing $700,000 instead of $500,000. SDI needs approximately $4 million in order to reach its goal of surveying 10-15 countries by the end of 2015 (more).
• SDI continues to seek funding to reach its target number of countries, and is considering options for sustainable funding (more).

SDI’s progress

Data collection, analysis, and publication

As of 2013, SDI aimed to complete assessments for 10-15 countries by the end of 2015.^[1]“SDI will assess 10 – 15 countries in Africa on each of the indicators based on a random sample of health and primary school facilities.” GiveWell’s Service Delivery Indicators Program page. In 2011, SDI was aiming to survey 15-20 countries and reassess these countries every two-three … Continue reading SDI still believes it has the capacity to reach that goal with sufficient funding.^[2] “In line with its original objectives, SDI hopes to complete 10-15 countries by the end of 2015.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.  (More on funding needs below.)

Eventually, SDI hopes to be working with 20-25 countries in Africa.^[3]“In Africa, once SDI is working in 20-25 countries, it will have covered all of the countries where the political environment and public domain is sufficiently open to use its information.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on … Continue reading Once it is surveying this many countries, it will have covered all of the countries in Africa where it believes the political environment is sufficiently open to use its information.^[4]“In Africa, once SDI is working in 20-25 countries, it will have covered all of the countries where the political environment and public domain is sufficiently open to use its information.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on … Continue reading However, there is a growing interest in SDI’s surveys from other continents.^[5] “Beyond 2015, there is interest from other regions, which SDI will have to consider.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

SDI’s pilot programs began in 2010.^[6] SDI Timeline  As of July 2014, SDI had:

• Released six complete sets of survey results (Tanzania and Senegal in April 2012, Kenya in July 2013, Uganda in October 2013, Nigeria in June 2014, and Togo in July 2014).^[7]Gayle Martin, World Bank Senior Economist, email to GiveWell, June 1, 2014. “SDI has already gathered and analyzed data for six countries: Senegal Uganda Kenya Nigeria Togo Tanzania” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July … Continue reading
• Completed the fieldwork of gathering data in two additional countries (Tanzania’s second round and Mozambique’s first round).^[8]“The fieldwork of gathering data has recently been completed in two more countries: Tanzania (the second round of data was collected) Mozambique (the first round of data was collected)” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July … Continue reading
• Made preparations to collect a first round of data in four further countries (Niger, Mali, and the Democratic Republic of the Congo).^[9]“SDI is almost ready to collect a first round of data in several more countries: Niger Mali Democratic Republic of the Congo Zambia (SDI does not expect to be ready with Zambia until late 2014 or early 2015)” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank … Continue reading
• Started discussions with two new countries about future rounds of data collection (Senegal and Cote d’Ivoire). The latter would be SDI’s first collaborative survey with the African Development Bank.^[10] This would be Senegal’s second round of data collection and Côte d’Ivoire’s first round. SDI has postponed implementing its surveys in South Sudan because of conflicts there.
  Gayle Martin, World Bank Senior Economist, email to GiveWell, June 1, 2014.

Public use of the indicators

As we wrote in our January 2014 overview of the SDI program, SDI monitors its intermediate success using two indicators:

1. “Public debate on education and health service delivery [is] initiated and/or informed.”
2. “Stakeholders (policymakers, media, NGOs, CSO) reporting use of SDI analysis within 6 months after any of the SDI dissemination events.”^[11] SDI Program Document 2011, Pg 11.

Usage in public debate

SDI has told us that there has been public debate involving SDI’s data in Uganda and Kenya, where both of these countries were in the midst of teachers’ strikes when SDI’s survey results were released.^[12]“Another way in which SDI is having an impact is by generating public discussion. For example, when the information from SDI’s surveys was released publicly in Uganda, it sparked a heated debate about why the education results were very poor. Additionally, teachers were on strike in both Uganda … Continue reading SDI claims that its data fueled debates on education, particularly in Uganda, and that its findings are now discussed at many of Uganda’s and Kenya’s education policy forums.^[13]  “Another way in which SDI is having an impact is by generating public discussion. For example, when the information from SDI’s surveys was released publicly in Uganda, it sparked a heated debate about why the education results were very poor. Additionally, teachers were on strike in … Continue reading

Usage by stakeholders

Tanzania
On the policy side, SDI believes its greatest success so far has been in Tanzania. SDI has told us that in 2011 SDI results were presented in Tanzania to a closed cabinet meeting.^[14] “2011: SDI results were presented at a closed Cabinet meeting: Minister of Education starts doing unannounced visits to schools and confirms SDI’s findings.” SDI PowerPoint slide sent to GiveWell on August 8, 2014.  Afterwards, the Minister of Education began to visit schools unannounced in order to confirm SDI’s data.^[15] “2011: SDI results were presented at a closed Cabinet meeting: Minister of Education starts doing unannounced visits to schools and confirms SDI’s findings.” SDI PowerPoint slide sent to GiveWell on August 8, 2014.  Then, in April 2013, SDI claims that its data was used during planning for the “Big Results Now” initiative.^[16] “April 2013: Presidential initiatives use SDI data as diagnostics during planning for major reforms: Reform Compact and Big Results Now!” SDI PowerPoint slide sent to GiveWell on August 8, 2014.  The “Big Results Now” initiative is part of Tanzania’s effort to become a middle-income country; it focuses on improving six areas of the economy, one of which is education.^[17] Big Results Now, The United Republic of Tanzania Prime Minister’s Office  Several blogs began to reference SDI’s Tanzania data, especially its data on teacher absenteeism.^[18]

• SDI PowerPoint slide sent to GiveWell on August 8, 2014.
• Blog post referenced on slide: DFID blog.

 In 2013, SDI’s data appeared in a report on Tanzania’s education system.^[19] See pages 21 and 50 of Education: NKRA Lab Report.  Finally, SDI tells us that in 2014 its data was used to further track the progress of the “Big Results Now” initiative.^[20]“June 2014: SDI indicators feed into and use to track performance of Government of Tanzania and World Bank/ project values $122 million.” SDI PowerPoint slide sent to GiveWell on August 8, 2014 “12. Low rates of teacher time-on-task are major problems in education service delivery in … Continue reading

The World Bank
Another policy accomplishment was the World Bank’s decision to use SDI’s data to help appraise programs that are requesting funding.^[21] “Finally, the World Bank has begun to use SDI’s indicators as some of its disbursement-linked indicators.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.  The World Bank will agree to fund a country’s program if the program is shown to be achieving certain performance targets, and in both Tanzania and Uganda, the World Bank is now using SDI’s data as a tool to measure whether or not programs are reaching their targets.^[22]“Disbursement-linked indicators are sometimes used when the World Bank makes a conditional funding agreement with a country. Essentially, the World Bank will agree to fund a country’s program if the program is shown to be achieving certain performance targets. Disbursement-linked indicators are … Continue reading This could end up making SDI’s task more difficult, because if $10-15 million were dependent on the outcomes of SDI’s surveys, that could incentivize people to inflate the countries’ performance metrics, meaning SDI would have to increase its quality-assurance efforts.^[23]“There is also a risk: this could end up making SDI’s task more difficult, because if $10-15 million is dependent on the outcomes of SDI’s surveys, that could incentivize people to inflate the countries’ performance metrics. This risk will have to be carefully tracked to avoid unintended … Continue reading

In addition, the World Bank and the Gates Foundation are looking to start a global initiative focused on primary health care. SDI surveys may be used in the future as part of a “Primary Health Care Performance Initiative.”^[24] Gayle Martin, World Bank Senior Economist, email to GiveWell, June 1, 2014.

Media and other
Articles in high-profile news sources (e.g., the Economist and BBC) are beginning to discuss the data that SDI is collecting.^[25] “Furthermore, articles in high-profile news sources (e.g., the Economist and BBC) are beginning to discuss the data that SDI is collecting.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.  SDI is not mentioned in these articles explicitly, but it claims to be the only possible source of the information discussed.^[26]“Furthermore, articles in high-profile news sources (e.g., the Economist and BBC) are beginning to discuss the data that SDI is collecting. SDI is not mentioned in these articles explicitly, but it is the only possible source of the information discussed in the articles.”GiveWell’s … Continue reading SDI’s results have also been presented at fora such as:^[27] Gayle Martin, World Bank Senior Economist, email to GiveWell, June 1, 2014.

• The 10th Anniversary of the 2004 World Bank Development Report (February 2014)
• The Oxford University Conference on African Studies (March 2014)
• The World Health Organization and Spain Colloquium on Water and Sanitation in Health Facilities (April 2014)
• The third Uganda Economic Update on Decentralization and Service Delivery (June 2014)
• The World Health Organization Consultative Meeting on Health Facility Surveys (June 2014)

SDI has also developed its own website and is working to improve data visualizations for visitors.^[28] Gayle Martin, World Bank Senior Economist, email to GiveWell, June 1, 2014.

Building capacity

SDI aims to build up countries’ capacities to gather and analyze data by having local researchers work on its surveys.^[29]“A key principle is that the SDI implementation model relies on is the use of national research institutes as implementing partners. Continuing this model will be critical since countries benefit from the capacity built through implementing the SDI surveys.” GiveWell’s non-verbatim summary … Continue reading Therefore, SDI and its partners strongly prefer to use national research institutes to implement the surveys.^[30] “A key principle is that the SDI implementation model relies on is the use of national research institutes as implementing partners.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.  Pursuing this goal of building capacity, a grant in the amount of $840,270 was successfully mobilized to support capacity development activities implemented by the African Economic Research Consortium (AERC), one of SDI’s partners.^[31] World Bank, Service Delivery Indicators, email to GiveWell, March 25, 2015  With funds from the grant, AERC successfully launched first training workshop in Kenya on the use of SDI data by national researchers.^[32] World Bank, Service Delivery Indicators, email to GiveWell, March 25, 2015

Funding updates

Program Costs

Each round of surveys costs SDI $700k (this includes both the health and the education surveys).^[33]“Although it originally predicted $400k-$500k for 2 surveys (health and education) per country, actual costs have been closer to $700k per country (this covers both the health and the education surveys).” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior … Continue reading This is higher than SDI’s original predictions of $400k-$500k per round of surveys.^[34]“Although it originally predicted $400k-$500k for 2 surveys (health and education) per country, actual costs have been closer to $700k per country (this covers both the health and the education surveys).” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior … Continue reading SDI underestimated the costs of contracts with survey implementers, the costs of additional oversight for quality assurance, and the costs of disseminating its findings.^[35]“SDI has also experienced some funding challenges. Although it originally predicted $400k-$500k for 2 surveys (health and education) per per country, actual costs have been closer to $700k per country (this covers both the health and the education surveys). In part this is because contracts with … Continue reading

Room for more funding

SDI does not have enough funding for the last 5-6 countries it would like to survey to reach its 2015 goal of 10-15 countries surveyed; thus, its immediate funding gap was roughly $4 million in July 2014.^[36]“SDI is currently not fully funded for all 15 countries that it hopes to reach by the end of 2015. It is underfunded in about 5-6 countries, so its immediate funding gap is roughly $4 million.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, … Continue reading It was hoping to have this funding by July 2015 in order to start the processes needed to finish the surveys by the end of 2015.^[37] World Bank, Service Delivery Indicators, email to GiveWell, March 25, 2015  If SDI is unable to find the funds it needs, it will simply survey fewer countries.^[38]“If SDI is unable to get the funds it is hoping for, it will survey fewer countries, rather than decreasing the budget per country. Scaling down the per-country budget would require SDI to decrease activities like quality assurance and dissemination, which are essential to the project’s … Continue reading

Sustainability

SDI is also considering plans for its long-term sustainability. So far, SDI has been opportunistic about leveraging sources of funding within the countries it has surveyed.^[39]“So far, SDI has been opportunistic about leveraging sources of funding within the countries it is surveying to complement the resources in the multi-donor trust fund resources.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, … Continue reading For example, it has used funds from the Department for International Development (DFID) and the Gates Foundation on a country by country basis.^[40] “For example, it has used funds from the Department for International Development (DFID) and the Gates Foundation (in Nigeria) on a country by country basis.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.  This allows SDI’s trust funds to last longer.^[41]“Given the funding constraints SDI has preferred to leverage resources at the country level, including World Bank resources, so that its multi-donor trust fund resources last longer. However, the trust money is still being spent to supplement the in-country funds.” GiveWell’s non-verbatim … Continue reading However, SDI would prefer to have a more sustainable funding model; its current model is costly because of the many transactions and different sources of funding that SDI must keep track of.^[42]“This is not SDI’s preferred model of funding because it elevates transaction costs—costs would be fewer if the funds were all in one pool.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014. Also: Gayle Martin, World Bank … Continue reading

In thinking about how to create a new model, it is important to SDI that it remain neutral and transparent.^[43]“The future funding model has implications for transparency. A model where funding is exclusively coming from on particular party – whether it be governments or the World Bank – has risks for transparency.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, … Continue reading Given the public good nature of the data generated through SDI, the preferred model is where most of its funding is split between governments and development partners, with some additional funding coming from the World Bank.^[44]“Furthermore, information is a global public good and a joint government-development partner model will be optimal.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014. “SDI will probably never be fully funded by the World … Continue reading

Sources

DOCUMENT	SOURCE
Big Results Now, The United Republic of Tanzania Prime Minister’s Office	Source (archive)
DFID blog	Source (archive)
Education: NKRA Lab Report	Source
Gayle Martin, World Bank Senior Economist, email to GiveWell, June 1, 2014	Unpublished
GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014	Source
Project Appraisal Document on a Proposed Grant to the Government of Uganda	Source
Program Appraisal Document on a Proposed IPA Credit to the Republic of Tanzania	Source
SDI PowerPoint slide sent to GiveWell on August 8, 2014	Source
SDI Program Document 2011	Source
SDI Senegal	Source (archive)
SDI Tanzania	Source (archive)
SDI Timeline	Source (archive)
World Bank, Service Delivery Indicators, email to GiveWell, March 25, 2015	Unpublished