Financial sector development in Tanzania: developing a common approach to impact assessment
This project has provided a framework for measuring and evaluating the impact of the financial sector deepening trust in Tanzania (FSDT). Supported by both donors and Government, and operational since 2005, the programme has yet to evaluate its own impact on both the financial sector and its contribution to livelihoods within Tanzania. This reflects the challenges of measuring market development programmes more generally. Our team worked closely with FSDT and its funders (DFID, Danida, Gates, Canada’s DFATD) to assess their priorities for impact measurement, using consultations and stakeholder interviews to support the development of both retrospective (2005-2013) and prospective (2013-2018) impact assessment designs. The evidence gathered from these assessments will help inform programme improvements, ensuring funding is better directed towards those initiatives with the greatest impact on improving inclusion and reducing poverty.
FSDT is a market development programme aimed at improving financial inclusion and reducing poverty in Tanzania. It has been operational for close to a decade, investing in a range of areas, from developing innovative financial products to improving the financial sector policy and legal environment. But as yet there has been no assessment of its overall impact. Understanding the impact of the programme will help inform improvements in its design, helping ensure that funds are targeted towards those initiatives that best support the programme’s original aims. Measuring the impact of market development programmes is difficult and there is limited global best practice to draw on. The traditional approach of measuring results against static logframe indicators throws up a number of challenges for programmes which are heavily influenced by changing external factors and must constantly adapt to take advantage of emerging opportunities. They are also typically complex, operating across a number of different levels, sectors and parts of the market. To be effective, any results measurement system needs to capture impact at all these levels. This project was established to address this knowledge gap by developing impact assessment approaches that will provide valuable insights into the performance of the programme to date as well as recommendations for its future assessment and improvement.
Our team worked closely with FSDT and its funders to develop guidance for more effective measurement of FSDT interventions. We took stock of lessons learnt by the Donor Committee for Enterprise Development (DCED), and other evaluations of market development programmes to build consensus around a common, pragmatic approach to impact measurement. As a first step this included developing a clear definition of what ‘impact’ stakeholders prioritized, and at what level of the market (e.g. should FSDT focus on households or financial inclusion?). We interviewed a number of country stakeholders to understand their views of what the main FSDT impact pathways were that should be assessed. This required the project team, in consultation with FSDT, to retrospectively develop a theory of change and prioritise the pathways which should be explored in more depth. Two sets of options, complete with various trade-offs and costings, were presented to FSDT and its investment board. One focused on what FSDT could do immediately to undertake a retrospective impact assessment of its operations to date. The second was broader in nature, focusing on what types of impact assessment could be undertaken prospectively, which approaches would improve the evaluability of the FSDT programmes, and how to improve monitoring by collecting information on a real-time basis.
We undertook the following activities:
- Completing a literature review of international best practice of impact assessment of market development programmes, including building on previous work undertaken by OPM for FSD Kenya
- Conducting stakeholder interviews with FSDT representatives, market actors and funders
- Developing a Theory of Change (ToC) and coordinating consultations to identify main impact pathways
- Producing options papers for impact assessment design and presenting them to the investment board
- Developing terms of reference for future impact assessment studies
Our project has helped FSDT and its funders understand the challenges, trade-offs and opportunities in attempting to measure the impact of a large, complex and multi-faceted programme. From a funder’s point of view, access to stronger evidence will help ensure greater transparency, accountability and value-for-money as funds can be targeted towards those initiatives that have the biggest impact. From an FSDT management perspective, better information will feed back into programme design improvements and translate into stronger, more inclusive financial markets in Tanzania that support poverty reduction outcomes. More broadly, the project has contributed to the development of a common approach to measuring the impact of market development programmes, in turn, supporting meaningful comparisons between programmes operating in a number of different contexts.