Measuring impact of the Tanzania Financial Sector Deepening Trust: ensuring a quality assessment
In this project, we developed an approach to quality assurance to help maximise the usefulness of a retrospective assessment of the impact of the activities of the Tanzania Financial Sector Deepening Trust (FSDT). Drawing on our earlier work supporting the FSDT in developing an approach to retrospective and forward looking impact assessment exercises, we developed an approach to quality assurance to help maximize the quality and usefulness of such exercises. Our approach can now be used in future impact assessment work carried out by FSDT Tanzania or other similar organisations.
Building on our previous work from 2014, where we supported FSDT Tanzania in developing an approach for conducting both retrospective and forward-looking impact assessments, we were asked to develop an approach to assure the quality of these assessments.
The subsequent retrospective impact assessment, implemented externally by FSDT and others, was incredibly complex as it involved:
- Assessing a programme made up of over 70 different projects and interventions.
- Reconstructing a theory of change for a programme which has evolved over time.
- Assessing the contribution of a market development programme which aims to leverage wider changes in the operating environment and amongst market actors rather than producing direct and discrete project outcomes.
- Managing expectations on the ability to generate data about impact on the lives of end clients rather than the institutions serving these clients.
We developed a constructive, rigorous and transparent approach to quality assurance. This involved ensuring that the quality assurance started as early as possible in the assessment process, was based on a clear and detailed set of criteria and was carried out in a spirit of sharing and openness with the evaluators. This included:
- Development of a quality assurance protocol setting out the objectives, process and outcomes of the quality assurance process from the start of the evaluation. The main tool used in the protocol was a set of criteria based on formats developed by the Specialist External Quality Assurance Service, SEQAS.
- Assessment of the impact assessment Terms of References against agreed criteria.
- Assessment of the Inception report against agreed criteria, covering: structure and clarity; context, purpose, scope and objectives; evaluation framework; methodology and data; inclusion and ethics; planning, management and governance; usefulness.
- Assessment of the draft final report against agreed criteria covering the issues outlined above, in addition to implementation; analysis; findings; recommendations; lessons.
- Advice to FSDT Tanzania and the impact assessment team on the dissemination strategy for the final report.
The approach developed in this project will help support the completion of a rigorous, comprehensive and balanced impact assessment of FSDT Tanzania’s activities. More broadly, the quality assurance protocol developed under this project may be of wider benefit to FSDT Tanzania and other similar organisations - including other African Financial Sector Deepening Trusts -in the management of future impact assessment exercises.