Evaluation of the Accelerated Peer-Support Partnership (PAP-APP) Programme

An external evaluation of the first phase of the Accelerated Peer-Support Partnership Programme (PAP-APP) concludes the programme has been well designed and implemented. The SAIs have responded above expectations and most of the planned outcomes and outputs are likely to be achieved, though there have been shortcomings which are largely recognized and lessons learnt.

Background 

The PAP-APP programme was established in early 2018 by as a partnership between IDI and two sub-regional organisations of INTOSAI - AFROSAI-E and CREFIAF. The three partners are providing intensive support to nine highly challenged SAIs in sub-Saharan Africa to enable them to build their sustainable capacity for strategic planning and implementation. Two other objectives are to improve the coordination of external support and to be model employers with regard to gender balance, diversity and inclusion.

For 2018-2020 (phase 1), the intention was to prepare project proposals built on SAI-led needs assessments and strategic and operating plans. Phase 1 was funded by the Austrian Development Agency (ADA), the Ministry of Foreign Affairs, Iceland, Ministry for Europe and Foreign Affairs, France, DFID, Irish Aid, SAI Qatar and IDI basket funds. In-kind contributions have also been provided by AFROSAI-E and CREFIAF, as well as the SAIs of Gabon, Ghana, Kenya, Namibia, Norway, Senegal, Sierra Leone and Sweden. 

In Phase 2 (2020-2024), the projects identified in phase 1 are expected to be funded and implemented by various partners. The PAP-APP programme will also continue to provide overarching support. 

Observations, analysis and conclusions

The evaluation has been done by AG Professional and concluded in September 2020.

The conclusion is that PAP-APP is well designed and implemented, that SAIs have responded above expectations, that the partnership with AFROSAI-E and CREFIAF has on the whole worked well, and that most of the planned outcomes and outputs are likely to be achieved by the end of Phase 1, though there have been shortcomings in some respects, which are largely recognised and lessons learnt. Most SAIs in the Programme have improved their strategic planning capacity, and thus, are more sustainable institutions today compared with the situation before the Programme was initiated. The positive results are attributed to the high level of trust in IDI, AFROSAI-E and CREFIAF, as well as the approach of letting the SAIs lead their own needs and development with IDI and partner support and clear synergies with AFROSAI-E and CREFIAF training programmes.

Distance communication technology has evolved, and use has grown rapidly. Videoconferencing, blogs, Webinars and software such as Skype and Ventrilo allow groups of people with a common interest and agenda to interact, mostly for free. PAP-APP makes considerable use of these aids to communication, saving time and travel expense as well as the climate.

The use of peer institutions such as AFROSAI-E, CREFIAF and regional SAIs is effective at less cost than alternatives such as private consulting firms. Thus, the PAP-APP phase 1 approach is inherently efficient and cost-effective. Whether it can be maintained in Phase 2 remains to be seen.

It is noted that auditors are not experts in all areas where support is needed, though IDI and AFROSAI-E focus on institutional/organisational capacity building including internal and external communication, leadership and management development, HR management etc. Adequate expertise should be engaged for, amongst others, organisational development and development of internal and external processes.

The generic implicit Theory of Change (ToC) was not understood by all key stakeholders before the Programme started. However, in individual planning phase with each SAI, the relevant outcomes, outputs and activities per SAI were discussed and tailored. The generic implicit ToC should be customised to each SAI and explicitly used for Phase 2 to ensure the understanding of the ToC and the relevance of the support to each SAI.

There has been progress in advancing gender equality in most SAIs, though the performance indicators in this area need review and better specification. The objective of gender equality is controversial in countries having contrary cultural traditions and will need to be advanced with care and awareness of potential reactions. However, the importance of gender equality should be advocated with emphasis.

There has been some confusion on the respective responsibilities of the partners, which need to be reconsidered and clearly spelt out in any future partnership agreement. There have been problems with the availability of members of Peer Teams from regional SAIs and SAI Norway. Though senior members of regional SAIs are competent and willing to support a fellow SAI, when the need for their participation arises, they are not always available, particularly in the months of audit and rendering annual reports. Greater care needs to be taken to check the timing of peer support.

Peer to peer (P2P) support is generally welcomed. However, auditors are not experts in capacity building, strategic planning and organisational development. Thus, the SAIs need also that kind of support and advice from experts in these areas.

Though programme administration has worked well to ensure that all planned country level activities have been adequately administered, there have been a few complaints of late release of IDI funds. Procedures for funds release need to be fully explained.

Recommendations

In summary, the Evaluation Team has the following recommendations:

1. The generic implicit Theory of Change should be customised to each SAI and used for Phase 2 to ensure the relevance of the support to each SAI.
2. The PAP-APP phase 1 approach is inherently efficient and cost-effective and should be maintained in Phase 2.
3. Though SAIs have taken on board the advice and proposals of their Peer Teams, the sustainability of their new strategic management capacity will depend on follow-up support in Phase 2 and at least one further round of medium-term planning.
4. Project Support Groups in which all donors and potential donors are willing to meet quarterly with the SAI do not suit the political realities or SAI strategies for dealing with external stakeholders in some countries, and alternative means should then be found to ensure regular transparent coordination (see recommendation in the report on Component 1, and in section 6.2 below).
5. Meetings with development partners (DPs) could also be arranged ad hoc when the SAI has something to present, such as a new Strategic Plan, mid-term review or project proposal.
6. The importance of gender equality should be advocated with emphasis. Performance indicators need to be defined more precisely and applied consistently.
7. Monitoring of PAP-APP projects may be integrated with monitoring of SAI strategic and operational plans, as recommended for Phase 2 projects, to consolidate SAI ownership and save administrative time.
8. In future partnership agreements the division of roles and responsibilities should be clearly spelt out together with business procedures.
9. The partners should look further into the reasons for underspending and learn the lessons for Phase 2.
10. The timing of peer support should be assured by pre-planning the availability of members of the peer team and widening the pool of peer providers.
11. As auditors are not experts in capacity building, strategic planning and organisational development the SAIs need also that kind of support and advice from experts in these areas.
12. Distance communication technology should be extensively used also in Phase 2 in line with the global response to climate change.
13. If possible, the IDI Bilateral Support Unit should be expanded to allow country specialisation of its advisers

The full evaluation can be read here.

See this document for IDI’s response to the recommendations.

For more information about the PAP-APP Programme, please click here.