A joint project between the Office of the Auditor General of Somalia, the INTOSAI Development Initiative (IDI) and AFROSAI-E between 2018-2020 has now been evaluated. According to the evaluation, the project has made a good contribution to the strengthening of the external audit function in Somalia. The objectives were mostly met and the project scores reasonably well in terms of relevance, coherence, efficiency, effectiveness, and impact and sustainability. Still, the evaluation points to some areas of improvement for a new phase of collaboration between the partners.
The overall objective of the project was to enable OAGS to successfully implement the strategic priorities in the period 2017-2020, and especially strategic goals 1, 2 and 6:
- Goal 1: Timely, relevant and high-quality audit reports in line with international standards;
- Goal 2: Strengthening internal governance for efficient and effective Audit Services;
-
Goal 6: Amend the old legal framework OAGS currently operates under.
During project implementation, this support was broadened to also include communication, human resource management and IT infrastructure. Funding was provided by the Norwegian embassy in Kenya. In total 5,1 million was allocated by the embassy.
Overall conclusion of the evaluation
In summary, the Project’s objectives are mostly met and the project scores reasonably well on the OECD-DAC evaluation criteria. The high level of confidence and trust between IDI and the AG in each other’s objectives and activities has ensured that the Project has generally been implemented smoothly and without significant issues. According to the evaluator, the project’s contribution has been recognized particularly with regard to the introduction and training on ISSAI-based financial and compliance audit manuals, the introduction of a strategic and annual planning and control cycle and various human resource management tools. The improvement in the performance of the OAGS is evidenced by its ability to audit the Government’s annual budget execution report in a timely manner for the first time in 2019 and its willingness to report openly on challenging topics involving corruption and malpractice with regard to donor-funds. The Project has also contributed to the drafting a new Audit Law. If enacted, following approval by the President of Somalia, the OAGS will achieve a higher degree of independence from the Executive. Once the Act is in place the OAGS will be in a stronger position to make recommendations for the improvement of public financial management by the Somali government in the coming years.
The evaluation has arrived at this positive overall conclusion by a standard methodology consisting of document review and interviews. The interviews were done remotely due to the travel restrictions relating to the COVID pandemic. The methodology has also made use of two questionnaires with open and closed questions that were filled by the OAGS management.
The evaluation has been guided by the six OECD-DAC evaluation criteria. The assessment of each criteria are presented in more detail below.
Relevance
The Project was highly relevant in the context of Somalian societal needs and the specific requirements of the OAGS. The Project’s design reflects a logical and consistent sequence between input, outputs and outcomes and is fully aligned with the OAGS Strategic Plan.
Coherence
IDI’s engagement with the OAGS complied with the conditions stated in IDI’s Bilateral Policy. In deciding to proceed with the OAGS Project, IDI made a strong case which met all of the requirements identified in the Policy. The identification of AFROSAI-E as the main delivery partner for IDI was well founded in the light of its capacity, regional network and successful delivery record; the long history of collaboration between IDI and AFROSAI-E; and the preferences of the OAGS.
Efficiency
The Project’s performance against the ‘efficiency’ criterion was mixed. It performed well in respect of the quality of its work and on the management of risk. It was less successful in securing delivery of all the expected outputs and in managing budgets and costs in line with the original plan.
There were a number of significant risks within the initial project design arising out of IDI’s decision to support a SAI in one of the most fragile and least secure countries in the world. These risks have not materialised or their impact on the outcome and impact of the project was mitigated to an acceptable level. Especially, IDI’s decision to work primarily remotely and organise field workshops in neighbouring countries worked out well. Although the OAGS senior management would have preferred a higher presence of the advisors on the ground, overall the IDI approach proved to be effective. The quality of the project’s outputs was perceived by the OAGS to be high across all activities. The Project also made a very positive contribution in promoting donor coordination at both the strategic and operational level.
One risk inherent in the project design - the low allocation to personnel costs of the implementing partners IDI and AFROSAI-E and high reliance on in-kind support from resource persons from peer SAIs - was not sufficiently mitigated. During the Project, this has led to significant overspending on IDI personnel costs in the first year of implementation (2018). The overspending reflected partly the positive momentum in project implementation in 2018, but also poor control of IDI’s personnel costs and the absence of controls or ceilings on the number of staff working days spent on the Project. The overspending by IDI staff was also a response on the limited availability of the resource persons. The principle of engaging experienced peers from other African SAIs to assist SAI capacity building is well established in both IDI and AFROSAI-E’s work. However, the quantity and quality of the inputs received from the resource persons are difficult to manage due to the informality of the arrangements put in place by IDI.
Together with IDI staff costs, the main cost driver for the Project were the field workshops and classroom training. Although costs were controlled by using IDI procurement for travel and accommodation and AFROSAI-Es terms and conditions for subsistence allowances, these events organised for significant numbers of OAGS staff in Kenya and Ethiopia, were relatively expensive. The Project would have gained in efficiency by the organisation of these events in the secured airport area. Although this option was considered feasible in terms of safety, it was not selected as it would have jeopardized the involvement of the resource persons from other SAIs,
Effectiveness
Overall the Project the project performed well against the ‘effectiveness criterion. Although out of the nine outcome indicators, only two targets were fully achieved, very significant progress was made on six other targets and the margin by which the targets were not met were small. On performance audit and the audit of IT risks the planned activities were postponed by agreement of all parties because they were not prioritized given the circumstances currently faced by the OAGS. For many outputs, most of the credit should be given to the staff of the OAGS. Given the emphasis on ownership among IDI Project’s design principles, the importance of the contributions of the OAGS staff is further proof that the Project’s approach has been effective.
Impact and Sustainability
In order to achieve impact, the OAGS needs to ensure that the progress made in the recent years will be sustained. The Project’s efforts to institutionalize a holistic strategic and annual planning and control cycle is expected to have a positive effect on sustainability. As a deliberate choice the Project aimed to involve a great number of staff in the Project’s activities including the most talented key agents and champions in most areas. This very broad approach increased significantly the Project’s costs. it remains to be seen how well the new skills developed through the Project are embedded and utilised compared to an approach with a more strongly targeted focus on the most important change agents in the OAGS.
Recommendations
On the basis of the conclusions, the evaluation suggests a number of ways to improve project design and implementation. These recommendations focus on improving several aspects of the management of personnel costs, shortcomings in the accountability mechanisms, the need for strengthened policy guidance in the Bilateral Policy and the design of a possible second term of the IDI-OAGS Project for the years 2021-2023. With regard to the latter, it is advised to focus the Project’s efforts on deepening the results achieved in the period 2018-2020 at the OAGS at the Federal Government level rather than broadening the scope to other areas such as the SAIs of the Federal Member States of Somalia. Although, the OAGS has made progress in the last years, the progress is still fragile and consolidation of the existing capacity building efforts seems necessary to assure sustainability.
IDI follow-up of evaluation findings and recommendations
The IDI response and actions to follow-up the evaluation is presented here.
Read the full report here.