SAI Story
Bhutan
14/09/2023
In a constantly changing world, any organisation needs to continuously think about how to strengthen its capacities, processes, and products to ensure it provides value for its key stakeholders. How did the Royal Audit Authority of Bhutan (RAA) ensure continuous strengthening of its office over a period spanning the past eight years?
The former Auditor General, Dasho Ugen Chewang, made a forwardlooking decision in 2014 to be one of the first SAIs to conduct a SAI PMF assessment, applying the SAI Performance Measurement Framework. This was one of the sparks leading to a positive upward spiral in the years to come. The purpose was to:
The assessment was conducted by a team from the INTOSAI Donor Secretariat within the INTOSAI Development Initiative (IDI) and the Office of the Auditor General, Norway (OAGN). Mechanisms were built into this approach to ensure internal capacity building. An internal team was established to shadow the peer assessment team, with the objective of achieving a better understanding of the SAI PMF methodology.
Importantly, the assessment provided input to the further improvement of the SAI PMF framework to the benefit of the INTOSAI community.
Committed leadership and staff tackled the results of the assessment head-on, appreciating the strong performance areas identified while accepting the weaknesses. The RAA used this insightful knowledge constructively to develop the SAI Strategic Plan (2015-2020) and as a tool to facilitate communication with donors. By demonstrating its dedication to improve, the RAA received further donor support from the World Bank and the Austrian Development Agency where the SAI PMF results ensured support was targeted towards the real needs of the SAI. The funds were used to address gaps in the audit methodologies to further strengthen the implementation of the ISSAIs.
Based on the recommendations of the SAI PMF assessment conducted by OAGN in collaboration with IDI in 2014, the RAA prioritized ISSAI implementation as a part of its strategy for 2015-20. The report indicated the need to further strengthen the audit processes and documentation of audit work performed.
The repeat SAI PMF assessment reassured us that the targeted efforts to strengthen our office have had positive effects. We could clearly see performance improvements in key areas. Importantly the results have provided us with a direction on capacity gaps that still needs to be addressed moving forward.
The journey of the RAA over the past 8 years demonstrates how a baseline SAI PMF assessment and a repeat assessment with a clear purpose are powerful tools. It has provided the RAA with the necessary information to analyse the situation, address capacity gaps, monitor performance change, and revise its strategic direction to ensure value is provided to its stakeholders.
Towards the end of the strategic planning period, the Leadership of RAA acknowledged that there was still a need for further capacity building in strategic management, hence the decision in 2019 to enrol in the IDI Initiative on Strategy, Performance Measurement and Reporting (SPMR). The initiative has been implemented in collaboration with the ASOSAI Capacity Development Administrator (SAI Japan) and co-funded by the Swiss State Secretariat for Economic Affairs (SECO).
An important first step in the initiative was to conduct another SAI PMF assessment, making RAA one of the first SAIs to conduct a repeat assessment. The purpose was two-fold: firstly, to develop the new SAI Strategic Plan (2020-2025) and secondly, to monitor performance change over the previous strategic planning period by comparing the results with the baseline results from the 2015 assessment. This comparison and analysis enabled the SAI to demonstrate performance improvement in several key areas as well as the current capacity gaps.
Leaning on the specialist capacity built within its office over the past years, the RAA was able to conduct the SAI PMF as a self-assessment, using a team of internal staff with suppor t from IDI.
The RAA has put words into action and has published its SAI PMF assessment reports with its first in 2015. Dasho Tashi, who was appointed the new Auditor General under the Constitution of the Kingdom of Bhutan on 14th October 2020, followed suit by deciding to publish the repeat assessment report. This demonstrates a continuous and admirable dedication to being a model organisation and promoting accountability and transparency by sharing both strengths and weaknesses, for the benefit of the office, key stakeholders an d as an inspiration for the INTOSAI community.
Although you can’t fully predict the future, you can strive to have strong governance processes in place to ensure regular review of the situation and a change of course where needed.
In 2014, SAI Costa Rica became one of the pioneers of applying the SAI PMF methodology. Their findings then fed into a series of capacity development projects.
The projects targeted improvements in the auditing process, including stronger alignment to the ISSAIs, a sharpened risk-based selection approach, and changes to the audit training plans. The SAI also focused on better internal and external communication and started preparing an annual evaluation of the government’s internal control system. Through these changes, the SAI improved its image and role as a model institution, which adds value to public sector efficiency, transparency and accountability. The 2014 assessment also served as a crucial input to further develop the SAI PMF tool.
In 2019, the Head of the SAI, Ms Marta Acosta Zuniga, commissioned the team to carry out a repeat assessment using the final approved 2016 version of the SAI PMF framework.
Comparing the findings with the situation observed in 2014 revealed that the SAI had made significant progress in many of the initially identified areas of improvement, and particularly in the audit and human resource areas. The SAI demonstrated how each Costa Rican colón (CRC) invested in its audit work resulted in 2,30 CRC of public finance efficiency gains, through savings and recoveries to the value and benefit of Costa Rican citizens.
Among those was the need for further refinements in the audit methodology, the lack of specialised knowledge in some emerging areas such as big-data audits and public sector innovation, and internal communication and technology in support of the audit process. The SAI subsequently joined IDI’s Strategy, Performance Measurement and Reporting (SPMR) initiative, which builds on the SAI PMF findings to support SAIs in developing robust strategic management processes. The new strategic plan of the SAI developed under SPMR underscores the SAI’s role in supporting digitalisation in the public sector, which has become a critical priority in the context of the COVID-19 pandemic.
In fact, the SAI of Costa Rica was just the second SAI worldwide to do a repeat assessment.
The changes between the pilot and the final versions of the SAI PMF framework meant that comparing performance for the different indicators was not always possible. The Costa Rican SAI PMF team undertook a detailed mapping and analysis to assess comparability between the SAI PMF versions, which fed into the design of the official repeat assessment methodology by IDI in 2020.
The Costa Rica assessment also uses five custom-designed performance indicators related to its special functions to investigate public service officials’ liability and carry out ex-ante audits. This was the first time a SAI developed unique indicators, which the SAI PMF methodology foresees as a possibility to reflect the diversity of SAI practices. Both innovations had a strong resonance amongst the other SAIs of the OLACEFS region that were embarking on a repeat SAI PMF assessment and so were adapted by various other SAI PMF assessment teams. Both assessments were also peer-reviewed by the SAI of Brazil, with the financial support o f the Inter-American Development Bank.
The publication has clearly demonstrated that even when the assessment reveals some shortcomings, the SAI can use the opportunity to reflect in the publication on how it has begun to, or will address the findings. In the words of the Head of the SAI Costa Rica, Ms Marta Acosta Zuniga:
The role of the SAI of Costa Rica as a supporter and forerunner of SAI PMF has been impressive.
From testing the pilot version, through methodological work, to publishing both assessments and showing how SAI PMF results feed into strategic planning and support performance improvements and a meaningful contribution to the public sector, the Costa Rican example shows how experimentation and innovation start internally, only to spill over to the national and even regional levels.
The IDI, in collaboration with the CAROSAI Secretariat, recently conducted a SAI PMF Review Workshop in the CAROSAI Region. This workshop was organized for the SAIs participating in the second round of the IDI-CAROSAI SAI PMF Facilitation Programme. The workshop took place in Curaçao from 3 to 7 July 2023 and was attended by 12 participants. The participating SAIs included Curaçao, Grenada, Montserrat, and St Vincent and the Grenadines.
The main objective of the review workshop was to assess the progress made by the SAIs in their SAI PMF self-assessments, which began with fieldwork in February 2023. Furthermore, the workshop aimed to provide the assessment teams with the necessary knowledge and skills to conduct a root cause analysis and produce a qualitative assessment of the SAIs’ performance. This review workshop plays a crucial role in finalizing the draft performance reports.
Going forward, the assessment teams are expected to revise their draft performance reports based on the guidance provided during the workshop. Additionally, they are required to conduct a root cause analysis of performance and draft a section highlighting key findings and observations on the SAI’s performance and impact.
According to the agreed roadmap, the final performance reports, including an independent review statement issued by the IDI, are anticipated to be completed by 31 October 2023. To ensure high-quality and timely completion, the IDI will continue to offer remote support to the assessment teams throughout the remaining phases of the assessment process.
The objective of the course is to develop capacity for conducting a SAI PMF repeat assessment.
Welcome!
The objective of the course is to develop capacity for conducting a SAI PMF repeat assessment. Having the right skills in place will enable a high-quality assessment that will contribute to the capacity development and further strengthening of your SAI. The main target group are SAI PMF assessors, but the course can also be relevant for others that want to enhance their SAI PMF skills.
The estimated time to complete the course is 2-3 days. A certificate can be provided upon completion. To request a certificate please send an e-mail to saipmf@idi.no once you have finalised the entire course
See below a key message from the Director General of IDI, our repeat assessment video, description of prerequisites necessary for taking the course and link to access the course through LMS. “The importance of a SAI PMF repeat assessment can not be overemphasized. As the operation lead on the global roll-out and implementation of SAI PMF, the IDI encourages more SAIs to conduct a repeat assessment as a means to monitor progress in performance”. – Mr Einar Gørrissen, Director General of IDI – What is a SAI PMF repeat assessment? It is an assessment of a SAI which has previously assessed its performance using the SAI PMF. A repeat assessment is particularly useful when a SAI wants to monitor its progress in performance over time, for instance, its strategic plan period. Who is the target group of this course? It is open to all interested individuals and can be taken at any time convenient to the learner. This course is designated to be taken by anyone planning to conduct a repeat assessment, whether SAI staff, consultants or others. If you are currently not planning to conduct a repeat assessment, it can still be helpful to take the course to enhance your SAI PMF expertise and acquire a better understanding of the specificities of a repeat assessment. It is important to note that this course is intended to cover only part of the SAI PMF methodology related to key aspects of a repeat assessment. It is a prerequisite that you have previous SAI PMF knowledge and already have a good understanding of the following subject areas: SAI PFM process SAI PMF scoring methodology SAI PMF framework and the areas that are being assessed How to analyse the SAI PMF results through a root cause analysis and presentation in the SAI PMF performance report. Link to the course The self-learning course is now available on IDI LMS platform: https://lms.idielearning.org/course/view.php?id=493 Information on how to access the course can be found here
The SAI PMF Team in the IDI has developed a SAI PMF repeat assessment tool kit to provide advice to assessment teams on how to conduct a SAI PMF repeat assessment and how to compare the results of two assessments.
2nd Wave: What is a SAI PMF repeat assessment? The SAI PMF Team in the IDI has developed a SAI PMF repeat assessment tool kit to provide advice to assessment teams on how to conduct a SAI PMF repeat assessment and how to compare the results of two assessments. The repeat assessment tool kit will assist assessors conducting a SAI PMF repeat assessment to understand how the performance of an SAI has changed over time. The tool kit comprises a guidance document and annexes for registering scores and analysing performance change. The guidance document covers limitations of a repeat assessment, key principles and process for conducting a repeat assessment. Further, the guidance document describes a step-by-step approach on how to compare the results of two SAI PMF assessments, particularly where the pilot version of the SAI PMF was used in the first assessment and the 2016 version in the repeat assessment. The main changes between the 2016 version and two earlier versions, namely pilot version and version 3.1, have been highlighted in the guidance document. It is envisaged that moving forward, more SAIs will conduct a SAI PMF repeat assessment for the purpose of monitoring progress over time. This will necessitate comparison of results by assessment teams. To supplement the SAI PMF repeat assessment tool kit, SAIs intending to conduct a repeat assessment are free to contact the SAI PMF Team in the IDI for any technical assistance with the process. SAI PMF Repeat Assessment Toolkit can be found here: https://www.idi.no/work-streams/well-governed-sais/sai-pmf/resources
A SAI PMF repeat assessment is a performance assessment conducted by a SAI which has previously assessed its performance using the SAI PMF
The Supreme Audit Institutions Performance Measurement Framework (SAI PMF) is an INTOSAI tool that provides Supreme Audit Institutions (SAIs) with a framework for voluntary assessments of their performance against the INTOSAI Framework of Professional Pronouncements (IFPP) and other established international good practices for external public auditing.
The SAI PMF Team is conducting a SAI PMF eLearning basic training course for SAIs in the CAROSAI region. The training, which runs, from 3 November – 2 December 2022 launches the second IDI/CAROSAI SAI PMF facilitation programme in the region, following the successful delivery of the first facilitation programme in the period 2020/2021.
A SAI PMF facilitation programme entails a regional coordinated approach to implementing SAI PMF where a group of SAIs simultaneously conduct their SAI PMF assessments while receiving support from resource persons and IDI staff throughout the process. In attendance at this training are 11 participants from the 4 participating SAIs.
The objective of the IDI-CAROSAI SAI PMF facilitation programme is to support SAIs in CAROSAI to assess their performance using the SAI PMF framework in line with strategic priority No. 2 of the regional strategic plan 2017-22. SAIs can then use the assessment results to formulate capacity development projects that will improve their efficiency and effectiveness in the delivery of their mandate.
Visit the IDI website for more on SAI PMF eLearning.
 |
 |
|
