Frequently asked questions
1. Why was MOPAN created?
MOPAN’s immediate objective was to monitor the performance of multilateral organisations at the country-level against their own mandate. More specifically, it aimed to: • Improve the flow of information on multilateral performance from the country-level to their headquarters; • Strengthen the engagement of the MOPAN members at the country-level in the assessment of multilateral performance; and • Promote a more informed dialogue with the MOs at both headquarters and the country-level about their performance. Over time, MOPAN’s ‘raison d’être’ has evolved. Today, most members see its value in the joint effort to gather, assess and analyse information on Multilateral Organisations’ overall performance (at an organisational level, not only at country-level).
2. What is MOPAN’s status?
The Network was established as an informal entity aiming to exchange information and discuss possible cooperation on the monitoring of multilateral organisations. Its Secretariat was however established through a formal Hosting Arrangement with the OECD, starting in 2013.
3. Who funds and governs MOPAN?
MOPAN is funded by voluntary contributions of all member countries. The network is governed by its main body, the Steering Committee who make all decisions on strategic, financial, and administrative matters.
4. Is MOPAN membership confined to any specific category of countries?
Absolutely not. It had initially started with 6 like-minded members and has since expanded its membership to 18 governments. All MOPAN members provide funding to multilateral organisations. They have a common interest in assessing the performance of the organisations they fund. All members must fully endorse the primary accountability of the multilateral organisations to their respective executive boards and respect their full independence.
5. Does the Hosting Arrangement with the OECD compromise the neutrality of MOPAN and its work?
No. MOPAN remains an independent network and retains full ownership of its branding, strategic orientation and annual work programme. The network is managed exclusively by its Steering Committee. The Arrangement – a Memorandum of Understanding - sets this out very clearly.
6. Why does MOPAN matter?
The main objective of MOPAN is to assess the performance of multilateral organisations in a cost-effective way and with a view to improve their effectiveness. Multilateral organisations are a key player in development. Improving their relevance and effectiveness is crucial for the international community to support the 2030 agenda. With its 18 members that together provide 95% of funding to and through multilateral organisations, MOPAN has the leverage to use the assessments to help the organisations to reform and make them fit for purpose.
7. Has the MOPAN methodology evolved?
Yes. Since the start of MOPAN, 3 different approaches have been developed to meet the needs of members. 2003-2008: Annual MOPAN Surveys Synthesis reports were drawn from Annual surveys conducted at country level. These were used as an investigative, learning and dialogue tool to improve knowledge of multilateral organisations’ activities at the country-level and present perceptions of their partnership behaviour by bilateral donors active at the country-level, engaged with the multilateral organisations and providing funding to them. 2009-2014: Common Approach Broader than the surveys, the Common approach brought in the views of national partners of multilateral organisations and those of multilateral donors, that is, MOPAN members at both headquarters and country level. It took a more systematic look at organisational effectiveness and was organised around the widely recognised balanced scorecard approach that examines four dimensions of organisational effectiveness – strategic management, operational management, relationship management, and knowledge management. A document review was introduced to complement the lines of evidence. 2015-present: Under MOPAN 3.0, the Network is assessing more organisations concurrently than previously, collecting data from more partner countries, and widening the range of organisations assessed. The key elements of this new approach are that it: • assesses organisations in five performance areas: strategic, relationship, operational and performance management, and on their contribution to relevant, inclusive and sustainable humanitarian and development results; • combines four evidence streams: document review, survey, interviews, and consultations, both at headquarters level and in up to 16 countries; • measures performance through a framework based on 12 Key Performance Indicators; • applies a rating for each Key Performance Indicator for all organisations.
8. What does membership entail?
MOPAN members contribute to the Network through an annual financial contribution which covers the cost of the delivery of the assessments and the needs of the Secretariat. Members participate in the meetings of the Steering Committee, 2-3 times per year. Additionally, Members offer in-kind services, e.g., chairing MOPAN, acting as Institutional leads and country facilitators throughout the assessment cycle and participating in specific working groups.
9. How is MOPAN adjusting to the current multilateral landscape?
With its 3.0 Methodology applied since 2015, MOPAN has adjusted its approach to ensure the assessments are relevant in the evolving development context. The key performance indicators were adjusted to cover new normative frameworks (e.g. Agenda 2030). It was set in a generic form so that it can apply to a wide diversity of organisations, thereby reflecting the broader range of multilateral organisations working on development issues.
10. Can MOPAN replace ALL bilateral evaluations?
Although some Governments still conduct their own bilateral assessments for political reasons, the overall number of comprehensive bilateral assessments of multilateral organisations is diminishing. MOPAN’s assessments – with their focus on organisational performance - increasingly provide a solid evidence base that donors can draw upon on to minimise the multiplication of demands on the Multilateral Partners. Today most donors are drawing on MOPAN for performance-based information, with some complementing this with an assessment of the fit of an MO’s policies with their own national policies and strategies.
11. What learning opportunities does MOPAN offer?
A key component of the mission of MOPAN is to contribute to organisational learning within and among multilateral organisations, their direct clients/partners and other stakeholders. MOPAN seeks to embed learning in the assessment process as well as share learning through reports. Specifically, the design of MOPAN 3.0 presents a range of opportunities for learning. This may occur at both the theoretical and the empirical level on different dimensions, including: • The dynamics and characteristics of the multilateral system, such as characteristics and specificities of individual organisations, as well as characteristics of MOPAN and its members • Methodological/evaluative issues in assessments • In-process dialogue on various aspects of the assessments and the findings • Best practices and benchmarking highlighted throughout the assessments . In addition, MOPAN plans, for the first time, to publish a cross-assessment report in late 2017 in which it explores several topics across the 12 organisations assessed in 2015-16.
12. What is MOPAN’s Theory of Change?
IF a multilateral organisation has effective systems, practices in place, THEN its interventions/activities will be more ‘effectively delivered’ AND HENCE delivery will achieve relevant, inclusive and sustainable contributions to humanitarian and development results in an efficient way.
13. How does MOPAN ensure its own accountability and learning?
In line with its strong belief in organisational learning and the establishment of a culture of evaluation, MOPAN underwent an Independent External Evaluation in 2013, which can be found on our website under Other Products. The findings of the evaluation in collaboration with consultation from a range of stakeholders were used to inform changes to the design and implementation of the MOPAN methodology and its operational structure. Beyond this, MOPAN solicits feedback from its stakeholders on a regular basis and has developed its own internal learning agenda to ensure continued relevance and improvement both on its own practices as well as those of the organisations which MOPAN assesses. The next Independent External Evaluation of MOPAN is scheduled for 2017-2018.
14. What is a sequenced assessment approach?
MOPAN agreed to sequence data collection under its MOPAN 3.0 approach, so that each of the four data streams can build on previous evidence. - What is the Document Review? The Document review constitutes the first of the evidence streams. It contains information taken from a strategic selection of publically available documents from the Organisation, and aligns it to MOPAN’s Key Performance Indicators (KPIs) and Micro-Indicators (MIs) of MOPAN 3.0. - How does MOPAN undertake surveys? MOPAN then undertakes an online survey among clients and partners in-country, facilitated by an external survey provider. Finally, the assessment team spends one week conducting interviews with management and consultations with technical staff at headquarters, and in regional offices as needed. - Interviews? Interviews and consultations are informed by document review and survey data, in order to triangulate information coming from the different lines of evidence and fill any remaining gaps in the information collected.
15. How are organisations selected for each assessment cycle?
MOPAN’s Steering Committee selects multilateral organisations for assessment. It does so considering the following criteria: • Organisations chosen are perceived as important and are of interest to a maximum of MOPAN members, • the list of organisations chosen represents a mix of international financial institutions (IFI), UN funds, programmes, specialised agencies, and a balance between humanitarian, development organisations; with operational and/or normative mandates; • if possible, selection takes into account medium-term strategic planning cycles (or equivalent) or replenishment cycles. Their selection follows a dual-track process of 1) applying these criteria to multilateral organisations (a process facilitated by the MOPAN Secretariat); and 2) a discussion among member states. A final decision for the 2017-18 cycle was made at the MOPAN Steering Committee meeting in Paris in October 2016.
16. How are countries in which the survey is conducted selected for each assessment cycle?
Countries are selected based on agreed upon criteria. They include the following aspects: • Countries classified as « developing, ODA eligible » countries by the OECD • Presence of as many as possible of the multilateral organisations selected for assessment • Some balance in terms of geographic location, economic situation (GNI) and other factors such as HDI, GINI and GDI, Fragile situations and Governance indicators. The list of criteria is laid out in Section 10 of the Methodology Manual.
17. What role do MOPAN members and assessment organisations play within the assessment process?
The MOPAN assessments are external and independent. To ensure this, the assessments are conducted by an external service provider which is contracted out following an international bidding process. Meanwhile members of MOPAN are involved in the assessment. One or two of them accompany each review and serve as “Institutional Lead”. They are the face of the Network in that particular review, provide insight and represent MOPAN politically in the process. In the 2015-16 cycle, members of MOPAN also played a facilitating role for the survey and provided names of surveyees in partner countries. At the same time, the assessed organisations internally facilitate coordination on the assessments, and are consulted throughout the process. They support each stage of data collection, either through the provision of relevant documentation, identification of stakeholders at the country level and facilitate interview processes at headquarters. They are periodically provided with opportunities to review and fact-check the draft reports and provide comments where necessary. They do not decide on the ratings and outcomes of the reports.
18. Who does MOPAN engage with apart from its members and organisations?
MOPAN’s primary objective is to deliver high-quality assessments. In line with this technical focus, it links up with other partners in evaluation – such as DAC EVALNET or UNEG. MOPAN also stands ready to contribute to seminars and other events on assessing multilateral effectiveness organised by policy fora such as the Global Partnership for Effective Development Co-operation (GPEDC).
19. What is considered a good MOPAN score?
Scoring and Rating is a critical dimension of the MOPAN assessment process, however, it is important to situate this within a wider narrative of the organisation’s status at the current time. The Scoring and Rating framework for MOPAN 3.0 brings together evidence from all four data streams (document review, survey, interviews and consultations) at analysis stage to comprise an aggregate evidence base per Micro Indicator (MI). The scoring and rating system for each MI comprises a set of elements present. When taken together, these elements demonstrate the presence or otherwise of international best practice. The top end of the 4-point scoring and rating scale reflects the implementation of a full set of elements which, when combined, represent international best practice against that MI. Conversely, the lower end of the scale reflects lesser presence of these elements – and, consequently, weaker performance. Added to this, to embed the commitment under MOPAN 3.0 to ‘function over form,’ an organisation needs to have actually implemented the elements of best practice in order to score more highly. Lower down the scale, an organisation either has fewer of the elements of international best practice in place, or has these in place (form) but not comprehensively implemented (function). The MIs form the basis for a consolidated rating for each KPI, thus enabling a high-level indication of performance across the 12 KPI dimensions. MOPAN does not compare or rank organisations as they have different mandates and various operating modalities.
20. How can you apply the same model to very different organisations?
The MOPAN 3.0 performance areas are broadly applicable to a wide range of development and/or humanitarian organisations. In shifting from the Common Approach methodology (applied from 2009-2014) to MOPAN 3.0 (launched in 2015), and in seeking to produce credible, fair and accurate assessments, the indicators have been formulated to intentionally adopt a generic model. Therefore, all or most indicators should apply to organisations under assessment, unless there is a clear evidence-based case of non-applicability. Furthermore, as adaptation and tailoring under MOPAN 3.0 will be lesser in extent as compared to previous methodologies, indicators will be applied as relevant to the organisation’s mandate and operating practice. For example, development results will be assessed according to the organisation’s own corporate results framework.
21. On what ground and terms do you assess organisations?
The MOPAN 3.0 indicator framework is purposefully: • geared to the Theory of Change, reflecting updated concepts and ‘continuum’ or organisational and development effectiveness • aligned to all five performance areas • a generic model, subject to adaptation; the indicators are designed to be applicable to varying types of multilateral organisations, and varying types of activity/intervention • built to enact the principle of ‘function’ over ‘form’, with the presence of a system, behaviour or practice not sufficient to equate to evidence of ‘effectiveness’ (which is rather reflected in its application in practice)
22. Is there specific data at country-level?
Country data is primarily collected via the survey component of MOPAN, and complemented through the document review and consultations held with regional or country offices. For MOPAN 3.0, country level data is subsumed into the wider evidence base, rather than being individually reported.