Our Methodology
MOPAN 3.0 applies four methods in its assessments in a sequenced approach which is rated in an evaluative manner at the end of the assessment process:
1. Document review including meta-synthesis of evaluations: Key lines of evidence for assessing both organisational and development effectiveness.
2. Survey: The survey dimension has been a foundation of the assessment approach since the formation of the Network, providing valuable perception data for members. Survey data enhances the value of the documentary review by providing insights into how partners of multilateral organisations have experienced co-operation, both at headquarters and decentralised level.
3. Interviews: At organisations’ headquarters and in countries, to: deepen and interrogate the evidence from the document review; confirm or clarify trends or tentative findings emerging; provide information about an organisation’s on-going reform agenda and any key changes since documentation was analysed; provide contextual insights; generate new evidence in areas where documentary and survey evidence is lacking.
4. Consultations: Mostly with the organisations’ staff at the headquarters level, for contextualisation, triangulation and validation.
MOPAN 3.0 takes a sequenced, holistic and systematic approach, with emphasis on country/regional level information:
Sequenced approach: A sequenced approach is applied through a ‘building blocks’ model. Each layer of evidence is generated through the sequential assessment process, informed by, and building on, the previous one. Each layer of information in the sequence of the assessment process informs the next, as far as feasibility permits. The survey, for example, will be informed if possible by findings from the document review, and interviews and consultations will be shaped by findings from both the document review and, if possible, survey.
Holistic approach: This involves applying multiple lines of evidence to as many indicators as possible, to ensure that findings are based on as solid an evidence base as possible. In particular, this will be reflected at the analysis stage, where findings from the different evidence streams are brought together to generate triangulated findings.
Systematic approaches will involve:
- For data collection: the use of structured frameworks for each evidence line (document review, survey, interviews; and consultations)
- For data analysis: structured analysis at aggregate level, applying the different lines of evidence within a composite analytical framework
Country/regional level information: The emphasis in MOPAN 3.0 on ‘function over form’ requires the ground-truthing and testing corporate statements and intentions on field-level practice, particularly for development and/or humanitarian-focused multilaterals. Country/regional level evidence will therefore be applied against all relevant indicators, through the different data streams including document review and survey information particularly. Analysis/reporting will incorporate this information throughout, in aggregate form, with relevant examples provided where appropriate. Analysis will not result in an overall rating for an organisation’s country level performance, but individual data pieces gathered from country level will be collated and stored as part of the assessment.
The indicators: MOPAN’s analysis takes place against a generic Indicator Framework aligned to the five Performance Areas set out under “Our approach”. This generic framework is somewhat adjusted to each assessed organisation depending on its mandate, structure and operations.