Background Technical Document II: Note on Documentary Evidence
- International Monetary Fund. Independent Evaluation Office
- Published Date:
- August 2010
1. As noted in the main report, the evaluation used three main sources of primary evidence—surveys, interviews, and internal documents. This background paper focuses on the internal documents. It has three sections. The first describes the documents themselves, as well as how they were obtained. The second sets out how the documents were used in the context of the evaluation’s work on the 49 sample countries, which covered the entire evaluation period. The third discusses the evaluation’s cross-country document review of selected issues in interactions, which focused on the last two years of the evaluation period.
2. This examination of the internal IMF documentary record was an important pillar of the evaluation. It helped the team to clarify, validate, and/or reject hypotheses that arose in the course of its work, and played a key role in its ability to triangulate across evidentiary sources and draw conclusions. This said, the evaluation team clearly recognizes that the documentary evidence is but a partial record of the relationship between the IMF and member countries, not least because many IMF interactions are not reflected in the written record. The evaluation team also recognizes that the evidence must be interpreted carefully, mindful that most documents were written for another time and another purpose, often under strict word limits.
II. Documentary Evidence
3. This section describes the document identification, request, and collection process. At the outset of the evaluation, the team identified a set of documents necessary to understand the nature of the staff’s interactions with the authorities and other stakeholders, and how these interactions were managed. This set consisted of documents related to departmental guidance and policy on interactions, and to country-specific documents related to interactions. The team also examined Article IV consultation papers and program documents prepared for the Board, and the related Board minutes. Most of these background documents were available to the IEO from the Fund’s internal institutional repository (some of which were also available on the IMF’s external website). In a few cases requests were made through the Fund Secretary to obtain confidential documents. The relevant set of identified documents covered the entire evaluation period (2001–08).
Box 1.Core Set of Documents Reviewed for Sample Countries
1. Briefing papers/memoranda for Article IV, use of Fund resources (UFR), Policy Support Instrument (PSI), or other staff visits, including cover notes.
2. Correspondence with, and comments by, management related to briefing papers/memoranda for Article IV, UFR, and other staff visits.
3. Back-to-office reports for Article IV, UFR, PSI, or other staff visits.
4. Correspondence with, and comments by, management related to back-to-office reports for Article IV, UFR, PSI, or other staff visits, when available.
5. Cover notes to staff reports.
6. Correspondence with, and comments by, management related to draft staff reports.
7. Surveillance Agendas.
8. Staff reports.
9. Executive Board minutes.
4. The IEO requested the staff documents from the relevant area departments, and the request was fulfilled through the combined efforts of the departments and the Strategy, Policy, and Review Department (SPR). Initially the IEO worked with a contact person within each area department to establish efficient ways of transmitting information and avoiding duplication wherever possible. Many documents were sent electronically, and when an electronic version was not available, hard copies were sent. The team also collected documents from SPR when they were not available from the area departments. For all sample countries, the team gathered a core set of documents, as summarized in Box 1. IEO staff reviewed all the documents delivered for the 49 case study countries and prepared inputs based on them for both the country case studies and the cross-country analysis discussed below.
III. Country-Based Document Review
5. The most important use of the documentary evidence was to underpin the evaluation team’s country-based analysis. This section highlights issues that arose in the course of this analysis, organized around the three country groups: advanced economies, emerging market economies, and PRGF-eligible countries. In conducting this analysis, IEO staff reviewed the documents for the 49 sample countries to gain an overall perspective on interactions during the full review period and to study specific issues, in both cases building on the survey and interview evidence. Particularly important for the review were briefing papers and back-to-office reports, along with, in some cases, one-off memoranda between an area department and either management or a functional department on specific issues.
6. In its review of the documentary evidence related to advanced economies, the evaluation team paid particular attention to three issues. The first, as in all country cases, was the exploration of issues on which the dialogue was conducted over the years, and especially the identification of any recurring disagreements and how they were resolved (or not). The second was any intervention by management in the resolution of any disagreements, especially important in view of the survey findings that staff provided cautious assessments, that staff did not feel they would get sufficient backing from management if tensions arose with the authorities, and the degree of satisfaction of the authorities with the outcomes of complaints against staff. The third was the question of outreach, especially in light of survey evidence (from both the authorities and staff) and staff interview evidence that suggested that authorities of some advanced economies discouraged staff from holding press conferences on their mission findings. The documents were also reviewed for evidence on the analysis of international policy coordination and spillovers, the use of cross-country analysis, and the macro-financial sector linkages, as set out in the companion paper on the advanced economies.
7. In reviewing documents related to emerging economies, the evaluation team focused on program dynamics, country strategy, and outreach, inter alia. For program dynamics, the team concentrated on countries where transition to a surveillance-only relationship was under way or anticipated and/or where the design and/or flexibility of conditionality had surfaced as an issue in the evaluation interviews. These considerations also raised questions of country strategy, such as (i) how staff considered possible political economy constraints to reform, and the implications for sequencing in its policy advice; (ii) how they discussed with senior staff and management the evolution of the country relationship in post-program and other surveillance-only economies, including with respect to global imbalances and exchange rate policies in systemically important economies; and how they perceived the Fund’s value added to the authorities. The team also examined the documents for different facets of outreach, especially with respect to any noted tensions between the Fund’s role as confidential advisor and its evolving communications strategy, which stressed outreach as a way of influencing public debate. The documents were also reviewed for evidence of discussions about excessive staff turnover, a recurring theme of interviews with authorities from small states and other emerging economies in the evaluation sample.
8. As in its analysis of the documentary record for the advanced and emerging economies, the evaluation team studied the documents on PRGF-eligible countries in the first instance (i) to understand the nature of IMF-country interactions over the evaluation period and (ii) to follow up on specific issues raised by the authorities (and by staff in a few instances) in the evaluation interviews and surveys. Of particular importance in the analysis of interactions with the PRGF-eligible countries was the exploration of the theme of flexibility as it played out in the different country cases, for example in connection with conditionality-related program interruptions, delays in HIPC completion, and the development of the PSI, as well as tensions related to safeguards assessments and misreporting. The documents for the PRGF-eligible countries were also studied to see the extent to which they set out a coherent strategy for Fund engagement over time, taking into account (i) the political economy constraints to reform that the authorities faced, and how they affected the sequencing of policy actions and outreach; and (ii) relevant implementation capacity constraints and priorities for technical assistance—as these two sets of issues were recurring themes in interviews with authorities of PRGF-eligible countries.
IV. Cross-Country Documentary Review
9. Supporting and complementing this country-based review of the documentary evidence, which covered the full evaluation period, the team also carried out a cross-country analysis of specific aspects of interactions, more narrowly focused on 2007–08. In part, this exercise was motivated by the need to develop and ensure the usage of common standards for the document review by the different IEO staff and consultants engaged in the review work. But it also allowed for the exploration of several issues of interest looking across the entire sample group.
10. This cross-cutting analysis involved a review of 116 surveillance and use of Fund resources “events” over this two-year period for the 49 countries in the evaluation sample using a consistent approach. “Events” included the set of activities from missions leading up to Executive Board meetings, for both Article IV consultations and program discussions (negotiations and reviews). Reviewed documents included briefing papers and back-to-office reports (and, where available, comments from management); cover notes and clearance memoranda for draft staff reports sent to management (including, where available, the Surveillance Agendas); final staff reports; minutes of Executive Board discussions; and any press statements.
11. In undertaking this analysis, the evaluation team examined several issues, including:
- Clear identification of the authorities’ views as distinguished from the staff’s views in the Fund’s internal documentation: The analysis found that in about two-thirds of the studied surveillance cases, there was clear attribution of the authorities’ views in the documents. For the use of Fund resources, the corresponding share was lower (about one-third), which may reflect in part that agreement between authorities and staff is reached before a request for the use of Fund resources is presented in the documents to the Board.
- Staff flexibility in negotiations and discussions, in light of complaints raised by some authorities during the evaluation interviews: For surveillance activities, the analysis found few substantive changes in staff positions between (b) Staff flexibility in negotiations and discussions, in light of complaints raised by some authorities during the evaluation interviews: For surveillance activities, the analysis found few substantive changes in staff positions between the briefing paper and the back-to-office report, though there were some examples of briefing papers with flexibility written into the staff position to account for political economy concerns. Flexibility in this narrow sense was found to be somewhat more common in the context of programs—where either a change in position was flagged in the back-to-office report as a result of the discussions with the authorities, or the briefing paper had been drafted at the outset with flexibility embedded in the initial policy stance. However, this analysis is partial as it does not capture the extent to which the IMF staff position evolved over time.
- The content of the Surveillance Agendas, a new initiative launched in late 2006: The analysis found that Surveillance Agendas covered economic issues without linking the various elements of the Fund’s work program into an overall strategy, including their sequencing and links to the proposed outreach actions. Most did not contain specific/explicit references to authorities’ views or disagreements between the authorities and staff; fewer suggested that they had been discussed with the authorities. Technical assistance priorities were often listed, but without the broader context of the capacity building goals and complementarities with other providers of technical assistance.