Opinion of the Committee of the Regions on the "Report from the Commission Mid-term review of structural interventions Objectives 1 and 6 (1994-1999). Developing a management culture through evaluations: towards best practice"
C 226/64 EN Official Journal of the European Communities 8. 8. 2000 Opinion of the Committee of the Regions on the ‘Report from the Commission “Mid-term review of structural interventions Objectives 1 and 6 (1994-1999). Developing a management culture through evaluations: towards best practice”’ (2000/C 226/17) THE COMMITTEE OF THE REGIONS, having regard to the report from the Commission on the ‘mid-term review of structural interventions Objectives 1 and 6 (1994-1999). Developing a management culture through evaluations: towards best practice’ (COM(1998) 782 final); having regard to the decision taken by its Bureau on 2 June 1999 under Article 265, fifth paragraph, of the Treaty establishing the European Community, to issue an opinion on this subject and to instruct Commission 1 for Regional Policy, Structural Funds, Economic and Social Cohesion and Cross-border and Inter-regional Cooperation to prepare it; having regard to the draft opinion (CdR 389/99 rev. 2) adopted by Commission 1 on 2 February 2000 (rapporteur: Sir Ron Watson, Member of Sefton Metropolitan Borough Council Town Hall, UK/PPE), adopted the following opinion at its 33rd plenary session on 12 and 13 April 2000 (meeting of 13 April). 1. Introduction 2. The Mid Term Evaluation Process 1. 1. A continuous evaluation process for the Structural Funds programmes was established during the 1998 reform and includes ex-ante, mid term and ex-post evaluation of all European Union co-financed programmes. The aim is to ensure the effective disbursement of EU resources, in particular with regard to reducing economic and social disparities in objective 1 and 6 regions. The key element of the Structural Funds is to reduce 2. 1. economic and social disparities within specific national stra- tegies and priorities. As such the mid term evaluations provide a tool for assessing what adjustments should be made to programmes to reflect changing national economic circum- stances. 1. 2. The Commission’s report includes an examination of the overall process, the principal findings, the main outcomes and adjustments made and the implications for future pro- grammes. It also makes some recommendations for the future programming period. The COR wishes to add its own recommendations. This is particularly the case in the Cohesion Four 2. 2. countries, where the CSFs cover most, if not the entire, country, where the example of Ireland resulted in a re-direction of resources to public infrastructure to sustain rapid economic growth. Other exogenous factors such as the earthquake emergency in Italy and the introduction of employment as an over riding EU wide priority also led to shifts in the programmes. 1. 3. The report proceeds to summarise the main elements of more than a 100 mid term evaluations carried out in relation to the 1994-1999 Objective 1 and 6 programmes. These were mostly carried out between 1997 and 1998, thus taking into account the Commission’s own guidelines, which were published in May 1997. 2. 3. In May 1997, the Commission issued its guidelines for priorities for interventions for the Structural Funds post the mid term. These had been drawn up on the request of the member states. They set out the changing EU wide policy context and stipulated what elements should be covered in the mid term evaluations. 1. 4. The mid term evaluation is a shared responsibility between the member state, regions and the European Union and on average 0,1 % of the total budgets have been utilised for these evaluations. 2. 4. The mid term evaluations have tended to focus on financial data supplied by the relevant monitoring systems. At the stage of the mid term evaluation, little information about the wider and longer-term impacts of the Structural funds will not be available. 8. 8. 2000 EN Official Journal of the European Communities C 226/65 3. Lessons from the Mid Term Evaluations 3. 1. The extent of mid term evaluations (100+) bear witness to the importance that both the Commission and the member states attach to the evaluation process. A key point of the mid term evaluations is to establish the extent to which a programme lends itself to evaluation and to establish the subsequent methodology. The main aims are to assess the degree to which the programme has met its objectives at the half way stage, determine the initial impact of the interventions, and propose, where necessary, recommendations to improve the management of the programme. It is clearly essential that the evaluators are independent 3. 2. of both the managing authorities and the executive bodies. The report does, however, give two examples of a highly effective internal evaluations in Italy (the Evaluation Unit of the Budget Ministry in Italy, and Ireland). Whatever methodologies are used, evaluations must be based on the initially agreed criteria and objectives of the programmes to ensure that progress (or lack of it) can be measured and appropriate action taken. 3. 3. In some cases partnerships may also wish to engage with suitably qualified research institutes, which may be located within higher education institutions or the social partners. In many cases, representatives of such bodies are members of the programme monitoring committees, yet possess the technical knowledge and competence to undertake evaluations and assessments. The COR believes it would be wrong to rule this out, provided clear rules of engagement and separation of duties exists. In general, Programme Monitoring Committees have 3. 4. been keen to take on the responsibility for the implementation of recommendations from the evaluations and appropriate sub structures have been established to deal with the evaluations, both in terms of managing the evaluations and taking forward their recommendations. 3. 5. The involvement of regional authorities has varied, depending on the political and institutional specificities of the individual member states. There appears to have been more direct involvement of regional authorities in case where the Structural Funds have been delivered through SPDs rather than CSFs, which by their nature tend to be national programmes. The Commission’s report states that 3. 7. it has been possible to assess some of the key macro-economic impacts of the CSFs in the larger member states (the Cohesion Four, Italy and Germany) at the mid term. There is a clear positive impact on both economic growth and employment from the Structural Funds interventions, although more emphasis needs to be placed on the use of integrated models, which examines the impact on both supply and demand. Generally speaking, the impact on employment has 3. 8. been less than on economic growth in percentage terms, due to an assumption of increased productivity where companies become more efficient. There is clearly an inherent tension between achieving economic growth and the creation of jobs and while the Structural Funds have to have a proper regard to the employment effect, economic efficiency of the investments should be an overriding principle in the assessments. 3. 9. For the smaller programmes (generally SPDs), results from macro economic modelling are harder to achieve because of the inter linkages with the wider economy. The report quotes the Belgian experience as an example of best practice, where the results showed a significant risk of a return to slower growth rate after the completion of the SPD. 3. 10. The Commission’s report also includes a section on the effectiveness of the interventions, including a review of indicators used. the evaluations led to a substantial review of agreed physical and impact indicators. Efficiency has not been covered in any great detail. In many case, 3. 11. The Commission’s report accepts that it is difficult to present an overview of the outputs and results from across the programme because programmes are not comparable across regions. Clearly this is to do partly with the lack of a common core set of indicators agreed by the Commission and member states prior to implementation. This aspect also requires further analysis as the implication could be that indicators are changed to suit is clearly acceptable to do so where economic and social circumstances require this, this must not be a common practice. local circumstances. While it 3. 6. In general, evaluation reports have been of a high quality and standard, showing the progress, which is being made, in this important field. This has been partly driven by the MEANS Programme, which aims to facilitate evaluation work across Structural Funds. The Commission’s report expresses some concerns relating to the highly academic nature, narrow scope (management rather than results and impacts), lack of independence and lack of support from managing authorities of some of the evaluation reports. 3. 12. A key feature of the mid term evaluations were the assessment of the monitoring systems and project selection criteria. The current programmes contain a significant develop- ments in the use of quantified indicators. However, a number of programmes still show a weakness in information available and, more generally, related to the quantification of indicators. Clearly, many programmes do not fully utilise information available as a proper management tool, combining financial and physical outputs, results and impacts. C 226/66 EN Official Journal of the European Communities 8. 8. 2000 3. 13. In some cases, sophisticated project selection criteria have been developed to promote transparency for the selection process. Many mid term evaluations have questioned the effectiveness of such systems as having transparent scoring systems may not lead to the best results in terms of project selection. 3. 14. Most mid term evaluations have been a useful tool and source of knowledge for decision-makers. In most cases, programme managers have implemented recommendations of the evaluations. An essential part of a successful evaluation has been the substantial input from the partnerships themselves to improve the delivery of the programmes. to support a re-allocation of The mid term evaluations have been used, in most 3. 15. case, resource within the CSFs/SPDs. In almost all case, the Commission report shows that these financial re-allocations were made without affecting the strategic priorities of the programmes. In some case, new measures were introduced. 3. 16. More generally, the Commission’s report concludes that it has not always been easy to assess the extent to which EU wide priorities have been addressed. Generally, more emphasis has been given to employment, environment and sustainable information technology, and development, reflecting shifts over time. 3. 17. In its conclusions, the Commission report stresses the need for monitoring procedures to address financial as well as programme results and impacts. This includes clear quantified targets and indicators for monitoring and evaluation, more appropriate project selection criteria, simplified management procedures and synergies between the difference funds. 4. Future Challenges 4. 1. The Commission’s report emphasises the need to build on the best practice which have been established by the mid term evaluations. These include the soundness of the evaluation, the involvement of the partnerships, the organis- ation of the evaluations and the feedback role in supporting programme decisions. 4. 2. Good evaluations are an important tool for good management of the programme. This in turn depends on adequate structures for managing evaluation activities within the monitoring systems, the partnership, development of better integration between financial and physical indicators, adequate quantification of baselines, progress in the area of evaluation methodology through the MEANS programme, and dissemination of best practice. The Commission is proposing the publication of a guidance document on methodological issues, including an indicative list of indicators. 5. Recommendations 5. 1. While the Committee of the Regions (COR) acknowl- edges the value of EU guidelines for the mid term evaluations, it is concerned that these should be available to programme managers early to inform the process more clearly. 5. 2. The next programming period sees the introduction of a performance reserve and the COR wishes to stress the need for early, clear guidance on a core set of EU wide indicators, against which performance will be judged. Every effort should be made to make these available prior to the start of the programmes. 5. 3. The COR expresses its concern that more clarity is needed in the design of evaluations to enable the production of comparable reports across the European Union as to the effectiveness of the Structural Funds. 5. 4. The COR would welcome a more pro-active approach to the sharing of best practice among evaluation and pro- gramme executives to ensure that programmes are designed and delivered in such a manner as to facilitate evaluation to establish best practice. 5. 5. The COR would also welcome the development of a set of core common indicators which focus on results and impacts rather than purely activities, commitment and spend to facilitate a comparison of performance across the Structural Funds. These indicators must cover not just GDP but also the wider economic, social and environmental impact of the Structural Funds, particularly in relation to the new Objec- tive 1. 5. 6. This would also enable a mid term evaluation at the mid term rather than during the fourth year of implementation, which has generally been the case during the current program- ming period. The COR believes that the impact of the mid term evaluations have been reduced because of the time lag between evaluations and the implementation of their respective recommendations. 5. 7. The COR welcomes the statement that the mid term evaluation is a shared responsibility between the member states and the Commission. In the light of the recently agreed general regulations for the 2000-2006 programming period, it believes that this principle should be extended to include regional and local authorities. 5. 8. The COR would also welcome the opportunity to work with the regional Policy Committee of the European Parliament to promote a culture of rigorous evaluations and monitoring by member states to assess the effectiveness and efficiency of the Structural Funds interventions. 8. 8. 2000 EN Official Journal of the European Communities C 226/67 5. 9. While COR recognises the need for independent evaluators, the COR would be concerned if this became a formal requirement as surely the aim is to improve programme In cases, say, where management and effective delivery. technical knowledge and competence resides within higher education institutions, technology institutes, etc. , such bodies should not be prohibited per se from carrying out evaluations during the programme period. Clear rules should be established so that selection of any evaluators is transparent and under- stood within the partnership. 5. 10. The COR believes that better and more timely dissemination of best practice is a must for the next program- Brussels, 13 April 2000. ming period. The Means Programme provide a vehicle for this dissemination, including the development of tool kits and best practice handbooks. It is essential that such activities are addressed not to evaluators but also the managing authorities. just 5. 11. While the COR welcomes the use of macro-economic modelling as an evaluation tool in the larger member states, it is essential that models which integrate the supply and demand side are used in the next generation of programmes. The President of the Committee of the Regions Jos CHABERT