Report from the Commission - Mid-term review of structural interventions - Objectives 1 and 6 (1994-99) - Developing a management culture through evaluation: towards best practice
COMMISSION OF THE EUROPEAN COMMUNITIES Brussels, 07. 01. 1999 COM(1998)782 final REPORT FROM THE COMMISSION MID-TERM REVIEW OF STRUCTURAL INTERVENTIONS OBJECTIVES 1 AND 6 (1994-99) DEVELOPING A MANAGEMENT CULTURE THROUGH EVALUATION: TOWARDS BEST PRACTICE si tSHlfc. ' s»ëf-. mm i$fja£ m •5 &£$ïk E X E C U T I VE S U M M A RY The Mid-Term Review: raising effectiveness 1. The Mid-Term Review of Objective 1 and 6 programmes is a process informed primarily by the various Mid-Term Evaluations carried out at programme level. It also takes account of other elements including national policy contexts, the EU's own priorities and other relevant strategic analyses. The principal aim of the Mid- Term Review is therefore a thorough and systematic assessment of the programmes, leading where necessary to the adjustment of priorities and reallocation of budget for each programme. 2. The present Mid-Term Review was able to draw upon a large number of evaluation exercises (more than a hundred) and was greatly facilitated by discussions in partnership with national authorities. 3. The Mid-Term Review covered virtually all Objective 1 and 6 CSFs and SPDs. The importance of this process varied from one Member State to another. In general, it has enabled a more effective deployment of resources and provided support for specific strategic priorities. Mid-Term Evaluations have proved useful to inform programme adjustments 4. Mid-Term Evaluations are not an end in themselves but a means to improve the quality and relevance of programming. They involve substantial amounts of human and financial resources as well as, in some cases, considerable preparation and discussion involving programme partners. 5. These evaluations were not implemented simply so as to comply with existing Community programming rules. They were also introduced because of a growing interest in evaluation on the part of national or regional authorities. Increasingly, these authorities are coming to see evaluation as a means of improving the management of Funds and maximizing the results obtained. 6. Mid-Term Evaluations offer an important means of assessing the desirability of particular changes to original programme objectives. Programming of structural interventions can be influenced by changing external factors and by changes in the general policy framework. Factors such as these can necessitate a review of strategic priorities and lead to more systematic and effective programming. 7 Benefits arising from the present evaluation include enhanced value for money and a revised balance of priorities matched to changing needs and circumstances. These should be seen as part of a more fundamental learning process, one which includes benefits which might not be fully recognised in the short term such as the increase of existing evaluation capabilities and improved co-ordination between national and regional authorities. Assessing performance is relatively straightforward 8. Compared with previous exercises in respect of earlier programming periods, the in the use of Mid-Term Evaluations revealed significant developments quantitative indicators as a means of monitoring the implementation of interventions, assessing their impact and determining progress towards their stated objectives. Although the development of indicators has been uneven to date, elements of 'good practice' identified in certain programmes suggest that measuring and assessing performance is relatively straightforward, and does not necessarily require supplementary work on the part of national administrations. 9. The use of formal performance indicators for measuring progress in relation to the main objectives and priorities of a programme can be of greater use than has hitherto been recognised. This should lead to more efficient management capable of delivering demonstrable improvements in terms of actual outputs and results. 10. Assuming that Member States will have to carry out ex-ante evaluations of future programmes, it is essential that programming documents contain quantified objectives and indicators relevant to the Mid-Term Review process. The availability of this type of information can be genuinely useful for the management of the programmes in question and can serve as an objective basis for judging performance. It will, following proposals by the Commission in the draft Structural Fund Regulations, determine the allocation of a reserve to programmes according to their own internal performance. 11 As regards the operational framework, monitoring systems will have to ensure that relevant indicators are measurable in a short time period and at low cost, using adequate existing information sources, and that they are subject to control by national and regional administrations. In addition, realistic targets should be established in a transparent way and initial objectives maintained during the implementation phase. These should be adjusted downwards only where this is justified by changing circumstances. The Mid-Term Review has been flexible enough in meeting changing needs and priorities 12. The current approach to Mid Term Review seems, in general, to be both appropriate and sufficiently flexible in its response to changing circumstances and the need to ensure that current interventions are made more efficient. Both of these objectives were able to be pursued within the same operational framework. 13. Strategic priorities are designed in a particular economic and institutional context and are subject to both external and internal influences. Although continuity of policy is desirable, there is always scope for further improvements, even in the case of the most successful interventions. In rapidly growing economies, (e. g. the Irish economy), economic performance enables structural changes to be made to the programming framework so as to address major problem areas such as inadequate physical infrastructure or skills shortages. 14. The Mid-Term Review carried out for Objective 1 and 6 programmes did not radically affect their overall strategic orientations. The extent to which these interventions have been adjusted varies between Member States and regions, as well as between CSFs and SPDs. These latter, although smaller in scale, involved transfer between the different measures which were proportionately more substantial. 15. In advancing its own priorities for adjustments to be made in the context of the Mid-Term Review, the Commission gave particular emphasis to sharpening the focus on employment, especially by supporting more job-intensive interventions. Although more could have been done to support this overriding priority, encouraging signs such as the results of the Mid-Term Review have emerged from the current experience. For example, there has been some reorientation of major aid schemes so as to benefit initiatives favouring SMEs and employment. In addition, Territorial Employment Pacts have been included within the mainstream programmes and specific actions introduced with regard to Information Society and equal opportunities between men and women. Notable progress in evaluation, but further improvements are needed 16. Lessons drawn from the present Mid-Term Review show that it is not a purely academic exercise, but operationally and strategically oriented. In many Member States, programme managers and policy makers took a similar view and asked, on the basis of the evaluation findings, what kind of adjustments needed to be made, particularly with regard to the current interventions. 17. Notable progress has been made regarding the quality and content of the Mid- Term Evaluations and their incorporation into the overall Mid-Term Process. In time, this will lead to an improved management culture. In identifying good practice, factors such as process management (e. g. identifying key issues, reaching consensus, etc. ) and more careful planning of the Mid-Term Review process have been noted. These will be included in any future guidance on best practice. 18. As a result of the Mid-Term Review, national and regional authorities have taken steps to improve further the effectiveness of their interventions, especially with regard to their own delivery systems for Structural Funds. In particular, significant improvements will be made to the monitoring systems, notably by raising the quality and relevance of the indicators set out in the programmes. Other arrangements are currently being sought to simplify the management of small- scale programmes and establish more coherent selection criteria in line with programming objectives. 19. With regard to the next programming period, the main issue will be to consolidate evaluation practice further, taking stock of the results achieved so far. Reinforcing evaluation as a genuine tool for decision making, strengthening monitoring and indicator systems, supporting evaluation methodologies and promoting a multilateral exchange of experience across the European Union appear, in this perspective, as challenging areas for further raising the effectiveness of structural interventions. in E X E C U T I VE S U M M A RY INTRODUCTION /. Key elements of the Mid-Term Review process 1. 1. Policy context and evolving needs 1. 2. Commission guidelines 1. 3. Mid-Term Evaluations 1. 4. Financial implementation //. Lessons from the Mid-Term Evaluations 2. 1. Implementing the Mid-Term Evaluations 2. 2. Principal findings 2. 3. Utilisation of evaluation results ///. Main outcome and adjustments 3. 1. Link between evaluation and reprogramming 3. 2. Overall reallocations of Structural Funds 3. 3. Inclusion of EU priorities IV. Policy implications and future challenges 4. 1. Short-term implications for current delivery systems 4. 2. Initial reflections for future programmes 4. 3. The challenge of evaluation 3 4 4 5 5 6 6 6 11 21 22 22. 24 26 29 29 30 30 2- INTRODUCTION The Mid-Term Review1 aims to ensure a more effective and efficient implementation of structural interventions delivered through Community Support Frameworks (CSFs) and Single Programming Documents (SPDs). This process is particularly important given public budgetary constraints. These latter make it increasingly necessary for Structural Funds resources to be fully used and for Structural Funds interventions to be of high quality in terms of their individual content. Evaluation represents an important means by which these goals might be realised. An important innovation of the current Structural Regulations is the requirement for systematic evaluation to be introduced for all EU programmes. In the context of the SEM (Sound and Efficient Management) 2000 initiative, the Commission regards evaluation as a key element for improving the management culture in which Community programmes operate2. More recently, the Commission's policy document Agenda 20003 covering the period 2000-2006 has aimed for greater cost-effectiveness from all structural interventions through reinforcing monitoring, financial control and evaluation in the context of decentralised management. The present report summarises the principal elements of the Mid-Term Reviews carried out in the Member States and analyses, in particular the role of the Mid-Term Evaluations as well as the extent to which EU priorities feature in all Objective 1 and 6 programmes. The Mid-Term Evaluations carried out by the individual Member States are the main source of information for this Report. It attempts to establish the role of the evaluation process in the development of the present Mid-Term Review and to identify best evaluation practice as a guide for the future programming period. This reflects a management culture which is emerging through the development of evaluation practice. By identifying good practice (Annex 1), this report contributes to a better understanding of how evaluations were implemented and used tp inform the Mid-Term Reviews particularly with regard to how they will be used in future in the context of the new Structural Fund regulations. It thus offers some experimental basis for actions to be taken after 1999. The following four areas are examined in the Report: 1. the overall Mid-Term Review process; 2. the principal findings of the Mid-Term Evaluations; Throughout this report a distinction is made between Mid-Term Evaluation and Mid-Term Review. Although the two terms are sometimes used interchangeably, Mid-term Review more properly describes the complete process comprising the evaluation exercise and the resulting adjustment of priorities and programme budget reallocations. Mid-term Evaluation is simply the actual evaluation exercise which informs the Mid-Term Review. SEM 2000 - Communication on Evaluation, 8 May 1996. Agenda 2000, For a stronger and wider Union, 1997. - 3- 3. the main outcomes and adjustments made; 4. the implications for programme delivery and priorities as well as future challenges I. KEY ELEMENTS OF THE MiD-TERM REVIEW PROCESS The Mid-Term Review process draws on four main elements: the policy context; the Commission guidelines; the Mid-Term Evaluations; and the state of financial implementation of Objective 1 and 6 programmes. /. /. Policy context and evolving needs The main aim of CSFs and SFDs is to reduce disparities between Member States in terms of their respective per capita income. These interventions seek to effect a long-term change in the structure and the performance of recipient economies. They are driven by specific national priorities and strategies to tackle basic structural problems, such as differences in infrastructure and human capital. These are important factors influencing regional competitiveness and hence growth performance. The four poorest Member States - the "Cohesion countries"- have seen a notable improvement in their per capita income levels. However, continuing disparities still exist among the European regions: in 1995, the 25 most prosperous regions had an average per capita income three times as high as that of the least-developed regions. The unemployment gap is even. more acute, affecting, in particular, the most socially disadvantaged groups4. Changes in the economic, social or political context in the Member States have affected their respective needs and problems, and thereby influenced the way in which they have adjusted their programmes. For example, Ireland's recent economic success has altered the need for certain types of intervention, notably interventions which aim to assist the productive sector. The Mid-Term Evaluation carried out in respect of that Member State has shown that there is a need to redirect resources towards public infrastructure (e. g. non-urban roads) to keep pace with the rapid economic growth. Similarly, the acknowledgement of employment as an overriding EU priority has also led to a number of programme adjustment such as the formal introduction of Territorial Employment Pacts. In addition, changes in national employment policies, such as structural reforms in the labour market have contributed to the refocusing of some training actions. Unexpected developments were also addressed in the context of the Structural Funds. For example, the earthquake emergency in the Umbria and Marche regions of Italy resulted in those regions benefiting from CSF resources an example of mutual support between Italian regions. See First Report on Economic and Social Cohesion (1996), COM (96) 542 final of 6 November 1996. 1. 2. Commission guidelines The changing economic and social context at Member State level has led the Commission to adopt general guidelines5 to foster the value-added of Community interventions. These aimed to provide a general policy and priority framework within which adjustments to their current 1994-1999 Objective 1 and 6 programmes could be made as part of the Mid-Term Review process. The guidelines set out a number of thematic priorities in support of the main aim of Structural Funds interventions: helping to establish the conditions for sustainable economic development, growth and competitiveness and, thereby, lasting-jobs. This overall aim was underpinned by the following specific priorities: Basic Infrastructure, Productive Environment (all types of measures to improve the growth and competitiveness of business and industry), Research and Technological Development, Environment and Sustainable Development, Human Resources Development and Equal Opportunities. 1. 3. Mid-Term Evaluations In accordance with Structural Funds regulations and the programmes' standard clauses, all Member States (with the exception of Italy6) have managed to carry out Mid-Term Evaluations to inform the review of their structural interventions. These evaluations were designed to contribute to the process of improving the execution and management of the programmes for the remainder of the period. The guidance document7 published by the Commission stated that Mid-Term Evaluations should involve: a critical analysis of all of the data collected (particularly monitoring data); some measurement of the extent to which objectives were being achieved; an explanation of any discrepancies between the actual and expected results of the intervention; an assessment of the rationale for the intervention and the continued relevance of its objectives. ' In addition, evaluations were expected to provide the data necessary to assist Monitoring Committees to form an opinion and propose corrective measures to the relevant authorities so as to enable adjustments to be made to the programmes. As a general rule, in the case of programmes lasting longer than three years, a Mid-Term Evaluation is carried out at the end of the third year of implementation. These Priorities for the adjustment of Structural Funds programmes to the end of 1999* - Commission Guidelines, May 1997. The formulation of guidelines had been endorsed by the Member States at an informal meeting of Ministers responsible for regional policy and spatial planning held in Ballyconnell, Ireland on 14-15 November 1996. The Member States formally asked the Commission to "formulate policy guidelines established in accordance with the principle of subidiarity, aimed at adapting the current Objective 1 and 6 programmes towards increased job-intensiveness for the period to the end of 1999 ". The Mid-Term Evaluation was undertaken on an interim basis by the Italian authorities (Evaluation unit of Budget Ministry) in the first half of 1997. 7 Common Guide on Monitoring and Interim Evaluations (1996). - 5- programmes are also subject to ex-post evaluation at the end of the reference period. Evaluation arrangements should recognise this timescale. 1. 4. Financial implementation Evaluations deal normally with assessing impacts and progress in relation to objectives and priorities. They rely on financial data relating to the implementation of programmes which are supplied by the relevant monitoring systems. In addition, budget implementation at EU level provides an overview of commitments and payments. Progress here is summarised in the next paragraph for programmes under Objectives 1 and 6 (see Annex 2). Since 1994, there has been continuous and regular progress in the implementation of the Objective 1 programmes. This was sustained during the first half of 1998. By the end of 1997, 64% of the funds had been committed (compared with 46% at the end of 1996). Similarly, payments had been made covering 48% of the funds (compared with 33% at the end of 1996). Progress was particularly marked in Spain and Portugal where, over the 1994-1997 period, respectively 69% and 70% of funds were committed and 53% and 56% of funds were paid. Other Member States reported significant delays in implementation in terms of commitments, (particularly Netherlands, France, Belgium), while for Italy and Austria, delays related mainly to payments made. With regard to Objective 6, the two Member States offer contrasting situations. Implementation is more advanced in Finland than in Sweden. In Finland, by the end of 1997, 59% of funds had been committed compared with 32% in Sweden. II. LESSONS FROM THE MiD-TERM EVALUATIONS Mid-Term Evaluation has been an important development in the management of Structural Funds. It offers a valuable framework for reflection on the overall efficiency of EU interventions. The main lessons to be drawn concern the implementation process and the value of the main findings in terms of policy. 2. 1. Implementing the Mid-Term Evaluations a) Mid-Term Evaluation: a shared responsibility One innovation in the current Regulations governing Structural Funds is the strengthening of requirements regarding the evaluation of efficiency and effectiveness of the interventions. These requirements are clearly set out in the Regulations. 8 The Commission and the Member States have worked together to define the operational procedures to be followed. Evaluation is thus a shared responsibility. During 1995, the Member States and the Commission were actively involved in trying to reach a consensus According to art. 26. 2 of the Co-ordination Regulation, "appraisal and evaluation shall, according to the circumstances, be carried out by contrasting the goals with the results obtained, where applicable, and by reference to macroeconomic and sectoral objectives and indicators based on national and regional statistics, to information yielded by descriptive and analytical studies and to qualitative analyses". The Standard Clauses in the CSFs and SPDs, agreed with the Member States, indicate that "monitoring [. ] is backed up by interim evaluations so that any adjustments required to the CSFs and to operations in progress may be made". on the content of such evaluations and on the operational procedures to be followed in each case. While the operational framework proposed by the Commission was discussed multilaterally, the task of implementing the evaluation has, in general, been carried out at the level of each Monitoring Committee. By and large, despite the problems which accompany any new procedures and the difficulties some Member States had in defining and initiating an evaluation process, there have been no major differences of opinion between the Member States and the Commission9. b) Setting up the Mid-Term Evaluations Beyond the regulations - Improving management Most of the forms of intervention (CSFs, SPDs, Operational Programmes, Global Grants, e t c. ) programmed in each Member State were subject to a specific evaluation. Some Member States also instigated additional evaluations to complement their Mid Term Evaluations. These were either thematic (examining, inter alia, SMEs and Research related measures) or operational (e. g. selection criteria, procedures, etc. ). The high than a hundred for Objective 1 & 6 number of Mid-Term Evaluations (more programmes10) indicates that there is now a widespread recognition of their importance by the Member States. This is shown, not just through their compliance with the regulatory provisions, but also through their perception of evaluation as an instrument for improving the management of funds and optimising the results obtained thereby. The cost of the evaluations In the SEM 2000 Communication on Evaluation (1996), it was recommended that the overall budget for evaluation activities should not exceed 0. 5% of the programme budget. Evaluation costs were generally financed under the "Technical Assistance" budget of each programme. The percentage of funds allocated to this activity has been in general below 0. 1% of total cost of the programmes. In any case, fixed percentages would have been inappropriate since the cost of an evaluation is not necessarily proportionate to the cost of the programme in question. The evaluation procedure The starting point for the Mid-Term Evaluation process is to analyse the extent to which the programme in question lends itself to evaluation (the practicability of carrying out an evaluation or "evaluability ") and to establish a methodology to apply in the subsequent phase. An assessment of how readily a given prograrnme can be evaluated is necessary in order to address some gaps or inconsistencies in the information contained in the programming documents such as an insufficient quantification of targets for certain measures. Important to this process is the recognition that, before a successful evaluation can be undertaken, there is a need to ensure that sufficient arrangements have been put in The Madrid Informal meeting of Ministers of regional policy and spatial planning on 30th November-1st December 1995 marked an increased awareness among the Member States to the importance of Mid-Term Evaluation and its operational aims. 10 See list in Annex 3. - 7- place to allow the programme ir^question can be evaluated. In particular, the evaluation of a programme generally requires that relevant indicators and targets have been set in advance and monitored over the programme's lifetime (Box 1). Box 1: Assessing the practicability of evaluation - ERDF Andalusia OP (Obj. 1) This evaluation aimed to describe the rationale for the programmed actions, objectives and goals, the degree of synergy between them and the adequacy of programme management. It focused on three main elements: quality of planning; information systems; management capacity. The approach adopted involved an analysis of the programming documents (using a Logical Framework Scheme) and interviews with key programme managers. The application of this methodology led to some practical conclusions. With regard to the quality of the programme, the evaluator observed some deficiencies in the SWOT (Strengths-Weaknesses-Opportunities and Threats) analysis as well as problems with some of the objectives (e. g. Tourism) which were poorly targeted. What is more, the information base was deemed to be too reliant on output as opposed to impact indicators. An assessment of the management system was also undertaken in order to bring about possible improvements, with particular regard to the division of tasks and responsibilities within the regional administration. Although not all of the recommendations made were taken on board, this first assessment raised an increased awareness among programme managers of the need to promote an evaluation culture and establish a better understanding of their own programme. The Mid-Term Evaluation itself is carried out halfway through a programme's implementation. It is intended to assess the degree to which the programme has met its objectives, determine the initial impact of the interventions and, where necessary, propose recommendations in order to improve the management of the programme and optimise its results. Mid-Term Evaluations should, in general, take as a reference point the outcomes of the ex-ante evaluations conducted on the basis of the plans submitted by the Member States. Independence of the evaluators The Member States agreed that it was particularly important to ensure that evaluation reports were drawn up independent of both the authorities responsible for managing the Structural Funds and the executive bodies with responsibility for co-financed actions. Mechanisms have therefore been established for appointing external evaluators (consultants, specialist academic teams, etc. ). In those Member States in which evaluation is a well-established part of public expenditure programmes, external evaluators have tended to benefit from continuous advice and support from the relevant authorities. Internal evaluations can have some benefits (e. g. promoting "learning by doing") since managing authorities are closely involved in questioning the 'how' and the 'why' of their activities. However, for mid-term and ex-post evaluations, internal evaluations may not be practical, cost effective or even desirable. A notable exception was Italy where the Mid-Term Evaluation, performed by the Evaluation Unit of the Budget Ministry, raised a number of critical issues, especially on the quality of the programming documents and the lack of appropriate targets and indicators. -8 c) Building up evaluation capacities in the Members States Establishing management structures Responsibility for the implementation of Mid Term Evaluations usually rests with the Monitoring Committees. Typically, these are involved in defining the content of evaluation reports and establishing a work programme for the evaluators. More importantly, they have analysed and discussed the completed evaluation reports, and, where applicable, proposed or advised on possible programme changes to maximise the efficiency and effectiveness of the Structural Funds allocated. In a significant number of cases, Monitoring Committees have established technical groups or sub-committees (see Box 2) in charge of ensuring ongoing liaison between the Committees themselves and the evaluators. These have then drawn up reports and proposals (based on evaluation results), for discussion and clearance by the Committee as a whole. Box 2: Managing evaluations - the creation of ad-hoc structures. In Ireland, the CSF as a whole and its three largest component Operational Programmes were each served by a dedicated evaluation unit providing on going advice and expertise. External evaluators supplied a similar on-call service to five other Operational Programmes. The Monitoring Committees have discussed the Mid-Term Evaluation reports at two separate meetings. Their conclusions and recommendations, along with the content of individual Operational Programme evaluations, have provided input to the overall CSF Mid-Term Evaluation. In addition, a regional evaluation report informed the overall CSF evaluation and account has also been taken of the Commission's guidelines. In Portugal, technical evaluation groups were set up to orient and control the evaluation process. Their main tasks included, inter alia, validating terms of reference, assisting in the selection of evaluators and discussing final reports prior to submission to the Monitoring Committees. These groups consisted of Commission officials and representatives of the relevant national and regional administrations. They normally met at least twice a year in advance of the Monitoring Committee meetings. Constructive relations between the partners within these groups have contributed to an improvement in the quality of evaluation work. Role of the partnerships In general, programme partners have shown a positive attitude towards Mid-Term Evaluations and willingly accepted responsibility for taking these forward. However, regional authorities vary in terms of their specific involvement in the tasks associated with Mid-Term Evaluation. These differences reflect characteristics of the political and institutional systems (i. e. the degree of decentralisation), the different forms of Structural Fund interventions (national, sectoral, regional and sub-regional programmes) and differing practices and experiences at the different Member State level. In some cases, regional authorities were responsible for setting up and managing the evaluation process. They were involved in selecting the evaluators, defining and facilitating their work and ensuring that the Monitoring Committees discussed and analysed the outcome of the evaluations. In addition, they ensured that evaluation recommendations were applied at programme level. In other cases, responsibilities and work were taken on by both national and regional authorities. But normally, where there were Community Support Frameworks (as in the case of some Objective 1 regions), the national authorities were responsible for the evaluation. - 9- A further situation is that of Member States where the regional authorities do not have executive responsibilities for the programming and management of Structural Funds. Their participation in the process, consisted of facilitating the provision of the data and information needed to draw up evaluation reports, co-operating in the analysis of the impact that policy measures had made at local level, and, to some extent, participating in the the recommendations of the evaluation reports. Social partners were also associated with this process to the extent of their participation in Monitoring Committees. In general, partnerships have been developed in a pragmatic and flexible manner. redefinition of strategic and operational guidelines stemming from Creating professional skills in evaluation Over the past four years, the development of evaluation activities across all Member States has been notable. Structural Funds regulatory requirements have been a main factor in this development. The provision of evaluation services remains largely outside of the various administrations, although Ireland and Italy are notable exceptions, having created their own specialised teams. The development of evaluation as a specialised service has led to growing professionalism and expertise. Although the evaluation teams appointed for different programmes tend to have multi-disciplinary expertise and include both consultancy companies and academic research centres, their main strengths are usually in the fields of economics and management. d) Improving the quality of evaluations The average quality of Mid-Term Evaluations carried out during the 1994-99 programming period has undoubtedly improved compared with the previous period (1989-93) although factors such as delays in the launch of certain programmes or the adoption of inappropriate methodologies meant that some exercises were below standard. Evaluation reports were subject to assessments according to the eight criteria set out under the MEANS Programme11. These assessments suggest that most reports were of reasonably good quality (Box 3). The evaluations frequently provided updated information and analysis of the programmes in relation to their objectives and targets and some evaluations (e. g. Northern Ireland) have applied their own quality assessment in terms of strengths and weaknesses. Experienced evaluators and more extensive knowledge and research on the practicalities of evaluation have also contributed to enhancing the quality of the reports. 11 Quality Assessment of Evaluation Reports: A Framework. MEANS (Méthodes d'Evaluation des Actions de Nature Structurelle) is a European Commission programme, which aims to improve the methodological tools available for assessing structural actions. They were given by assessing the evaluation reports against the following 8 criteria: meeting needs: does the evaluations adequately address the information needs of the MC and fit the Terms of Reference?; relevant scope: is the programme's rationale, outputs, and impacts fully covered, including unexpected outcomes?; defensible design: is the evaluation design appropriate to answer the questions asked?; data: is the data used/collected appropriate and is its reliability duly taken account of?; sound analysis: is the information available subjected to appropriate analysis?; credible findings: do the findings follow logically from, and are justified, by the analysis?; impartial conclusions: are conclusions fair, unbiased by stakeholder views, and operational?; and clarity: is the report written in a way that is easy to understand? - 1 0- Box 3: Assessing the quality of an evaluation: the Irish CSF evaluation The evaluation report fulfils the following criteria information needs were adequately addressed; the rationale of the programme and its objectives were comprehensively examined; a specific methodological framework was developed to assess the effectiveness of measures; an adequate amount of reliable data were collected (although gaps and deficiencies still exist); a sound analysis of CSF's performance was provided; credible findings were produced and had practical effects on the progrgmme's structure objective conclusions were proposed and were sufficiently detailed for operational use; presented to stakeholders in an accessible and clearly written formats. The overall quality rating is therefore very high, due to a well designed evaluation framework. The report was published and made available to the public. The assessments made of rural development programmes, based on the interim evaluation, vary somewhat. In some cases this assessment is considered acceptable (Spain), and in others satisfactory (various German Lander) since it has achieved its aim of drawing attention to good results. Elsewhere, it is regarded as being of a very high level in view of the experience and quality of the independent assessors and a more precise definition of aims and specifications (Ireland). Sometimes, the scope of the evaluations has been more limited because of their academic or excessively general nature or because they looked only at implementation and management rather than at results and impact (France, Belgium). Some evaluations were also criticised as being insufficiently independent of the administration which commissioned them. In other cases, constraints were imposed on the assessor's work by lack of cooperation from the administrations concerned. The methodologies applied by evaluators have varied in nature and quality from one programme to another. Although the Commission has promoted a number of methods and tools through the MEANS programme, their implementation is still at an early stage. For example, impacts on employment are often seen as being difficult to measure, especially when deadweight and displacement effects have been taken into account12. Despite a number of qualitative improvements, there remains a need to reinforce, through appropriate guidance, the use of sound evaluation methods over the next evaluation stages. 2. 2. Principal findings Amongst their main goals, the Mid-Term Evaluations attempted to assess progress on the attainment of the various programme objectives, as well as their likely impacts on, for example, job creation. They also sought to examine the process by which the programmes had been managed and their component projects selected and monitored. See European Commission document, "Counting the jobs", 1997 and the European Job Challenge (1998). - 1 1- The purpose of this section is to review, on the basis of a representative cross section of evaluations, significant results relating to mainstream issues of concern and to suggest areas where improvement is needed. It will highlight some key features of the analyses carried out concerning three main themes: - macroeconomic impacts for the largest programmes, notably with regard to growth and employment; - microeconomic assessment, focusing on effectiveness issues; - the quality of delivery systems, particularly monitoring and indicators systems and project selection criteria. From this analysis, some general conclusions can be drawn in terms of how these results might be used and integrated into the Mid Term Review process. a) Macroeconomic impacts: growth and employment effects Most evaluations of Objective 1 CSFs generally include a macroeconomic assessment providing a number of valuable insights into the overall Mid-Term Review process. These illustrate the importance of taking into account supply-side, as well as demand side, impacts and provide a first quantification of the likely scale of these impacts. For large-scale interventions such as major Objective 1 CSFs, macro-economic effects in terms of economic growth and employment are likely to be visible. However, these effects are difficult to quantify due to the need to identify the counterfactual situation, i. e. the situation without Structural Funds. One option is the use of simulation techniques based on models which take into account the macro-economic interdependence of variables. Although there are several models quantifying the short-term demand side effects (e. g. input-output models), only a few take into account the long-term effects of supply side conditions the improvement of which is the main objective of Structural Funds13. In the context of the Mid-Term Evaluations, the supply-side effects of adding infrastructure, human capital and productive investment to an economy are estimated in different ways of varying sophistication. For Ireland and Spain, HERMIN14-type models have been applied, while for Greece, Portugal and Italy more general econometric models have been used. Many variables are endogenous but "external" influences include interest rates, exchange rates and economic growth among trade partners. Due to the non availability of a macro-model for East Germany, the German evaluation took an ad hoc approach, applying, step by step, estimates for the various parameters influencing GDP 13 14 The most recent attempts to quantify the combined demand and supply side effects of Structural Funds have been made using the QUEST II model, the results of which were presented in the First Cohesion report (1996), the HERMIN model used for the Single Market Review (1996). Hermin is a macroeconometric model specifically designed to quantifiy the medium to long-term impact of the CSFs. It provides a common framework focusing on the most important structural features of the 'Cohesion countries'and explains the mechanisms through which the CSFs will affect the supply-side of their economies. - 12- and employment; the authors themselves note in the report that the results carrying a high degree of uncertainty. Two studies did not provide an integrated supply side in their model and simply made additional calculations on supply side effects based on average values available in the literature. The Portuguese evaluation calculates an additional productivity effect of between 1. 17% and 2. 35%. The study on East Germany assumes long-term supply side effects on GDP to be 20% of the short-term demand side effect. For this reason, the results presented below for these two countries (and, at this stage, for Greece) feature demand side effects only. A specific problem of macro-level modelling is given for those countries whose territory is not fully eligible for Objective 1 assistance (D, E, I). Each of the evaluations has dealt with this problem in a different way: the German evaluation analysed the impact on East Germany alone whereas the Spanish evaluation only considered the impact on Spain as* a whole. The dualistic structure of the Italian macro-model enabled at the impact on both Southern Italy and Italy as a whole to be assessed. The results of the different Mid-Term Evaluations (see Graph 1) should be treated with caution regarding their comparability since the methods used varied across countries. Nevertheless, there can be no doubt on the positive growth and employment effects of Objective 1 programmes. Mainly due to the impact of increased productivity, effects on employment are usually estimated to be much lower than those on growth. Graph 1 shows, for example, in Spain, that the CSF is likely to increase GDP by 5. 1% which means an additional annual growth of roughly 0,8% average; and employment by 2. 4% by 1999 relative to the baseline position (i. e. the situation without CSF). The challenge for future evaluations of the macroeconomic impact of Structural Funds will be to make more use of methodologies which represent the state of the art. Macro- models should have integrated demand and supply sides, the latter allowing to distinguish the main categories of interventions such as infrastructure, human capital and productive investment. Main variables to give results for are GDP, investment, employment, consumer prices, budget deficit, imports and exports. The most interesting additionality scenarios are EU funding alone as well as EU funding and national funding together, both excluding private co-financing. Account should also be taken of opportunity costs of public spending, i. e. the effects of an alternative use of EU and national funds. Finally, the sensitivity of results to changes in CSF spending and in economic policy can provide more concrete results in terms of policy conclusions. - 1 3- Graph 1: Mid-Term Evaluation results -percentage deviations from baseline in 1999 % • GDP • employment Germany Greece (Obj. 1) Spain Ireland Italy (Obj. 1) Portugal Note: The employment figure for Portugal is the reduction of the unemployment rate in percentage points. Sources: Ifo Institut (D), KEPE (EL), Quasar (E), ESRI (IRL), Ministero del Tesoro-Nucleo di Valutazione (I), CISEP (P). Assessing the macroeconomic impact in small areas For small areas, specific models were designed to estimate the long term impact on growth, investment and employment. An interesting methodology was developed for the Belgian Hainaut region (Objective 1), notwithstanding the difficulties in collating regional data at a very disaggregated level (Box 4). Box 4: Hainaut (Obj. l) : the HELM model HELM (Hainaut Lead-in Model) is an econometric model assuming a key relationship between output growth and productivity growth. This model is extended to integrate different components of productivity growth (research and development, physical capital, skills). The results are presented for three situations: the baseline situation (A), the SPD without constraint (B) (measuring the additional impact of the SPD), the SPD with constraints (C) (measuring the additional impact but integrating, for the aid schemes, the rate of assistance and selection criteria adopted). The comparison between these situations allows for an estimate of the additional impact of the SPD against the baseline. The results of the simulations run for the SPD Mid-Term Evaluation relate to three main variables (investment, value-added, employment) in the manufacturing and business services sector. According to this model, the SPD will generate an additional investment growth of 0,7% per year for the 1994-2005 period and additional employment growth of 0,3% per year, amounting to 5100 net jobs in 1999 and 15800 net jobs in 2005. The massive absorption of funds at the end of the period will reduce the efficiency of implementing the SPD. Delays in execution may also have the effect that impacts are more visible at the end of the period and thereafter. The model shows that any increase in the rate of assistance will have only a moderate effect on investments, but these will produce some beneficial (indirect) effects within the area. Therefore, the risk of returning to slower growth compared with the EU average after the completion of the SPD is significant. 14 b) Microeconomic assessment Assessing effectiveness The evaluators were asked to provide an analysis of the effectiveness of the interventions. Most evaluations took a view on the extent to which the programmes were meeting their stated objectives15. As part of their work, they had to review the existing indicators, and in some cases proposed a number of new or revived core and key indicators, which had to be agreed with the regional authorities. This assessment has often required a substantial review of the indicator systems, in terms of both financial results and the monitoring of the agreed physical and impact indicators. A systematic approach to effectiveness was adopted for the Irish CSF evaluation, using a microeconomic model to assess whether the measures were designed properly and were able to meet their objectives. This approach implies that Structural Funds cannot only be judged on spending but rather on the social benefits they will bring to the economy. The methodology adopted by the evaluator aims to compare the effectiveness of the measures in order to ensure an optimal allocation of funds (i. e. comparable value of money across the programmes) and identify best practice. A further more detailed description is presented in Box 11. At a more operational level, evaluations should seek to measure the extent to which main programme objectives are being met. In the case of Merseyside, the evaluation contains some key indicators (SMEs, land improved, floor space), including output data and corresponding results of job creation and private sector leverage. Quantified targets are available for 1999 and the Mid-Term Evaluation presented figures measuring progress to date. These indicators provided in an aggregate fashion at the level of priorities. Subsequent to the Mid-Term Evaluation, a process of revision of the Merseyside baseline, target and output data was initiated. Box 5: Assessing effectiveness through programme indicators-the case of Mersey side-Objective 1 (1994- 1999) -actual outputs/results for ERDF interventions ^ ^^ Driver ERDF expenditure Total Gross Jobs (direct & indirect) Total Floorspace Total land mproved Total Private Investment 1 2 3 4 5 All 15 £m %of SPD 000s 28. 8 44. 4 15. 4 4. 8 22. 1 115. 5 21 54 42 16 26 31 5. 4 9. 2 0. 8 0. 7 3. 0 19. 2 % of SPD Target 15 142 16 64 454 39 000m3 (Hec) %of SPD Target % of SPD Target £m %of SPD Target 114 116 1 9 240 20 70 0 15 n/a 24 255 17 32 210 513 52 12 n/a 300 73 67 76 0 8 23 176 40 90 28 n/a 77 Efficiency issues were rarely addressed, being one of the most difficult aspects of evaluation. Analysing efficiency involves comparing programme inputs (financial resources) with outputs (the goods and services it provides) and results (the initial effects) to estimate if the same benefits could have been produced using fewer inputs or, alternatively, if the same inputs could have produced greater benefits. Discussion of efficiency necessarily entails comparisons with various counterfactual positions. The main difficulty in this area is therefore the choice of appropriate benchmarks. - 1 5- Note: Drivers (or priorities) for the Merseyside SPD are the following; 1) inward investment, 2) indigenous enterprise, 3) knowledge based measures, 4) cultural, media and leisure, 5) actions for Merseyside. Source: Mid- Term Evaluation report (Nov. 1996). Indicators defined at OP or SPD level provide a basis on which effectiveness can be monitored and assessecl. However, they do not in themselves allow definitive conclusions to be drawn, particularly on impact, because of the difficulty in establishing causation in some areas (i. e. : the indicators may be affected by external factors). They often need to be supplemented by qualitative analysis and by appropriate evaluation work. Overview of main outputs and results Microeconomic data on outputs and results arising from the evaluation reports are not easy to present in an aggregated way because of their specificity and lack of comparability across programmes. It is also recognised that the extent to which these data can be quantified at the Mid-Term stage will vary across both the areas of interventions (infrastructure, SMEs, human resources, etc. ) and regions concerned. The standard and coverage in the evaluation reports is inevitably somewhat variable as is the extent to which key elements such as outputs, results and changes in marked disparities could be quantified. In Box 6, some achievements relating to Objective 1 and 6 programmes are presented. Basic infrastructure - reducing disparities Basic infrastructure is the area where the impact of Community action is most visible and measurable. The incidence of these major investments is sometimes high in structural terms. For example, in Spain, CSF resources supporting transport, communication and energy networks represent on average more than 25% of national infrastructure. Notable progress in reducing disparities with the rest of the Union has been made in most sectors (see box 6). Box 6: Important achievements in basic infrastructure. • • • Substantial effort is continuing to support the development of the strategic road networks in the Cohesion countries through, for example, the improvement of four major corridors in Ireland (reaching 49% in 1996 against a target level set for 1999 of 53%) and the completion of the Athens-Thessaloniki motorway (400 km built to date). In Portugal, the construction of motorways and primary roads to date represents 74% of the 1999 target. In Spain, the stock of physical infrastructure for motorways increased by 13% between 1993 and 1995 (latest data available), benefiting in particular to the less developed regions like Andalusia. In all Objective 1 regions, major investment has gone into the telecommunication sector, providing more modern systems such as digital exchanges and fibre optic links. This has contributed to a significant reduction of disparities in provision compared with the rest of the Union. In Portugal, the average density of telephone lines (per 100 inhabitants) is aligning line with the EU level; this objective was expected to be attained by 1999. Similarly, in Italy, the rate of digitalisation in the Southern regions has reached the same level (85%) as that of the Centre North. In Spain, digitalised networks increased by 36% in absolute terms between 1993 and 1995. Progress in energy diversification, notably the reduction of dependence on oil has also been made. Deliveries from the new natural gas distribution should begin in Greece. The high pressure natural gas pipeline (513 km) has already been completed and another major project (1000 km of low pressure natural gas network) is nearing completion (84%). In Portugal, more than half of the gas network (600 km) had been laid by 1996 and this will account, by 1999, for 7. 5% of total energy consumption. Greece, Spain and Portugal have all seen significant reductions in their energy use relative to GDP in _^ recent years. - 1 6- As regards the environmental sector, support is being provided to improve systems of water supply and increase the capacity of wastewater treatment facilities. In Greece, water supply, sewerage and wastewater treatment systems, serving about 5 million inhabitants (half of the population) have already been completed. Investment has also gone into smaller Objective 1 regions, like the Northern Ireland Sub-programme where the construction of water supply and waste water facilities have generated some 2270 jobs in the area. Significant environmental benefits have also arisen from a whole range of specific interventions, including the clearance of major contaminated industrial sites, environmental audits ofSMEs, assistance in the development ofeco products and reduction of emissions due to the diversification of energy supply. Productive environment - supporting employment In terms of the productive sector, structural assistance has been directed at improving conditions for existing firms, thereby encouraging new activities. It is largely recognised that support, especially for SMEs is a major driver for job creation, and this contributes to bridging structural gaps in productivity and income levels. Some examples may highlight recent trends in Structural Funds impacts in this area (see box 7). Box 7: Some achievements in the productive environment • Major aid schemes have emphasised job creation, partly offsetting the negative consequences of industrial restructuring. In Hainaut, almost all funds available to aid schemes, notably SME schemes had been used before the end of the period; it has been estimated that job creation targets for those measures have already been exceeded to date (with more than 2200 jobs created compared with 1800 jobs expected). In Germany (Obj. I), Structural Funds assistance contributed actively to support employment in, for example, Sachsen-Anhalt where 18. 500 jobs were assisted in the chemical industry and a further 20. 000jobs in SMEs. In Italy, the industry Programme is likely to have created or safeguarded more than 75. 000 jobs over the 1994-96 period. • • In the tourism sector, increased benefits will derive from new capacity being made available in, for example, Sachsen (additional capacity of 11500 new hotel beds) and in Ireland, where 18 large projects which have already been completed providing additional facilities to increase the quality and standard of tourist attractions. New employment opportunities will be generated by these Structural Funds assisted improvements Physical regeneration of industrial sites will also contribute to the improvement of the economic environment and attract new business. This was a main priority in some German Lander: in East Berlin, for example, where 51000 m2 were refurbished, creating potentially 600 jobs or in Sachsen where 620 ha of land were transformed into business parks. In Nord-Pas de Calais, approximately 40- 45% of industrial land located in the Objective 1 area will have been improved. Human resources: quantifying impacts Specific evaluations conducted in the area of human resources included, as part of their workplan, an impact analysis of training measures on beneficiaries. The main results of these studies are presented below in box 8. Box 8: Impact of training measures on beneficiaries In Spain the evaluator made a comparative analysis of the placement of trainees by target group, using the INEM (The National Employment Institute) database as a quasi control group. For this analysis only the occupational training measures were taken into account. The main conclusions are in line with the results of the 1989-93 ex-post evaluation: in terms of gross impact, the placement rate of trainees is on average 20% higher than the control group; placement is higher for young people, men and those holding a secondary education diploma. However, if net impact is estimated, taking into account the characteristics of the trainees, the differences vary substantially. For example, in terms of age, trainees over 45 years old perform 32% better in finding a job than those in the control group. Training also seems to increase the - 1 7- chances of women finding jobs: the net impact is 27. 1%, whilst for men it is 14. 9%. If qualification levels are taken into account, the effect of training on those holding a primary education diploma is striking: going through a training action increases their chances of finding a job to 48. 8%. Germany: impact of the QfQsj (Gerneinsçhafi Initiative - Qst) In Germany, it has been estimated that the Bund programme in support ofGIOst (i. e. apprentice schools for the young) has brought about a reduction in youth unemployment of three percentage points in 1995. At Lander level, very few placement indicators have so far been collected. The combination of training measures with grants generally leads to higher performance, but this is the exception rather than the rule. Some regional results can be mentioned. For Berlin-Ost, the evaluator concluded that, for continuous training and unemployed training taken together, 60% of trainees had found or kept jobs after six months. Training programmes in Meckelenburg-Vorpommern achieved placement rates of 40-45% (21-32% in the first labour market). A programme for business starters, which basically tries to help people ensure that they start a viable business, resulted in about 50% of actual "successful" starters. Various continued training and education programmes for employees (Objective 4 type measures) in Mecklenburg- Vorpommern, Thuringen and Sachsen-Anhalt resulted in increases of 35 to 77% in the perception of job- security by participants. Portugal: impact on the education system Impacts in terms of strengthening the Portuguese education system (through PRODEP) have been found to be linked to the diversification of education pathways and to a focus on both quantity and quality issues. Training teachers and other agents appears to be one of the major contributions of the programme, both at 1st and 2nd level (representing 49. 3% of teaching staff) and at university level (provision of scholarships for Masters and PhD programmes). As regards the expansion and diversification of education / training pathways, major changes have included enlarging secondary education, bringing it closer to the workplace and increasing guidance and counselling in schools. In fact, support to middle level technician training has set up a whole sub-system providing technical skills directly relevant to the labour market and an alternative to mainstream education. Agriculture and rural development Since agriculture and rural development constitute separate programmes at national level, they have been assessed separately. In the case of SPDs and the agricultural sections of regional programmes, evaluation formed part of a more general evaluation. Some Member States such as Ireland carried out detailed evaluations of the main priorities of the programmes (the food industry, establishment of young farmers, compensatory allowances, equal opportunities, agricultural training, etc. ). In such cases evaluation proved a useful tool for obtaining better knowledge of the situation, analysing problems and finding the best ways of solving them. Box 9: Evaluation of agricultural interventions - the case of PAMAF (modernisation of agricultural structures in Portugal The evaluation report found that financial and physical implementation had been slow in getting started, mainly because of delays in drafting the relevant legislation and management changes. However, the initial delay has been made up and the assessor is optimistic about the chances of some adjustments in management and the allocation of funding between measures permitting programme implementation to get back on schedule so that the original goals can be achieved. The operational assistance measures concern primarily support for investment on agricultural holdings where the rate of implementation is high, infrastructure such as irrigation, rural roads, drainage and soil conservation, forestry, the processing and marketing of agricultural and forestry products, compensatory allowances in less-favoured areas, training and research. Measures of these types are intended to make agricultural holdings more competitive. Most of them represent a continuation of the earlier programme, hich had proved successful. They also form part of structural policy at Union level. The assessor stressed in particular that measures were to some extent scattered, which affected their real impcct as well as management and implementation. He suggested that the funding available should be rea!" seated to the measures regarded as priorities, particularly aid to agricultural holdings and irrigation, - 1 8- and insisted on the need to improve vocational training. He also noted the effort made to concentrate support on the private sector and a non-uniform regional distribution of projects. The report notes some of the results of the agriculture programme in 1994-96, including the financing of some 20 000 projects, supportfor 11 500 agricultural holdings and the establishment of 2 800 young farmers. Fisheries The FTFG provides support in five main areas of assistance: adjustment of the fishing effort (26% of the budget), renovation and modernisation of the fleet (24%), the processing and marketing of products (23%), the development of aquaculture (11%) and facilities in fishing ports (7%). It currently includes 31 programmes, of which 17 are under Objective 1, 12 under Objective 5(a) and two under Objective 6. 65% of the budget goes to Objective 1 regions. The Mid-Term Evaluations of these programmes provided interesting information about the effectiveness of assistance and reprogramming (the reallocation of funds between areas of assistance or between measures). They revealed some shortcomings in monitoring, particularly in the provision of socio-economic data. In some cases proposals to improve the indicators were implemented and initial analyses of the impact on employment carried out. Box 10: An example: the Spanish Objective 1 prqgramme The operational programme for fisheries, which has funding of about ECU 1 billion, is one of the priorities in the CSF. The measures in the programme are mainly concerned with adjusting the fishing effort, modernising the fleet, the processing of products, aquaculture and facilities in fishing ports. The evaluation report found that the average implementation rate was satisfactory at 57% but varied widely depending on the area of intervention: it was low for modernisation of the fleet and aquaculture (10%- 20%) and high for port facilities and processing of products (60%-100%). Effectiveness was lowest in aquaculture (2%): this measure has been affected by the cumbersome administrative procedures required to implement projects, poor returns which discourage potential investors and environmental constraints on project selection. Effectiveness was highest in the case of processing (65%), where beneficiaries often achieve high growth rates and so good investment capacity. The assessor proposes to reallocate funds from adjustment of the fishing effort and aquaculture to the modernisation of vessels and the processing of products. A survey of beneficiaries showed that 35% of the projects receiving grants under the programme had created or preserved jobs (initial figures suggest a total of 780 jobs), principally in the area of product processing. Proposals have also been made to improve the indictors of achievement and results, to reduce the number of specific indicators for aquaculture, establish impact indicators for the fleet and employment indicators for all areas of intervention. The assessor noted that aquaculture paid strict attention to environmental protection and recommended that greater account should be taken of this principle in other areas (processing of products and port facilities). c) Quality of delivery mechanisms Mid-Term Evaluations were also required to assess the way in which the Funds are managed, particularly in terms of monitoring systems and project selection criteria. Monitoring and indicators systems Compared with the previous programming period, the current programmes show significant developments in the use of quantitative indicators as a means of monitoring their implementation, assessing their impact and determining progress towards meeting overall objectives. -19 The Commission prepared a Common Guide for Monitoring and Evaluation with a view to ensuring consistency of approach and uniform standards16. This has been a primary impetus in bringing about these improvements. However, experience across programmes is mixed. In a number of programmes, the information available by way of indicators is often inadequate in terms of measuring performance or verifying if satisfactory progress is being achieved. Other principal shortcomings relate to the determination of appropriate and practical targets in relation to the measures in question as well as the lack of a consistent, regular system of data collection relating to physical outputs, results or impacts. Notwithstanding many improvements, the definition and quantification of appropriate indicators remains an unsatisfactory area of programme management. Whilst further work is required on indicator systems to improve monitoring and assist evaluation work, this should begin by identifying examples of good practice in this area. Interesting attempts at building appropriate management information systems can be found in a number of programmes. Some examples are provided in Box 11. Box 11: Enhancing the monitoring systems A well managed information system is a necessary (although not sufficient) condition for supporting the evaluation process. The case of Nord-Pas-de Calais (Objective 1) is an example of an integrated monitoring system, which couples financial and physical indicators at project level. In Northern Ireland, an important part of the Mid-Term Evaluation was an intensive scrutiny ofNIQUID (the database of all the indicators agreed for the SPD) to transform it into an effective management information system for the SPD. The evaluator proposed monitoring on a regular basis the progress of 52 indicators, rather than the 330 indicators identified in the SPD. In addition, for each sub-programme, a set of key indicators was developed to capture its specific impact. Significant work has also been done in Italy, where a standardised system of indicators was established for the ESF interventions. In Portugal, indicators have improved in almost all areas of intervention, even if there is still scope for further improvements in the quality of data. Project selection criteria The first two or three years of programme implementation have, in some cases, seen the development of relatively sophisticated scoring systems for co-financed projects. The systems put in place (e. g. in Merseyside) take account of a wide range of factors simultaneously and in addition to assessing eligibility they look at selection criteria both at priority and measure level. For example, priority criteria which are common to all projects include project outputs, value for money, private sector leverage and more recently environmental impact. Many Mid-Term Evaluations have questioned the effectiveness of such systems. Having transparent and objective criteria, which in itself is considered as good practice, may not be a sufficient condition to ensure that the best projects are selected. Further checks will 16 The Commission Guide distinguishes between indicators of output, results and impact. Output indicators refer to financial and physical implementation (e. g. number of kilometres of road built and cost, number of training courses provided. Results indicators refer to the immediate effects of an intervention (for example, time savings in a road project; the number of people who successfully complete a training course). Impact indicators refer to the outcomes of the interventions. A distinction can be made between the specific or immediate impact of an intervention (e. g. number of people placed into jobs) and the general or final impact, i. e. the socio-economic effects (e. g. increase in employment or GDP). - 2 0- have to be made, for example on the degree of credibility attached to the information and output forecast provided by the applicants. Box 12: Merseyside - scoring systems for ESF7ERDFprojects In the Merseyside SPD, the system put in place for selecting projects has three different aspects: eligibility or "core criteria", priority selection criteria common to all projects and measure selection criteria (which may be different from the priority criteria). The aim,was to develop sets of criteria which were understandable, transparent and which could be applied to a large number of heterogeneous projects. Main priority criteria include the expected outputs of the projects; value for money, the nature of beneficiaries (for ESF); labour market issues; evidence of partnership and linkage; private sector leverage; and environmental impact. 2. 3. Utilisation of evaluation results As described in the previous section, Mid-Term Evaluations represent an important source of knowledge for programme managers and decision makers. They inform them on key issues such as the relevance of the aims and objectives of their interventions and the effectiveness of the wider economic effects on the area concerned. An authoritative evaluation should be supported by rigourous analysis and be sufficiently operational to be fully utilised for policy purposes. However, the quality of the evaluation results does not guarantee that they will necessarily be utilised. In other words, the evaluations cannot be considered as the only input for carrying out the Mid-Term Review. Institutions and policy actors are continuously involved in exchanges of information. Other factors relating to the political context of the macroeconomic framework may also influence directly the final reprogramming decisions. Bearing in mind this wider framework, three different levels of utilisation should be distinguished in the Mid-Term Evaluation process. The first level of utilisation involves evaluation as a feedback mechanism relating to the effectiveness of policy measures. Evaluation results and recommendations have, in a significant number of cases, been largely followed and incorporated in the changes made to the programmes. The best example is the Irish CSF Mid-Term Evaluation which produced credible and useful conclusions leading to the reallocation of funds between measures or sub-programmes (see box 12). Secondly, evaluation played a major role in providing new sources of information or on going advice to programme managers and policy makers. For example, the evaluation of the Hainaut Objective 1 programme, which included a specific macroeconomic model (see Box 3), was also recognised as useful for informing the reprogramming actions. Conclusions here opened longer-term perspectives on the convergence trend of the regional economy. Other evaluations, having a narrower focus, stressed the importance of certain critical aspects of programme management, rather than being exhaustive with regard to the whole implementation cycle. Finally, the experience also showed that substantial input from the programme's partnerships is key to the success and effectiveness of the evaluation process. One important result is that decision makers were involved in the discussion of evaluation issues and a number of lessons learned in terms of improving their interventions. Although this learning process was not always formally structured, it had a significant impact on the final outcome of the Mid-Term Reviews. - 2 1- III. MAIN OUTCOMES AND ADJUSTMENTS The evaluations informed the Mid-Term Review process by providing an overall assessment of what had been achieved during the first half of the programming period. Taking stock of these elements, this part of the report assesses the main outcomes of this process priority in terms of the adjustments made to programmes. Account is also taken of the extent to which priorities laid down by the Commission have been incorporated in the programmes. 3. 1. Link between evaluation and reprogramming The evaluations were delivered within the agreed time period enabling the results to influence the Mid-Term Reviews carried out between 1997 and 199817. A special reference should be made to the three new Member States. Although their programmes started a year later than for those of the other Member States, all three made formal arrangements to launch Mid-Term Evaluations. The Commission promoted an exchange of experience between evaluators which brought new ideas into the conduct of the evaluations. A key feature of this process was that no significant adjustments were made prior to the evaluations and the formal review of programmes. Evaluations have, in general, proved useful in addressing major issues or highlighting the kinds of decision to be made. Their ability to do so has depended on the intrinsic quality of the analysis as well as on the operational focus of their recommendations. In the case of rural development, interim evaluations produced specific recommendations and proposals as requested both by the Commission and the managing authorities. These included the reallocation of resources. In Spain and Germany they coincided with the proposals by the responsible authorities since the grounds (progress of measures as shown by the monitoring data) were the same. In other cases, only some were taken into consideration (Portugal). Sometimes, evaluations were carried out after programmes had been amended and it was subsequently found that there had been no contradictions (France). In Italy the assessor concentrated on analyses of problems and obstacles to implementation and those responsible for the programmes proposed the reallocation of resources. In the case of Ireland, it has already been noted that the assessor's recommendations concerning the strengthening of certain measures could not be implemented because of lack of finance. The example of Ireland and Merseyside Two examples serve to highlight the entire Mid-Term Review process, from the formal evaluation exercise to the implementation of the financial reallocation decisions. These relate to two different levels of intervention: a large CSF (Ireland) and an SPD (Merseyside) (see Box 13 and 14). * ' There has been some delay in the negotiations for the Mid Term Review in France-Obj. 1 (especially la Réunion) and for Greece (reprogramming of ERDF interventions; ESF modifications scheduled for the second half of '98). In Italy, a number of reprogramming actions were decided in April '98 and the Mid Term Review will be finalised in October, on the basis of actual progress of some slow spending programmes. - 2 2- Box 13: Mid-Term Review (I): the case of the Irish CSF The report states that the Irish CSF represents a notable success and that the funds have been deployed effectively to support and enhance what has been a remarkable economic recovery. Under the CSF, medium term planning of public expenditure has allowed more systematic and effective programming in many areas. The CSF evaluation raised questions about the value of devoting public resources to a number of specific measures, notably free or subsidised provision of services for which firms or individuals would be willing to pay. A core assumption here was that markets are the most effective and efficient means of achieving economic and social objectives so that public intervention is justifiable only where the market is not working properly and the intervention in question creates no economic distortions of its own. In the light of these theoretical assumptions, four situations were identified, where in the opinion of the evaluators, public intervention in a market economy could be justified: the provision of public goods and services (such as transport infrastructure) which would not be provided in the absence of public intervention; the introduction of corrective subsidies designed to alter the price of goods and services where the market price does not adequately reflect their wider social benefit (e. g. the cost of loans to SMEs) the management of targeted schemes aimed at changing behaviour through correcting a lack of knowledge, awareness or understanding (e. g. introducing business owners or employees to new technology) the redistribution of wealth through subsidies or welfare benefits in pursuit of broadly social aims The CSF evaluation categorised the 166 CSF measures using the above headings. For each category, one or two "anchor measures "- measures which were large scale or which addressed long-established and well-understood elements of public policy - were selected and used as comparators for the other measures in that same category. On the basis of these comparisons, decisions were taken as to which measures were successful and which were not. Some 150 recommendations were in turn produced providing how CSF resources might be deployed more effectively, with particular focus on those measures poorly targeted or having undesirable effects. The evaluation pointed to a clear case for more public spending on physical infrastructure, especially non- urban roads. In some demand-led measures it was suggested that spending targets should be revised in line with actual evolution, for example a reduction of the grant to the Food Industry Sub-programme. It was also recognised, in discussions within the Monitoring Committees, that more emphasis should be placed on youth and long-term unemployment and that allocations to specific measures for education and the disadvantaged should be increased. The CSF evaluation also contains a number of detailed recommendations in relation to the management and implementation of specific measures. It is argued, for example, that there is insufficient competition in the provision of subsidised services due to the predominant position of state agencies and the lack of competition. Regarding monitoring, the report suggests further improvements to performance indicators, while admitting that the current CSF represents the most advanced experiment in monitoring and indicators systems. The outcome of the Mid-Term Review largely followed these recommendations. Accordingly. the shifts were mainly away from the productive sector towards human resources development and economic infrastructure. Financial reallocations amounted to 163 Mecu, focusing on RTD (37 Mecu), Transport Infrastructure (36 Mecu) and a range of measures dealing with early school leavers and other marginalised groups (46 Mecu). Box 14: Mid-Term Review (II) : the case of Merseyside (Obj. l) Structural Funds assistance to Merseyside comprises 816 MECU delivered through the Merseyside Single Programme's five "Drivers for Change": Inward investment and the corporate sector; Indigenous enterprise and local business; Knowledge-based industry and advanced technology; Culture, media and leisure industries; Action for the people of Merseyside. The latter receives the largest proportion of Programme funds (more than two fifths). The inward investment and local business drivers received, respectively, a quarter and a fifth of Programme funds. 23 The advent of the Single Programme coincided with a cyclical upturn in the UK economy characterised by falling unemployment and economic growth, both at rates higher than the EU average. Merseyside has shared in this upturn with the result that Structural Funds receipts are being deployed in a situation more favourable than that envisaged when the Single Programme was being developed. Although improvements in terms of unemployment have been small relative to the UK as a whole, they should be seen in the context of a recent picture of decline in the principal social and economic indicators. The Mid Term Evaluation assessed the current and potential impact of the Single Programme on the people and business life of Merseyside, commented critically on existing monitoring arrangements and made recommendations as to how the Programme should proceed over the remainder of the programming period. Desk research was supplemented with a review of 100 separate projects along with extensive survey and interview research focusing on Programme beneficiaries. The volume and quality of the evaluation research was significant. It compensated significantly for resource constraints which limited the amount of in-depth analysis possible. The evaluator made a series of detailed and reasoned recommendations, proposing in particular that future monitoring and evaluation work should focus on the larger projects likely to deliver the greatest impact as well as on issues such as displacement and sustainability. It was also recommended that the existing set of indicators be expanded to enhance the assessment of Programme impact. Proposed indicators included, inter alia, survival rates for SMEs and their use of new technology. In addition, it was suggested that insufficient focus was being given to synergies within the five Programme Drivers and to themes which cut across several measures. Reprogramming changes at the Mid-Term Review stage broadly followed the evaluator's proposals and were much in line with the Commission's Regional Policy Priorities to the end of 1999. A total of 97 MECU was reallocated, primarily to measures most likely to lead to direct job creation (SMEs, technology and social inclusion). New measures were created for fish processing and for the development and application of communications technologies. The Commission also set a deadline for a comprehensive review of baseline data and quantified targets, with particular regard to net employment impacts. 3. 2. Overall reallocations of Structural Funds Under Objectives 1 and 6, all CSFs, SPDs and other forms of intervention were subject to a Mid-Term Review. The only exception was the new German Lander, where adjustments to the programmes are being made on an on-going basis. All these reviews led to financial reallocations of varying significance in order to improve the overall efficiency of the interventions. Due to the financial amounts involved, the scope of these changes is more wide-ranging for the larger and more complex CSFs than the SPDs. But it is not directly proportionate to the success or failure of the programmes concerned. Even a highly successful programme (e. g. the Irish CSF) can be improved and necessary corrections made without altering the main priorities and objectives. The scope for reallocations varied widely among programmes. Financial progress has been the general focus for these actions, especially with regard to the under-spending programmes. In addition, the reallocations also reflect the degree to which the EU priorities have been taken into account (see 4. 3). In general, limited adjustments (fine- tuning) were made to the programmes without affecting strategic priorities. In some cases (e. g. Italy-Obj. l, Portugal CSF), these modifications involved a transfer of resources from the slower spending programmes or sub-programmes to the faster spending ones, with due consideration given to other qualitative criteria. In most cases, however, modifications were made within programmes and between measures. Additional. uurces from the deflator were also used for refocusing certain actions, particularly in favour of employment creation. Graph 4 indicates the extent of the transfers of resources for each Member State. 24- The largest modifications were made to the Objective 1 CSF in Italy (around 700 Mecu, including non-programmed resources and CSF indexation for 1998), followed by Spain (more than 600 Mecu) and Greece (400 Mecu only for the national programmes). Despite the large financial amounts involved, these only represent between 2% (Spain) and 5% of the total Structural Funds allocations. In Ireland, financial transfers amounted to 160 Mecu of Structural Funds, or 3% of EU Structural Funds for the 1994-99 period. In Northern Ireland, no transfer of funds between sub-programmes was made, aside from the allocation of resources deriving from the deflator. In Finland and Sweden, the most significant changes were made through the merger of a number of measures in order to simplify the programme's structure. Conversely, in some relatively small programmes, significant changes were made, in relative terms, i. e. as a share of total Structural allocations. In the Belgian Hainaut, about 19% of available resources were shifted particularly towards the SME aid schemes. The case of Flevoland, despite a relatively high amount of Funds transfered (13%) involved several minor changes in the SPD measures, without affecting its overall aims and objectives. It is somewhat difficult to present a comprehensive overview of the nature of these adjustments. More qualitative insights relating to some key EU priorities only will therefore be presented in the next section. Graph 4: Mid-Term Review Reallocation of Structural Funds by Member State (in Mecu) Mecu D Arhount in Mecu • Percentage of S. F allocation 1 2 3 4 of which Nord-Pas-de-Calais (19,6 Mecus), Corse (10,1 Mecus), Guyane (36,1 Mecus) and Martinique (22,4 Mecus). no Mid-Term Review only ERDF reallocation of which Northern Ireland (69,8 Mecus), Merseyside (97,0) and Highlands and Islands (21 Mecu). - 2 5- 3. 3. Inclusion of EU priorities The extent to which the EU priorities laid down in the Commission's Guidelines for the Mid-Term Review have been addressed is not always easy to assess. New measures added or existing measures receiving additional funding may provide some indication however. Basic infrastructure - reinforcing TENs Reducing disparities in infrastructure endowment in the areas of transport, energy, telecommunications and environment has been recognised as a long term commitment of Structural Funds. This is reflected, in particular, in the share of total funding (more than 30%) allocated to this area in the current Objective 1 programming documents 1994-99. Although still a major priority for most regions, no significant resource was devoted to funding new infrastructure. In general, the emphasis was placed on specific TENs projects with the aim of establishing efficient connections and systems, particularly in the area of transport (box 15). Box 15: Airport programme - Italy Objective 1 The 10 Mecu airport programme, recently approved by the Commission, aims to improve or upgrade existing airports in the South of Italy. Following the Mid-Term Review, it was agreed to provide additional funding (50 Mecu) to this programme, exclusively for TENs projects, in order to further enhance regional endowments in this area. Productive environment - a strengthened focus on employment to increase the The results of the Mid-Term Reviews show that there has been scope focus on employment through a variety of actions (reinforcement of aid schemes, Territorial Employment Pacts, local development initiatives, etc. ). A number of decisions already taken (e. g. to include Territorial Employment Pacts within the mainstream programmes, Box 16) reveal a political aspiration and commitment to combat unemployment more effectively. The increased focus on the employment generating role of projects, particularly SME projects, is also in line with the direction of policy within most Member States. This is reflected in the CSFs and SPDs. In this context, the Mid-Term Review has questioned the effectiveness of policy instruments in terms of supporting job creation and responding to the need for more active employment policies through human resources and education measures. At a more operational level, emphasis was also given to the measurement of employment impacts (e. g. UK regions, Nord-Pas de Calais) and to take better account of employment issues in project selection criteria (e. g. Italy). Box 16: Territorial Employment Pacts In three Member States (Spain, Greece, Italy), the Monitoring Committees allocated CSF resources to a programme or a specific measure to finance action plans for Territorial Employment Pacts. In Spain (Objective I), priority was given to five pacts which should receive 55 Mecu of EU funding (deriving from the CSF deflator), for integration into a new multi-regional programme. In Greece, it is envisaged that the resources arising from the 1997 and 1998 deflator will be allocated to specific sub-programmes within regional programmes which are currently under review. In Italy, national authorities proposed, in the context of the Mid-Term Review, a multi-regional and multi-fund programme of 280 Mecu, of which 140 Mecu from Structural Funds will provide assistance to 9 pacts. 26- National aid schemes, which are being cofinanced out of Structural Funds, have also been reviewed in order to increase their focus on employment creation as well as on SMEs (see Box 17). Box 17: SMEs : the case of the Greek CSF The Greek authorities have refocused their Industry Operational Programme towards SMEs, with additional funding of 136 Mecu. The national aid scheme has been modified in line with the Commission's guidelines by increasing support to business services and by providing indirect assistance rather than direct grants. Intermediary bodies have been set up to manage the SME actions and projects will be selected on a competitive basis. A prior appraisal of the SME projects to be funded (about 1450 projects over 4200 received) was also undertaken. Environment and sustainable development Environment, as an element of sustainable development is one of the wider EU priorities most frequently addressed in the context of the Mid-Term Review. The pursuit of this objective is considered both through the introduction of specific measures and more importantly, through horizontal integration with other programme priorities. Because of objective difficulties (lack of indicators, methodological aspects) the results of the Mid- Term Review could not in all cases lead to direct operational conclusions for the adjustment of programmes. Specific attention was drawn to a lack of systematic environmental impact assessment of projects and to the gains in project quality that could be secured by systematising such assessments. The way has been prepared for improvements in environmental protection and general project selection procedures over the new programming period (box 18). Box 18: Mainstreaming environment Environment is a long-standing concern of most Member States. For the Structural Funds to contribute properly to environmental protection, they must take account of this issue in all project decisions, so as to minimise damage to the environment and to maximise positive benefits, by for example giving preference to projects with positive environmental impact (preventive action, eco-products,. ). The Commission services are actively involved in Environmental Impact Assessment actions (specific thematic study already launched, a Handbook on Environmental Assessment of Regional Development plans and EU Structural Funds Programmes prepared, etc) with a view to improving the integration aspect. Progress has been made in operationalising environment as a "core criterion" for selection of projects, in particular in Finland and Sweden. Recently, this criterion has also been introduced in Italy within the industry programme, for the selection of new industrial projects. Research and Technological Development While research and development facilities are, in general, already in place, the use of existing assets needed to be optimised. Some improvements have been made in particular, in increasing RTD investment towards the local business and industry (see Box 19). Box 19: Research and Technological Development: the case of Ireland The importance of RTD was recognised in the original CSF and its position has been enhanced further in the context of the Mid-Term Review. Some 37 Mecu was devoted to additional RTD measures within the Industry operational programme. Elements of competitive bidding and reduced aid rates will be introduced. The Monitoring Committee also agreed that certain services directed towards the industrial sector could be provided on a cost recovery basis. 27- Information Society With regard to Information society, further resources have been directed to this priority in recognition of its importance in helping to reduce the effects of peripherality, and thereby increasing investment and productivity (Box 20). Box 20: Information Society (Portugml, Spain) In Spmm, a new sub-axis (6. 6. ) on Information Society was created within the CSF, following a specific recommendation by the evaluator. An initial funding of 46 Meat has been earmarked to support this prierity. ERDF measures will be devoted to the provision ef professional and interactive services for universities, tourism and businesses. ESF measures wM focus on specific training actions for citizem as well as for local authorities. A hey innovation brought about in the Portuguese CSF has been the decision to introduce a new measure to support Information Society within the Telecommunications sub-programme. This measure has received some 7 Mecufrom ERDF. In addition, significant efforts have also been made in other CSF interventions, in particular in the Local Development Programme (PDDR) where all the local development agencies will be connected or in the Education programme, with the introduction of Internet access to secondary schools. Human Resources Development A refocusing on the unemployed was deemed necessary with regard to interventions targeting human resources development following three years of strong emphasis on young people. This emphasis on the unemployed, in line with the preventive approach adopted in the Employment guidelines, took various forms: strengthening integrated interventions (guidance and counselling, training, employment aids), boosting micro- enterprise creation subsidies, developing new forms of training, and local employment services. Within interventions targeting young people, support was given to moves away from conventional education and training programmes towards interventions in favour of young people at risk of exclusion and early-school leavers. In this regard, a closer integration between mainstream education systems and training and employment systems has been supported, especially with regard to upper-secondary level professional training (box 21). Box 21: Human resources development - Ireland Objective 1 The Mid-Term Review of the Human Resources Development Operational Programme (HRDOP) in Ireland was approved by the Monitoring Committee (MC) in October 1997. The Mid-Term Review was informed not only by the HRDOP, the CSF and the ESF Programme Evaluation Unit reports but also by national and Community policy orientations, such as Government White Papers and the EU Employment Strategy. The Mid-Term Review package adopted by the MC consisted of: a broad policy framework, financial re allocations with a revised financial plan (there was approximately 40 MECU in changes within the programme, with a net addition of 14 MECU to the OP) and the identification of qualitative recommendations of the OP and CSF evaluations requiring action. The major reallocation within the OP has been to the Early-school leavers measure (reinforced with 26 MECU) aiming to provide a minimum of 1000 additional places (with capacity within the financial envelope to provide 1400 plus, if necessary)y. Accompanying measures were strengthened, allowing for a more flexible approach. Provision was also made for the re-allocation and ring-fencing of additional progression places for early-school-leavers in other measures. Qualitative recommendations were adopted by the MC to refocus on long-term unemployed; increase the responsiveness of Industry training to employers needs; and reinforce investment in childcare infrastructure. - 2 8- Equal opportunities The theme of equal opportunities has been strengthened in many programmes, although this priority often appears as a horizontal objective and not systematically at measure level. However, significant efforts were made to refocus some measures towards this priority, especially under Human Resources interventions. Under ERDF interventions, specific measures for female entrepreneurship were introduced, especially in Italy and Greece, within the respective industrial programmes of those Member States(see Box 22). Box 22: Equal opportunities: reinforcing female entrepreneurship Access for women to enterprise creation schemes has been encouraged in a number of programmes. In Italy and Greece, a specific measure on female entrepreneurship (supported by national legislation) was introduced within their industrial Operational Programmes. In Finland, it is envisaged to guarantee access to a 'soft loan ' scheme, which is currently under scrutiny. IV. POLICY IMPLICATIONS AND FUTURE CHALLENGES The elements of the Mid-Term Review process presented earlier in this report had various implications for policy purposes. They drew attention to certain implementation and management issues, which could already be addressed during the remainder of the programming period. In addition together with an indication of future challenges arising from current evaluation practice, they provided a basis for reflection on strategic priorities for future programmes. 4. 1. Short-term implications for current delivery systems In discussing the Mid-Term Reviews, the point was frequently made that monitoring procedures tend to place greater emphasis on financial execution than on physical achievements. The majority of Mid-Term Evaluations have pointed to the need for new sets of indicators enabling better measurement of programme results and impacts. The identification of such indicators - as well as ensuring reliability and proper quantification - will increase the quality and scope of the next round of Mid-Term Evaluations. Some Member states have already taken steps to improve a number of elements in their current delivery systems, inter alia: - - fixing clear quantified targets and indicators for monitoring and evaluation (e. g. Austria, Portugal, Italy, United Kingdom); setting more appropriate selection criteria for projects (including employment and environmental criteria (e. g. Italy); simplifying management procedures, in particular for small programmes (e. g. Finland, Sweden); - securing synergies between different Structural Fund interventions (e. g. Portugal, Spain). 29 4. 2. Initial reflections for future programmes The Mid-Term Review was concerned with making necessary changes to the programmes to reflect changing circumstances and needs arising in the course of the implementation phase. In parallel with the Mid-Term Evaluations, both the Commission and the Member States have conducted a number of strategic analyses which look ahead to the period beyond the current interventions. These were designed to prepare for future Structural Funds interventions by considering longer term issues which could not be addressed during the remainder of the existing programmes. For example, the Commission has supplemented the limited information basis offered by the interim evaluation in Italy with a set of thematic studies relating to key CSF priorities: industry- related aid schemes, RTD, environment, transport infrastructure and human resources development. These studies set out to address certain key policy issues and also to provide recommendations for amending future CSFs. Whilst necessarily broad in outlook, these exercises have nonetheless usefully informed the Mid-Term Review process on issues such as the future re-orientation of spending priorities. The process of re-examining funding priorities has already commenced in the context of the Mid-Term Review in some Member States. Informal discussions and seminars have also taken place with national authorities to address some key issues for the future. The Commission has launched four thematic studies (RTD, SMEs, environment and the partnership principle) to help identify the broad directions of policy and facilitate the introduction of new arrangements for the period after 1999. 4. 3. The challenge of evaluation This report has shown that notable progress has been achieved in evaluation practice throughout the various Objectives 1 and 6 programmes. Important good practice features of the Mid-Term Review have been highlighted throughout the report including: - The soundness of evaluation (methodology and quality criteria); - The involvement of programme partnerships in the evaluation process; - The organisation and timing of the Mid-Term Review process; - The feedback role of evaluation in supporting reprogramming decisions as well as anticipating key issues. Lessons arising from this experience may represent a basis for guidance on best practice for the next programmes in order to improve further the management of the funds. The crucial issue is, thus, how to take stock of the results to promote a more systematic dissemination of good practice both within and between regions. In the light of experience to date, the Mid-Term Review shows that there is still a need for continuing improvements to the quality of the process in view of the challenges to be faced in the context of future programmes. Consolidating evaluation as a tool for decision making Evaluation is increasingly seen as a management tool, which should assist policy makers and programme managers in their tasks by providing them with important insights as to - 3 0- the effectiveness of measures and programmes. In its organisation, this function should link more closely the production of reliable and operational results to their integration in the decision making process. This would imply the creation, where unavailable, of adequate structures for managing the evaluation activities in the context of partnership. Improving monitoring systems and indicators Some significant actions have already been taken by national and regional authorities to address some gaps or deficiencies in their monitoring systems over the remainder of the programming period. Improvements in the definition of indicators and data collection will be necessary to enable these to perform their management tasks more effectively. Better integration between financial and physical indicators is a further, notable concern for programme managers. For future programmes, there is a need to ensure adequate quantification of baseline and target indicators and provide benchmark data to allow for better comparison within and between programmes. In this respect, the Commission will provide, in due course, a guidance document on methodological issues, including an indicative list of relevant indicators for the main areas of Structural Funds interventions. Supporting evaluation methodologies Substantial progress has been achieved in the area of evaluation methodology. The MEANS Programme, initiated by the Commission in 1994, has contributed to the development of that organisation's expertise, drawing on first hand experience of Structural Funds. There is no single and standardised evaluation methodology but rather a wide range of methods which should be customised so that they match the particular needs and situation of each programme. Efforts will be made, however, to ensure a better knowledge of evaluation techniques, not only among evaluators but also programme managers, who will have to appraise the quality of evaluation results. Promoting multilateral exchange of experience Since evaluation has been recognised as a major instrument for decision making, national and regional authorities are generally keen to have informal meetings to exchange experience and opinions on this matter. The Technical Evaluation Group at EU level is made up of representatives of Member States and Commission officials involved in the management of Structural Funds and might be considered a suitable forum in which to discuss evaluation issues and disseminate best practice. The main issues of common interest include the type and relevance of evaluation methodologies (e. g. measuring employment effects) and the deployment of evaluation results for policy purposes. Overall effectiveness of Structural Funds interventions will also be influenced by the extent to which national and regional authorities take necessary steps to improve their monitoring systems and are able to adopt good practice features to conduct the next Mid- Term Review process. -31 Annex I: List of boxes Box 1 Box 2 Box 3 Box 4 Box 5 Box 6 Box 7 Box 8 Box 9 Assessing the practicability of evaluation - ERDF Andalusia OP (Obj. l) Managing evaluations - the creation of ad-hoc structures Assessing the quality of an evaluation: the Irish CSF evaluation Hainaut (Obj. 1 ) : the HELM model Assessing effectiveness through programme indicators-the case of Merseyside- Objective 1 (4994-99) -actual outputs/results for ERDF interventions Important achievements in basic infrastructure. Some key achievements in the productive environment. Impact of training measures on beneficiaries Evaluation of agricultural interventions the case of PAMAF (modernisation of agricultural structures) Box 10: An example: the Spanish Objective 1 programme Box 11 : Enhancing the monitoring systems Box 12: Merseyside - scoring systems for ESF / ERDF projects Box 13: Mid-Term Review (I): the case of the Irish CSF Box 14: Mid-Term Review (II) : the case of Merseyside (Obj. l) Box 15: Airport programme - Italy Objective 1 Box 16: Territorial Employment Pacts Box 17: SMEs : the case of the Greek CSF Box 18: Mainstreaming environment Box 19: Research and Technological Development: the case of Ireland Box 20: Information Society (Portugal, Spain) Box 21: Human resources development - Ireland (Objective 1) Box 22: Equal opportunities: reinforcing female entrepreneurship 32- Annex 2 : Financial implementation of Objective I programmes Commitments/ total SF assistance under Objective 1 Payments/ total SF assistance under Objective 1 1994 1995 1996 1997 1998 (1st 1994 1995 1996 1997 I annual • cumulated I semester) I annual •cumulated ! 1998 (1st semester) i. _. : Financial implementation of Objective 1 programmes by Member States (31. 12. 97) % Financial implementation of Objective 1 Programmes bv Member-State (31/12/97) 75 70 65 ] 60 55 50 45 40 35 30 25 20 15 10 5 0 69 61 61 63 59 47 44 70 62 64 51 Belgium Germany Greece Spain France Ireland Itaiy Netherlands Austria Portugal United- Kingdom Total IE Commitments/total SF assistance 1994-97 £3 Payments/ Total SF assistance 1994-97 (Austria: SPD approuvée! in 1995) 33 Financial implementation of Objective 6 programmes Finland Commitments/total SF assistance under Objective 6 Finland 57 57 Payments/ total SFassistance under Objective 6 Finland 1994 1995 1996 1997 1998(1st I annual -^--curnjteted L; 1997 1998 (1st semester) Financial implementation of Objective 6 programmes Sweden Commitments/ total SFassistance under Objective 6 Sweden Payments/total SFassistance under Objective 6 Sweden 1994 • annual 1995 1996 1997 —•— currulated ~] 1998 (1st semester) 1996 1997 I annual •cumulated 1998 (1st semester) 34 Annex 3: List of reports Objective 1 reports Austria Belgium Germany Germany Germany Germany Germany Germany Germany Spain Spain Spain Spain Spain Spain Spain BURGENLAND : ZWISCHENEVALUATION DES ZIEL1-PROGRAMMS BURGENLAND: Endbericht HAINAUT : EVALUATION INTERMEDIATE DU DOCUP OBJECTIF 1 HAINAUT - RAPPORT FINAL - REGION WALLONNE BERLIN: ZWISCHENBEWERTUNG DER EU- STRUKTURFONDSINTERVENTIONEN IM LAND BERLIN IM ZEITRAUM 1994 BIS 1996 BRANDENBURG : ZWISCHENBEWERTUNG ZUM EINSATE DES EFRE, DES ESF UND DES EAGFL-A IM LAND BRANDENBURG 1994-1996 MECKLENBURG-VORPOMMERN : ZWISCHENBEWERTUNG DES EINSATZES DER EUROPAISCHEN STRUKTURFONDS (EAGFL, EFRE, ESF) FUR DIE PERIODE 1994 BIS 1996 IM BUNDESLAND MECKLENBURG-VORPOMMERN SACHSEN : ZWISCHENEVALUIERUNG DES EINSATZES DER EUROPAISCHEN STRUKTURFONDS IM FREISTAAT SACHSEN 1994-1996 SACHSEN-ANHALT: ZWISCHENBEWERTUNG DES EINSATZES DER EU-STRUKTURFONDS IN SACHSEN-ANHALT FUR DEN ZEITRAUM 1994 BIS 1996 THURINGEN : DIE EUROPAISCHEN STRUKTURFONDS IN THURINGEN : ZWISCHENBEWERTUNG DES MITTELEINSATZESVON 1994 BIS 1996 NEUE BUNDESLANDER : DIE EUROPAISCHEN STRUKTURFONDS IN DEN NEUEN BUNDESLANDERN - ZWISCHENBEWERTUNG DES MITTELEINSATZES VON 1994 BIS 1996 (BERLIN/ BRANDENBURG/MECKLENBURG- VORPOMMERN/SACHSEN/SACHSEN- ANHALT/THURINGEN) PO CANARIES : EVALUACION INTERMEDIA PROGRAMA OPERATIVO DE CANARIAS (FEDER) 1994-1999 OP INCENTIVOS REGIONALES : EVALUACION INTERMEDIA DEL PROGRAMA OPERATIVO DE INCENTIVOS REGIONALES FEDER (1994-1999) PARA LAS REGIONES ESPANOLAS INCLUIDAS EN EL OBJETIVO N°l DELOS FONDOS ESTRUCTURALES EUROPEOS CANTABRIA : ESTUDIO DE EVALUACION INTERMEDIA DEL PROGRAMA OPERATIVO DE CANTABRIA (FEDER) 1994-1999 CASTILLA Y LEON : EVALUACION DE LOS PROGRAMAS OPERATIVOS DEL FEDER Y DEL FSE 1994-1999 DE CASTILLO Y LEON EVALUACION INTERMEDIA DE LA SUBVENCION GLOBAL FEDER-CDTI EN LAS REGIONES OBJETIVO 1 (1994-1999) ESTUDIO DE EVALUACION INTERMEDIA DEL PROGRAMA OPERATIVO DE INFRAESTRUCTURA CIENTIFICA DEI PERIODO 1994-1999, EN REGIONES ESPANOLAS INCLUIDAS EN EL OBJ. 1, COFINANCIADO CON FEDER Y COORDINADO POR CICYT EVALUACI6N INTERMEDIA DEL MARCO COMUNITARIO DE APOYO-REGIONES OBJETIVO 1- 1994-1999-ESPANA 35- Apr-98 OIR Apr-97 SEMA GROUP/RIDER- UCL May-97 PROGNOS BERLIN May-97 TROJE BERENTUNG HANN, MUNCHEN Apr-97 BUSTRO GmbH ROSTOCK May-97 DIW BERLIN May-97 ISW HALLEFLEIPZIG May-97 IFO INSTITUT FUR WIRTSCHAFTSFOR- SCHUNG DRESDEN Sep^97 IFO INSTITUT FOR WIRTSCHAFTSFOR- SCHUNG DRESDEN Jun-97 SERVICIOS OMICRON, S. A. May-97 SERVICIOS OMICRON SA Jun-97 INFYDE Jan-97 (UNIV. VALLADOLID) Jun-97 INFYDE/CDTI Jul-97 PRICE WATERHOUSE Sep-97 QUASAR Spain Spain Spain Spain Spain Spain Spain Spain Spain Spain Spain France France France France Jul-97 (UNIV. VALLADOLID) Jun-97 GEDESA May-97 AGRO CONSULTING INTERNATIONAL May-97 Jul-96 ARANDA Y BELTRAN S. L. ESTUDIOS Y ANALISIS SOCIO ECONOMICOS UNIVERSIDAD DE OVIEDO Dec-97 ANDERSON ARTHUR Aug-97 ECOTEC Jul-97 Mar-98 UNIVERSIDAD DE CASTILLA-LA MANCHA INSTITUTO DE DESARROLLO REGIONAL, FUNDACI6N UNIVERSITARIA ARENAL GRUPO CONSULTOR SL INSTITUTO DE DESARROLLO REGIONAL, FUNDACION UNIVERSITARIA FERE CONSULTANTS Jun-97 Feb-98 ERNST & YOUNG Dec-97 ACT Feb-96 CODE/ATHOS CASTILLA Y LEON : EVALUACION DE LOS PROGRAMAS OPERATIVOS DEL FEDER Y DEL FSE 1994-1999 DE CASTILLO Y LEON CEUTA : 2e ESTUDIO DE EVALUACION INTERMEDIA DEL MARCO DE APOYO COMUNITARIO DE LAS REGIONES OBJETIVO N°l PARA CEUTA (PO CEUTA 1994-1999) GALICIA: EVALUACION INTERMEDIA DEL PROGRAMA OPERATIVO DE GALICIA 1994-1999 cofinanciado per el FEDER (PO GALICIA) PO MELILLA : EVALUACION INTERMEDIA DEL PROGRAMA OPERATIVO FEDER DE MELILLA 1994- 1999 ASTURIAS : EVALUACION DEL PROGRAMA OPERATIVO DEL PRINCIPADO DE ASTURIAS (FEDER) 1994-1999 - INFORME PREPARATORIO OP LOCAL: ESTUDIO DE EVALUACION INTERMEDIA DEL PROGRAMA OPERATIVO LOCAL (FEDER) 1994-1999 PARA LAS REGIONES ESPANOLAS INCLUIDAS EN EL OBJETIVO N° 1 DE LOS FONDOS ESTRUCTURALES EUROPEOS OP MEDIO AMBIENTE: EVALUACION INTERMEDIA DEL PROGRAMA OPERATIVO DEL MEDIO AMBIENTE LOCAL (FEDER) 1994-1999 EN REGIONES ESPANOLAS INCLUIDAS EN EL OBJETIVO NO 1 DE LOS FONDOS ESTRUCTURALES EUROPEOS OP CASTILLA-LA-MANCHA:SEGUNDO INFORME DE EVALUACION-PROGRAMA OPERATIVO FEDER DE CASTILLA-LA MANCHA 1994-1999 ANDALUCf A : EVALUACION INTERMEDIA DEL P. O. ANDALUCIA 1994-1999 ANDALUCIA : SUBVENCI6N GLOBAL DE ANDALUCfA 1994-1999 : INFORME DE EVALUACION INTERMEDIA DONANA : EVALUACI6N INTERMEDIA DEL P. O. DONANA Ha PHASE (OBJ. l) Mar-98 May-97 CORSE : EVALUATION INTERMEDIAIRE DOCUP OBJECTIF 1 CORSE 1994-1999 GUYANE : EVALUATION INTERMEDIAIRE DE LA MISE EN OEUVRE EN GUYANE DES PROGRAMMATIONS REGIONALES (1994-1999) COFINANCEES PAR L'UNION EUROPEENNE (DOCUP ET REGIS II) - SYNTHESE DU RAPPORT NORD-PAS-DE-CALAIS (AVESNES / DOUAI / VALENCIENNES): PROGRAMME OBJECTIF 1 (1994- 1999): EVALUATION INTERMEDIAIRE, EVALUATION DENSEMBLE DU PROGRAMME, ASPECTS SECTORIELS EVALUATION DES MESURES GUADELOUPE. EVALUABILITE DU DOCUP GAUDELOUPE : RAPPORT D'ETUDE (DOCUP 1994- 1999) 36 Greece Greece Greece Greece Greece Greece Greece Greece Greece Greece Greece Greece Ireland Ireland Ireland Ireland Ireland Ireland Ireland Ireland Ireland Ireland Ireland Italy Netherlands Portugal Portugal PELOPONNESE : INTERMEDIATE EVALUATION OBJECTIVE 1 - OPERATIONAL PROGRAMME 1994- 1999 (a) VOL. A (b) VOL. B GRECE OCCIDENTALE (GRECE OUEST): INTERMEDIATE EVALUATION OBJECTIVE 1 - OPERATIONAL PROGRAMME CDS P. O. EPIRIUS : INTERMEDIATE EVALUATION 1994-1999 OBJECTIVE 1 - OP CRETE : INTERMEDIATE EVALUATION OBJECTIVE 1 - OPERATIONAL PROGRAMME 1994-1999 THESSALIE: INTERMEDIATE EVALUATION OBJECTIVE 1 -O. P. 1994-1999 ILES IONIENNES : INTERMEDIATE EVALUATION OBJECTIVE 1 - O. P. 1994-1999 MACEDOINE CENTRALE + THRACE : INTERMEDIATE EVALUATION OBJ. 1 - O. P. 1994-99 MACEDOINE OCCIDENTALE: INTERMEDIATE EVALUATION OBJECTIVE 1 O. P. 1994-1999 MACfEDOINE CENTRALE: INTERMEDIATE EVALUATION OBJECTIVE 1 -O. P. 1994-1999 PO GRECE CONTINENTALE (=CENTR. ): INTERMEDIATE EVALUATION O. P. OBJ. 1 - 1994-1999 EGEE DU SUD : INTERMEDIATE EVALUATION OBJECTIVE 1 ILES DE LA MER EGEE SUD - OPERATIONAL PROGRAMME 1994-1999 CSFGRECE : OBJ. l - 1994-1999 EXTERNAL EVALUATION OF THE OPERATIONAL PROGRAMME FOR INDUSTRIAL DEVELOPMENT 1994-1999 : MID-TERM REVIEW CSF MID-TERM EVALUATION MID TERM EVALUATION OF OPERATIONAL PROGRAMME FOR ECONOMIC INFRASTRUCTURE MID-TERM EVALUATION OF THE LOCAL, URBAN AND RURAL DEVELOPMENT O. P. MID-TERM EVALUATION OF THE HUMAN RESOURCES DEVELOPMENT-O. P. 1994-1999 MID TERM EVALUATION OF THE O. P. FOR ENVIRONMENTAL SERVICES 1994-1999 OPERATONAL PROGRAMME FOR TRANSPORT - MID TERM EVALUATION MID-TERM EVALUATION OF THE OPERATIONAL PROGRAMME FOR TOURISM 1994-1999 OPERATIONAL PROGRAMME FOR AGRICULTURE, RURAL DEVELOPMENT AND FORESTRY 1994-1999 MID-TERM EVALUATION OF OPERATIONAL PROGRAMME FOR FISHERIES 1994-1999 MID-TERM EVALUATION : REGIONAL IMPACT OF THE COMMUNITY SUPPORT FRAMEWORK FOR IRELAND 1994-1999 VALUTAZIONE DI MEDIO TERMINE QUADRO COMUNITARIO DI SOSTEGNO 1994-99 delle REGIONI ITALIANE DELL'OBIETTIVO 1 FLEVOLAND: ON-GOING EVALUATIE DOELSTELLING 1 PROGRAMMA AMBIENTE:AVALIACAO INTERCALAR DO PROGRAMA AMBIENTE - EVALUATION DU SOUS- PROGRAMME ENVIRONNEMENT DU CCA II PT ALENTEJO:AVALIACAO INTERCALAR DO PROGRAMA OPERACIONAL DA REGIAO DO ALENTEJO - CCA 1994-1999 May-97 OMAS LTD Jun-97 EEO GROUP SA Apr-97 May-97 May-97 PROPTIKI SA. - METPON ASTIKI DIACHIRISI S. A. YPODOMI LTD May-97 PROOVADISMA LTD Apr-97 INDECO S. A. May-97 EXANTAS LRD May-97 EUROTEC LTD May-97 LDK May-97 ENVIPLAN Mar-98 Feb-97 REMACO ERNST & YOUNG Apr-97 Feb-97 ESRI INDECON Jan-97 GOODBODY Feb-97 GOODBODY Jan-97 ERM Feb-97 Feb-97 Jan-97 Feb-97 DKM ECONOMIC CONSULTANTS FITZPATRICK ASSOCIATES FITZPATRICK ASSOCIATES INDECON Feb-97 FITZPATRICK ASSOCIATES Jul-97 Jan-97 Feb-97 NUCLEO DI VALUTAZIONE/INEA E ISFOL RESEARCH VOOR BELEID QUATERNAIRE (PINHO PAULO) Feb-97 CESO I&D - 3 7- Portugal Portugal Portugal Portugal Portugal Portugal Portugal Portugal Portugal Portugal Portugal Portugal Portugal Portugal Portugal Portugal Portugal Portugal AMBIENTE:ESTUDO DE AVALIACAO INTERCALAR DA INTERVENCAO OPERACIONAL-RENOVACAO URBANA DO PROGRAMA'AMBIENTE E REVITALIZACAO URBANA DO QUADRO COMUNITARIO DE APOIO 1994-1999 SOCIAL:SUB-PROGRAMA INTEGRAR AVALIACAO INTERCALAR LISBOA E VALE DO TEJO:ESTUDO DE AVALIACAO INTERCALAR DO PROGRAMA OPERACIONAL DA REGIAO DE LISBOA E VALE DO TEJO PPDR:ESTUDO DE AVALIACAO INTERCALAR PROGRAMA OPERACIONAL PROMOCAO DO POTENCIAL DE DESENVOLVIMENTO REGIONAL - QUADRO COMUNITARIO DE APOIO 1994-1999 SAUDE:AVALIACAO INTERCALAR DA INTER VENCAO OPERACIONAL DA SAUDE ENERGIA:AVALIACAO INTERCALAR DO PROGRAMA ENERGIA E DO PROJECTO DE GAS NATURAL (INTERREG II - CONCLUSAO DAS REDES DE ENERGIA) DO QCA II (1994-1999) PO CENTRO:ESTUDO DE AVALIACAO INTERCALAR DO P. 0. REGIAO CENTRO PO NORTE:ESTUDO DE AVALIACAO INTERCALAR DO PRONORTE PEDIP II: AVALIACAO INTERCALAR DO PEDIP II: RELATORIO FINAL INTERCALAR PESCA : AVALIACAO INTERCALAR DA INTER VENCAO OPERACIONAL DAS PESCAS TURISMO: ESTUDO DE AVALIACAO INTERCALAR DA INTER VENCAO OPERACIONAL TURISMO E PATRIMONII CULTURAL DO PROGRAMA MODERNIZACAO DO TECIDO ECONOMICO AGRICULTURA : QCA II: ESTUDO DE AVALIACAO INTERCALAR DA INTERVENCAO OPERACIONAL AGRICULTURA : PROGRAMA MODERNIZACAO DO TECIDO ECONOMICO - PERIODO 1994-1996 PROFAP : PROGRAMA INTEGRADO DE FORMACAO PARA A MODERNIZACAO DA ADMINISTRACAO PUBLICA - PROFAP 2 : ESTUDO DE AVALIACAO INTERCALAR ALGARVE : ESTUDO DE AVALIACAO INTERCALAR DO PROGRAMA OPERACIONAL DO ALGARVE TELECOMUNICACOES : ESTUDO DE AVALIACAO DA INTERVENCAO OPERACIONAL DAS TELECOMUNICACOES DO PROGRAMA INFRA- ESTRUTURAS DE APOIO AO DESENVOVIMENTO DO QCA SUB-PROGRAMME TRANSPORTES: AVALIACAO INTERCALAR DA INTERVENCAO OPERACIONAL DOS TRANSPORTES DO PROGRAMA INFRA- ESTRUTURAS DE APOIA AO DESENVOLVIMENTO DO QUADRO COMUNITARI DE APOIO 1994-1999, NO PERIODO ENTRE 1994 E 1996 SUB-PROGRAMME COMERCIO E SERVICOS :ESTUDO DE AVALIACAO INTERCALAR DA INTERVENCAO OPERACIONAL COMERCIO E SERVICOS ACORES=ESTUDO DE AVALIACAO INTERCALAR DO PROGRAMA OPERACIONAL DA REGIAO AUTONOMA DOS ACORES DO QUADRO COMUNITARIO DE APOIO 1994-1999 38- Mar-97 Jan-97 PARTEX-INSTITUTO DE GESTAO E ALIENACAO DO PATRIMONIO HABITACIONAL DO ESTADO CIES Jan-97 CEDRU Jan-97 CEETA Feb-97 CESO I&D Jan-97 IESE Jan-97 CEDRU Jan-97 QUATERNAIRE Mar-97 Apr-97 UNIVERSIDADE CATOLICA PORTUGUESA CESO I&D Feb-97 DELOITTE & TOUCHE Jun-97 AGRO. GES Mar-97 QUATERNAIRE Jan-97 CEDRU Mar-97 SILICON Apr-97 CISED Nov-97 CESO I&D Feb-97 CESO I&D Portugal Portugal Portugal Portugal United Kingdom (IRN) United Kingdom (IRN) United Kingdom (IRN) United Kingdom (IRN) United Kingdom (IRN) United Kingdom (IRN) United Kingdom (IRN) United Kingdom (IRN) United Kingdom (IRN) United Kingdom United Kingdom SUB-PROGRAMME EDUCACAO:AVALIACAO INTERCALAR DE PRODEP II: EDUCACAO CIENCIA E TECNOLOGIA:ESTUDO DE AVALIACAO INTERCALAR DO SUB-PROGRAMA CIENCIA E TECNOLOGIA OP PESSOA: ESTUDO DE AVALIACAO INTERCALAR PROGRAMA OPERACIONAL EMPREGO E FORMACAO PROFISSIONAL DO QCA II 1994-1999 (PROGRAMMA PESSOA) CSF PORTUGAL-OBJ. l:AVALIACAO INTERCALAR DO QUADRO COMUNITARIO DE APOIO II NORTHERN IRELAND SINGLE PROGRAMME 1994- 1999 - MID TERM REVIEW - EXTERNAL EVALUATION MID-TERM REVIEW OF THE ENVIRONMENTAL SERVICES AND PROTECTION SUB-PROGRAMME MEASURES 4. 1. 8 AND 4. 19. OF THE SUB- PROGRAMME FOR AGRICULTURE AND RURAL DEVELOPMENT : MID TERM EVALUATION MID-TERM EVALUATION OF THE NORTHERN IRELAND SINGLE PROGRAMME FOR AGRICULTURE AND RURAL DEVELOPMENT (SPARD) ENERGY SUB-PROGRAMME 1994-1999 : MID-TERM REVIEW MID TERM EVALUATION OF THE TRANSPORTATION SUB-PROGRAMME, 1994-1999 MID-TERM EVALUATION OF THE INVESTMENT IN THE DEVELOPMENT OF PEOPLE SUB-PROGRAMME, NORTHERN IRELAND SINGLE PROGRAMME 1995-99 MID TERM REVIEW OF PHYSICAL & SOCIAL ENVIRONMENT PROGRAMME (EU S. F. ) Scp-97 QUATERNAIRE Feb-98 IESE/GEOIDEIA Mar-97 IESE (INSTITUTO DE ESTUDOS SOCIAIS E ECONOMICOS jun-98 CISEP May-97 COLIN STUTT CONSULTING Dec-96 PIEDA CONSULTANTS Feb-97 COOPERS & LYBRAND Mar-97 ERM Jan-97 ERM Jan-97 ERM Dec-96 LRDP Jan-97 COOPERS & LYBRAND MID-TERM EVALUATION OF THE TOURISM SUBPROGRAMME 1994-1999 Fcb-97 NIERC MERSEYSIDE. MID-TERM EVALUATION OF MERSEYSIDE OBJECTIVE ONE PROGRAMME HIGHLANDS AND ISLANDS - OBJECTIVE 1 PROGRAMME INTERMEDIATE ASSESSMENT Nov-96 PIEDA PLC Jun-97 SQW Objective 6 report Finland SUOMEN TAVOITE 6 - OHJELMAN ARVIOINNIN VALIRAPORTTI [FI] Jan-98 Sweden HALVTIDSUTVARDERING AV SVERIGES NAL-6- PROGRAM Nov-97 HELSINGIN YLIOPISTON TIETOPALVELUT OY/ETLATIETO OY/OY FINNAGRO AB/INT. DEVEL. IRL/PE LLERVON NORD-REGIO-EPRC -39 ISSN 0254-1475 COM(98) 782 final DOCUMENTS EN 04 13 09 16 Catalogue number : CB-CO-98-786-EN-C Office for Official Publications of the European Communities L-2985 Luxembourg ÙO