Special Report No 15/98 on the assessment of Structural Fund intervention for the 1989-1993 and 1994-1999 periods together with the Commission's replies (submitted pursuant to Article 188c(4)(2) of the EC Treaty)
16. 11. 98 EN Official Journal of the European Communities C 347/1 I (Information) COURT OF AUDITORS SPECIAL REPORT No 15/98 on the assessment of Structural Fund intervention for the 1989-1993 and 1994-1999 periods together with the Commission’s replies (submitted pursuant to Article 188c(4)(2) of the EC Treaty) (98/C 347/01) TABLE OF CONTENTS Paragraph number Page 1. INTRODUCTION. 1. 1 — 1. 4 2. THE SCOPE OF THE AUDIT. 2. 1 — 2. 3 3. SUMMARY OF THE MAIN OBSERVATIONS. 3. 1 — 3. 12 4. GENERAL ISSUES. 4. 1 — 4. 36 Introduction. 4. 1 — 4. 11 Reference framework. 4. 12 — 4. 13 Statistical bases and information concerning objectives. 4. 14 — 4. 15 Indicators. 4. 16 — 4. 21 Taking evaluation into account. 4. 22 — 4. 27 The objectivity of evaluations. 4. 28 — 4. 30 The evaluation contracts awarded by the Commission. 4. 31 — 4. 36 5. THE EX POST EVALUATION OF THE OBJECTIVE Nos 1 AND 2 CSFs. (1989-1993) 5. 1 — 5. 17 Introduction. 5. 1 The ex post evaluation of the Objective No 1 CSFs (1989-1993). 5. 2 — 5. 8 Ex post evaluations of Objective No 2 (1989-1993). 5. 9 — 5. 17 3 3 4 5 5 6 7 7 8 8 9 9 9 10 11 C 347/2 EN Official Journal of the European Communities 16. 11. 98 Paragraph number Page 6. ONGOING ASSESSMENTS (1994-1999). 6. 1 — 6. 27 The development and integration of the principles of ongoing assessment 6. 1 — 6. 7 Implementation procedures. 6. 8 — 6. 13 Priority given to evaluation objectives. 6. 14 — 6. 20 Approaches, methods and techniques. 6. 21 — 6. 26 Implementation of Objective No 2 programmes (1994-1996). 6. 27 7. IMPACT OF THE STRUCTURAL FUNDS AND EVALUATION MODELS 7. 1 — 7. 22 Impact of CSFs and programmes. 7. 1 — 7. 15 Results of the models as regards the impact of the Structural Funds. 7. 1 — 7. 13 GDP growth. 7. 2 — 7. 3 Real convergence in terms of per capita GDP. 7. 4 — 7. 8 Employment and unemployment. Effectiveness of CSF instruments. 7. 9 7. 10 Additionality and joint-financing requirements. 7. 11 — 7. 12 Durable development. 7. 13 Role of macroeconomic the Structural Funds’ evaluation decision-making process. in 7. 14 — 7. 15 Impact of sub-programmes, measures and projects. 7. 16 — 7. 22 8. CONCLUSION. 8. 1 — 8. 19 TABLES. ANNEXES. The Commission’s replies. 13 13 14 15 15 16 17 17 17 17 17 18 18 18 18 19 19 20 22 31 40 16. 11. 98 EN Official Journal of the European Communities C 347/3 1. INTRODUCTION 1. 1. Under the 1988 reform of the Structural Funds (SFs), the Commission and the Member States were required to evaluate European structural policies. In 1993 this requirement was strengthened by the amendments to the Regulations that were introduced in the context of the revision of the SF reform (1994-1999) and the standard clauses of the new Community Support Frameworks (CSFs) and Single Programming Documents (SPDs). 1. 4. The 1993 versions of the Regulations reiterate the responsibilities of both the Member States and the Commission with regard to assessment and stress that it is to be carried out within the partnership system (3). In this context, the Commission has taken the initiative concerning the supervision and general coordination of the implementation of evaluation and, in accordance with Article 16 of Regulation (EEC) No 2052/88, submits every three years an evaluation report on the progress achieved towards economic and social cohesion and, in particular, the contribution made by the Funds and national co-financing. 1. 2. Evaluation, an essential the sound management of the SFs, is defined in the main provisions of the Regulations (1) as the evaluation of effectiveness on the following three levels: tool for 2. THE SCOPE OF THE AUDIT (a) the overall impact of the structural measures on the objectives set out in Article 130A of the Treaty; (b) the overall impact of the measures laid down in the CSFs; (c) the specific impact of each operational intervention (programmes and other forms of intervention). 1. 3. There are three main phases to this evaluation: in evaluating the progress achieved 2. 1. The general aim of the Court’s audit (4) was to check the Community’s structural interventions, in terms of systems and evaluation procedures and at a methodological level. The audit looked at the methods used to evaluate structural interventions both by the Commission and under the partnership system. It also examined the legality and the appropriations earmarked for evaluation in the budget of the European Communities and the quality of the financial management of these appropriations. the utilisation of regularity of (a) prior enable appraisals (2), which correct identification of the socio-economic problems that need to be resolved and the objectives to be achieved, on the one hand, and, on the other hand, of the strategy adopted and the means available for this purpose; a (b) ongoing assessments, which are carried out during implementation of a programme and make it possible to check whether the objectives are still appropriate and the desired results on the way to being achieved; (c) ex post evaluation, which is carried out after the programme has been completed. Its aim is to analyse whether the results and impact obtained are to be considered positive, bearing the long-term effects of structural measures can only be appreciated several years after their completion. in mind that (1) In particular, these are: (a) Article 6 of the framework Regulation (Regulation EEC No 2052/88 as amended by Regulation EEC No 2081/93) on the tasks of the Structural Funds and their effectiveness and on the coordination of their activities; (b) Article 26 of the Funds’ coordinating Regulation (Regulation EEC No 4253/88 as amended by Regulation EEC No 2082/93). (2) ‘Prior appraisal’ is also sometimes called ‘prior assessment’. The Court prefers to adopt the expression ‘prior appraisal’ as it appears in the Regulations. 2. 2. The audit also involved an appraisal of the techniques used for evaluating the socio-economic impact of structural measures. Although it did not aim to evaluate the programmes directly with the intention of producing new scientific results, it nevertheless made use of the results of work by experts so as to obtain a better understanding of the relationship between the economic policy instruments used and the results obtained. 2. 3. Specific audits were carried out on the ex post evaluations of the CSFs and Operational Programmes (OPs) for Objectives Nos 1 and 2 for the 1989-1993 programming period and on the ongoing assessments of the CSFs, SPDs and OPs for the 1994-1999 programming period. All the ex post evaluations for the 1989-1993 Objective No 1 CSF were taken into account, as well as (3) Community operations are to complement corresponding national operations or contribute to them. They are set up via close consultation, known as ‘partnership’, between the Commission, the Member State concerned and the competent local bodies, social partners and other regional and full is conducted competent bodies. ‘Partnership’ compliance with the respective legal and institutional, financial powers of each of the parties concerned. in (4) In its Annual Reports concerning, in particular, the financial years 1990 (chapter 7), 1993 (chapters 6, 7, 8 and 9) and 1994 (chapter 4), the Court had already reported on the numerous shortcomings in the field of evaluation for the 1989 programming period. C 347/4 EN Official Journal of the European Communities 16. 11. 98 approximately 20 % of the ex post evaluation reports for the 1989-1993 Objective No 2 CSF and 15 % of the ongoing assessment reports for the 1994-1999 CSFs, selected on the basis of their geographical distribution and the nature of the interventions (CSFs, SPDs or OPs). In addition, the Court analysed 15 out of the 25 evaluation reports that dealt mainly with macroeconomic aspects. The audit was carried out at the Commission and in eight Member States (Belgium, Greece, Spain, France, Ireland, Italy, the Netherlands and Portugal), where the auditors consulted the assessors and national authorities concerned. are often inadequate. The Member States should improve their quality (see paragraphs 4. 14-4. 21). 3. 5. Many evaluations are completed late, which may proportionately delay the adjustment of programmes. Furthermore, the decision-making system could take the performance or evaluation effectiveness of intervention schemes better into account for future programmes (see paragraphs 4. 22-4. 27). results concerning 3. SUMMARY OF THE MAIN OBSERVATIONS 3. 6. With regard to the objectivity of evaluations, the Commission makes no on-the-spot checks of the consistency and reliability of indicators or the systems that produce them (see paragraphs 4. 29-4. 30). at and spread, leading to its recognition 3. 1. Evaluation developed the Commission’s initiative, mainly during the 1994-1999 programming period, in particular in the context of the ERDF, in the field of management and programming. As from the beginning of this programming period, evaluation has been systematically applied to the first two phases (prior appraisal and ongoing assessment) and will be applied to the third at the end of the period (ex post evaluation) (see paragraphs 4. 2-4. 8). 3. 2. Although the Commission has the figures regarding its own direct expenditure on evaluation, it does not always have access to all the information for expenditure on evaluations carried out under the partnership system (see paragraphs 4. 9-4. 11). 3. 3. The Commission made a considerable contribution towards spreading the idea of evaluation, even though it did not introduce an adequate framework for evaluation indicators and methods in its programme on ‘Methods for Evaluating Activities of a Structural Nature’ (MEANS). In view of the complex nature of the methodological aspects of evaluation, there is no single methodology that can be applied to all specific situations. It is, however, important that the Commission should be in a position to be able to supply methodological frameworks that provide appropriate guidelines for the implementation of evaluations suited to specific situations (see paragraphs 4. 12-4. 13). 3. 4. The quality of the indicators is closely linked to the reliability of the information gained from the monitoring of existing interventions and statistical bases. The Member States are responsible for these latter and they in the 3. 7. There are still significant shortcomings Commission’s procedures for awarding evaluation contracts to consultants. The choice of consultants is not always clearly justified. Contract specifications are very general in terms of methodological techniques, indicators and the need to achieve results. This leads to delays and disturbances in the submission of evaluation reports, qualitatively variable analyses, and, sometimes, vague recommendations (see paragraphs 4. 31-4. 36). forward 3. 8. The implementation of the ex post evaluation methodology for Objectives Nos 1 and 2 (1989-93) came up against statistical shortcomings, a lack of objectives, an absence of quantified indicators, an ineffective monitoring of impact and was also not sufficiently stringent. Nevertheless, these evaluations revealed that the SFs did indeed contribute Community added value, both in terms of the achievements, results and effects of the CSFs, OPs and measures and from the point of view of programming, organisation, monitoring, coordination, (see partnership paragraphs 4. 15-4. 19, 5. 1, 5. 4, 5. 10-5. 16). additionality and 3. 9. With regard to the ongoing assessment of the 1994-1999 SF interventions carried out in partnership under the aegis of the Monitoring Committees, an examination of the physical and financial results always plays an important role in mid-term assessment, even though there may be variations in the priorities granted to the various evaluation objectives. The evaluation of the effects is hampered in particular by problems with indicators and the modelling of effects. This type of evaluation is still often only of relative value in terms of the decision-making process, the management of projects, reprogramming future interventions (see paragraphs 4. 16-4. 21, 6. 5, 6. 7, 6. 14-6. 26). preparation and the of 16. 11. 98 EN Official Journal of the European Communities C 347/5 In the specific case of Objective No 2, the 3. 10. evaluation revealed the following weaknesses: (a) a mismatch between programme implementation and Community funding, with significant carry-overs of appropriations from one programme to another (about 12 % on average for the shift from the 1994-1996 period the 1997-1999 one)(see paragraph 6. 27); to (b) the succession of short programming periods does not permit an optimum use of evaluation results (see paragraph 4. 23); (c) indirect effects are still not measured sufficiently to determine the impact of structural interventions, in particular as regards employment (see paragraphs 5. 12-5. 15). 3. 11. Despite failings in the programming, implementing and monitoring systems, the macroeconomic evaluations revealed positive results, in particular in respect of the impact of structural measures (increases in GDP, effects on supply and demand, improvements in terms of employment and convergence and the effectiveness of the CSF instruments), (see paragraphs 7. 1-7. 14). 4. 2. Since the 1988 reform, evaluation has been explicitly included within the partnership system (Article 26 of Regulation (EEC) No 4253/88 — described more fully in the amended Regulation of 20 July 1993). During the first programming period and up until 1995, evaluations were nevertheless mostly carried out at the in Commission’s particular the ex-post ones — were hampered by a lack of inadequate monitoring. Moreover, generally speaking, they were not really taken into consideration and discussed in detail by the Monitoring Committees. initiative (5). These evaluations — specific development strategies and 4. 3. The Commission initially concentrated mainly on setting up its own evaluation mechanism for all the CSFs covering the reform’s five priority objectives (6). Among the Commission’s initiatives in connection with the development of evaluation and evaluation methods, mention must be made, in particular, of the 1992 creation of the technical group on the ‘Evaluation of Regional Policies’ and, at a methodological level, the launching of the programme on ‘Methods for Evaluating Activities of a Structural Nature’ (MEANS) in 1994. 3. 12. Evaluations of sub-programmes and measures continue to be inadequate. Furthermore, they come up against technical difficulties caused by the dispersion and inconsistency of many measures and sub-programmes (see paragraph 7. 17). 4. 4. For the 1994-1999 period, the Commission was responsible for a new evaluation system, which is now systematically applied to the first two phases (prior appraisal and ongoing assessment) and will later be applied to the ex post phase. 4. GENERAL ISSUES Introduction is to determine 4. 1. The aim of evaluation the socioeconomic effects of the measures implemented, i. e. to examine the results obtained, compare them with the resources invested and analyse their cost. Furthermore, all these points must also be looked at in relation to forecasts. However, this evaluation task presupposes a good statistical basis, quantified objectives established on the basis of a system of indicators and the development of a methodological and analytical framework. At the same time, it must be supported by an analysis of the relationships between the means used, final results and intermediate results. The Court found failings at all these levels, and, despite real progress that has helped consolidate the system of evaluation, these have so far limited the usefulness of the evaluations carried out in connection with the Structural Funds. 4. 5. Although ongoing assessment is not explicitly provided for in the SF Regulations, as of 1995-1996 it has been applied to all the CSFs, SPDs and OPs for the 1994-1999 programming period. In concrete terms, it falls under the Monitoring Committees to turn it into an integrated system, allowing the partnership (5) This concerns the following evaluations: (a) For the 1989-1993 programming period: — prior evaluations of the O1 and O5b CSFs (1990-1991) ex post evaluations of the O1 CSFs (1994-1995) — ex post evaluations of the O2 CSFs (1996-1997) — thematic evaluations (SMEs, telecommunications); (b) for the 1994-199 programming period: — prior appraisals of the O1 regional development plans — prior appraisals of the O5b programmes — prior appraisals of the 1994-1996 O2 SDPs — prior appraisals of the 1997-1998 O2 SDPs — critical analyses of the O1 SDPs and CSFs (1995). The Commission also requested evaluations by Member State of the impact of European structural policies on the economic and social cohesion of the Union. (6) DG XXII’s memo of 14 March 1991. C 347/6 EN Official Journal of the European Communities 16. 11. 98 management tool and thus contributing towards a very idea of evaluation. The broad awareness of Commission the phased system implementation of ongoing assessment in the Member States (7). the set up a for to be used 4. 6. However, this system did not specify the actual procedures (the definition of project specifications, the criteria for the selection of assessors, the structure of evaluation reports, data collection, quality standards, timetables, etc. ). 4. 7. At the same time, the Commission organised a series of events accompanied by a number of publications (8). By bringing the SF managers together with experts in the field of evaluation, these events contributed to the development of the idea of evaluation and helped raise the managers’ awareness of this subject. Evaluation was presented primarily as a process to be learnt by experience on the job. 4. 8. In May 1998, in the context of the preliminary draft budget of the European Communities for the financial year 1999, the Commission presented its ‘evaluation report 1996/1997’. In this working paper, it describes in evaluations already 1996-1997, gives a the evaluation activities carried out list of (7) (a) The drawing up, in 1994, of the first ‘terms of reference’ for the on-going evaluation of the CSFs and the Objective 1 forms of intervention. These confirmed the role of ongoing assessment as an aid to decision making for the Monitoring Committees and its implementation within the partnership the procedures for its implementation, in particular the composition and role of the evaluation team; system. They also specified (b) the 1995 distribution of the ‘Common guidelines for monitoring and ongoing assessment’ for the Community SFs, which lay down the factors for analysis (overall and specific objectives) and the link between monitoring and ongoing assessment; (c) the sending of letters to the Member States (1995) reminding them that they were required to carry out ongoing assessment and again specifying the desired implementing procedures. (8) Such as: (a) the Brussels conference on the evaluation of European regional policies (16 November 1995) concerning mainly the results of the ex post evaluations of Objective 1 (1989-1993) and prior appraisals for 1994-1999; (b) the European conference on evaluation methods for SF interventions (Berlin 2 and 3 December 1996) mainly concerning the problems involved in assessing impact at thematic the micro-economic, macro-economic and levels; (c) the Seville conference of March 1998 aimed at promoting exchanges of experience in the field of evaluation under the partnership system. implemented and currently underway and supplies a summary of the results of the evaluations implemented in the 1996-1997 period. 4. 9. Expenditure on evaluation work may either be financed directly by the Commission, or co-financed with the Member States. Tables 1 and 2 show expenditure directly settled by DG XVI in the field of evaluation for the 1989-1993 and 1994-1999 programming periods respectively. 4. 10. During the 1989-1993 period, as Table 1 shows, around 10 Mio ECU was spent on evaluation. Around two thirds of this expenditure occurred in two fields which come directly under the Commission’s jurisdiction and for which it is directly responsible. These were, on the one hand, the Community initiatives under Article 11 of Regulation (EEC) No 4253/88, amounting to 3,1 Mio ECU, and, on the other hand, cross-disciplinary and thematic evaluations amounting to 3,2 Mio ECU, such as the SMEs, concerning those environment and human the amounts earmarked for the evaluation of Objectives Nos 1 and 2 of the Structural Funds were relatively small, given the fact that, during this period, partnership was not very well developed in the field of evaluation. As is shown in Table 2, the amount earmarked for evaluation for the 1994-1999 period amounted to 25,2 Mio ECU in April 1998. This increase over the previous period was due to the development of ongoing assessment and its methodology within the partnership system. telecommunications, resources. However, 4. 11. With regard to spending in this field by the Commission and the Member States, which has increased considerably since 1996, the Commission is not always fully informed, in particular where the evaluations are included under the ‘technical assistance’ measures of Operational Programmes, which may also involve other forms of expenditure. A specific allocation to cover the needs of evaluation could be provided right from the initial programming phase, which would make expenditure more transparent. Reference framework 4. 12. The MEANS programme (1995-1998) (2,8 Mio ECU) has played a specific role in the process of the development of evaluation (9). The procedure applied aims to encourage the development of a uniform and consistent methodology, via an inventory of the methods used in the Member States and the identification of valid (9) This programme is managed jointly by the Commission and an external consultant, whose involvement is financed from budget heading B2-1 8 2 0, for an annual amount of 932 000 ECU. 16. 11. 98 EN Official Journal of the European Communities C 347/7 examples. This essentially empirical and exhortatory approach does not enable the Commission to decide on the actual procedures to be used for the implementation of evaluation methods or dictate a choice of indicators, in particular those for measuring the socioeconomic impact in the field of regional development. In view of the complexity of the methodological aspects of evaluation, there is no single method applicable to every specific the situation. supply to Commission methodological enable appropriate guidelines to be drawn up for the implementation of evaluations suited to specific situations. important in a position is nevertheless frameworks should be that to It in particular by 4. 13. So far, the results of the MEANS programme have been used to lay the foundations of a new evaluation procedure, introducing a common language, and have enabled the preparation of ongoing assessments within the partnership system. Tables 3 and 4 show the generally accepted evaluation procedure. However, the MEANS programme has not yet provided detailed answers to the questions regarding the practical and operational aspects of evaluation. Thus, two fundamental questions remain unanswered: firstly, what is the most appropriate system of indicators for use in formulating an overall assessment of a given Community structural programme; and, secondly, what evaluation approach the suitable socio-economic impact of a given structural programme in the context of regional development? The Commission is not yet able to draw up precise instructions regarding the actual techniques to be applied in each specific evaluation situation. The absence of such instructions is felt particularly during the ongoing assessments of intervention for the 1994-1999 programming period. for measuring is most Statistical bases and information concerning objectives 4. 14. Evaluations (whether prior appraisals, ongoing assessments or ex post evaluations) are often confronted with objectives that have been formulated too generally, even with regard to the objectives of the measures in question. 4. 15. The level of development of existing statistical bases is an impediment to the monitoring and evaluation of programmes and measures co-financed by the Structural Funds. Shortcomings are evident at various points, including, in particular, the lack of statistical coverage limited to the areas concerned, the absence of statistics regarding certain aspects and the collection of information at intervals that are inappropriate in relation to the needs of monitoring and evaluation. Indicators the 4. 16. The existence of reliable and appropriate systems of indicators is an essential element in the planning process. For structural evaluation purposes, programmes must contain information on financial and physical achievement as well as on results and performance. These indicators should provide managers with information on the immediate or long-term effects of action taken and also provide a measurement of the progress achieved in relation to the overall objectives of the programmes (e. g. regional growth or the reduction of unemployment). 4. 17. The reliability of the financial indicators is closely linked to that of the expenditure certificates sent to the Commission prior to the relevant Community payments. Physical indicators are often considered to be a source of supplementary information and their reliability is not consistent. Existing databases do not always provide a record of actual physical progress and final recipients do not always provide managers with periodic information about the physical state of operations. The ongoing assessment of the Lisboa e Vale de Tejo OP in Portugal reported physical in strict proportion to expenditure without mentioning any variations linked to delays in implementation or other causes. The Member States should improve their monitoring systems and the databases on which their indicators are based. indicators the horizontal 4. 18. Both the measurement of results and impact at sub-programme and measure level and the measurement of impact at programme level are hampered by the fact that neither the causal relationships between the various levels of objectives and between objectives and measures/projects nor effects of complementarity and synergy between measures and between objectives can be identified with any certainty. It is therefore difficult to define the specific effect of a given measure on regional development or the impact of investment on demand or, a fortiori, on supply. In this context, with regard to the 1994-1999 CSF, the search for indicators that will serve as an appropriate tool for evaluating results and impact has still not been completed (see paragraph 6. 23). 4. 19. The prior appraisal of the CSFs is essential to subsequent evaluations. It ensures that all the result and impact indicators have been specified and quantified, thus enabling a judgement to be made on the estimated impact of the programme concerned and comparisons to be drawn with actual effects. the management by objectives approach has not always been applied in a uniform or satisfactory fashion, which has this context, In C 347/8 EN Official Journal of the European Communities 16. 11. 98 indicators the other hand, had an impact in terms of the indicators necessary for evaluation. Thus, the Italian 1994-1999 CSF contains a small number of indicators, a fact which could hamper subsequent evaluation. On the 1994-1995 Portuguese CSF, which was approved by the Commission without any major changes, includes too many (about 300), which might also complicate evaluation. This variation in the establishment of indicators from one CSF to another highlights the lack of consistency in programming. Where the system of indicators approved by the Commission during prior appraisal and unquantifiable objectives, they are sometimes revised for purposes of ongoing assessment because they are found to be badly defined and insufficiently functional. a number of contains general and structural interventions locally. Although partial the 4. 20. The socioeconomic situations upon which they are brought to bear give rise to a wide variety of indicators that are often developed lists of indicators have been sent to certain regional or national authorities and have already been the subject of debate, the Commission has not drawn up any lists of the indicators that it considers to be both essential and representative in terms of regional development and in connection with the various types of measure. 4. 21. The quantification of result and impact indicators in connection with further should be developed monitoring, and, at the same time, this quantification should also be reliable. The fact is that it has been known to be unreliable, thus failing to prevent cases where new jobs were counted twice or to allow a distinction to be made between the gross and net effects on employment. This was discovered by comparing the ongoing assessment report of the Brandenburg OP with the statistics concerning the people registered at the Social Security office. Taking evaluation into account 4. 22. The contribution of evaluations in terms of a more efficient utilisation of Community funds largely depends on the quality of their content, and on when the results are received and decisions are made regarding changes in strategy. However, a number of ongoing assessments for the CSFs, SPDs and OPs for the 1994-99 programming period were finalised rather late, thus delaying possible changes. 4. 23. With regard to Objective No 2, the succession of shorter programming periods (as compared with Objective No 1) means that it is particularly unlikely that the dates when programming decisions need to be taken will coincide with those when evaluation results become available, thus making it impossible to learn from the past and adapt interventions. The the subsequent Objective No 2 programming and evaluation cycles are, to a certain extent, out of step. For example, the final version of the summary report on the ex post evaluation of Objective No 2 for the 1989-1993 period was submitted to the Commission in the summer of 1997, while the prior appraisal for the 1997-1999 programming period was underway and the ex post evaluation for the 1994-1996 period had not started. for structural appropriations 4. 24. Budgetary programmes and subsequent programmes are rarely influenced by the programme results shown up by the evaluation reports. During the first programming period the decision-taking system very rarely took any account in programmes were of evaluations, and changes generally based primarily on the degree of financial absorption. This was the useful the recommendations made by the head consultant in the context of the ex post evaluation of the 1989-1993 Objective No 1 CSF (see paragraph 5. 7). case for identified 4. 25. The more qualitative aspects of performance or the ongoing programme effectiveness assessments or implementation reports did not have sufficient impact on the operational management of the SFs or on the development of the sense of responsibility of the partners affected by the development of the programmes. in 4. 26. Just as for monitoring, the results of the ongoing assessments of the 1994-1996 Objective No 2 SPDs, CSFs and OPs reveal a divergence between the financing plan and the implementation of the programmes. Because of this situation, the need to utilise appropriations becomes the main concern and as a result the importance of the role of evaluation in the analysis of the area in question may be under-estimated, especially as regards consistency between resources allocated and the impact of structural interventions. the objectives, the are often evaluations’ and 4. 27. The recommendations the intervention managers, although they may be useful for other parties involved. This is the case for both ex post evaluations (see paragraph 5. 8) and ongoing assessments under the partnership system. conclusions self-evident to The objectivity of evaluations 4. 28. The lack of a precise frame of reference is a hindrance to the application of suitable techniques that can be objectively applied in specific evaluation situations so as to produce easily verifiable results and guarantee the independence of the assessors. On occasion, however, the recommended method is not objectively justified, or 16. 11. 98 EN Official Journal of the European Communities C 347/9 assessors are all too willing to adapt their approaches to the difficulties encountered or select indicators on the basis of the information available. information 4. 29. Furthermore, independence from managers does not always appear to be guaranteed, particularly in the field of (see paragraph 6. 11), while evaluation reports do not always give the source of the information on which their observations are based. This failing may raise doubts as to the reliability and validity of the information submitted in the evaluation reports. information 4. 30. Objectivity should be guaranteed by a verification system based on the quality of the indicators (of implementation, results and impact) used, the reliability the degree of the basic of implementation of the Commission departments analyse the information and results submitted under the evaluation procedure, but do not carry out any on-the-spot checks or verification as to how well the evaluation systems used in the Member States actually work. the methodology. However, and The evaluation contracts awarded by the Commission 4. 31. The audit of the Commission’s procedures for awarding evaluation contracts showed up a number of shortcomings — unsatisfactory selection procedures, excessively vague specifications, delays and low quality reports. 4. 32. The minutes of the meetings concerning calls for tender do not always specify the judgemental and analytical factors that determined the Commission’s choice of consultants. The Court’s audit did not provide evidence to show that the best experts from the most appropriate fields of competence were selected. 4. 33. In the case of the ex post evaluation of Objective No 2 and all the 1989-1993 CSFs for Objective No 1, the selection procedure was used to appoint a chief consultant accompanied by other national consultants and coordinating consultants, who then subcontracted the evaluation work to other consultants. In this chain of consultants, the Commission may lose its control of the evaluation procedure, which is supposed to guarantee the consistency, uniformity and quality of the results. 4. 34. Specifications are sometimes vague in terms of the description of the work to be carried out and the obligation to achieve results. The wording of the contracts is very general. Often, procedures for carrying out the evaluation work are left open pending a further agreement with the Commission on the methodological framework. 4. 35. There are often delays in respect of the schedule for submitting reports. Thus, in the case of the evaluation of all the 1989-1993 Objective No 1 CSFs, the contract provided for the report to be submitted on 28 February 1995, when, in actual fact, it was not completed until November 1995. The Commission provided for no penalties to comply with contractual obligations on deadlines and, on the contrary, made the final 40 % payment on 11 July 1995 under the terms of Article 4 of the contract. failure for 4. 36. The contracts do not contain clear, precise guidelines as to the methodology to be followed and the the Commission In consequence, results expected. departments may react to the reports in different ways. For example, some of the reports had been approved, even though they were no more than simple descriptions or vague analyses of the usefulness of the work carried out, were based on general statements by the persons questioned, or produced over-general recommendations. In the case of the ex post evaluation of the 1989-1993 Ireland CSF, one department recommended terminating the consultants’ contract because of the bad quality of their report, whereas another department approved it. In the case of the ex post evaluation of the 1989-1993 Portuguese CSF, the departments concerned agreed that, rather than providing an assessment, the report did no physical of aspects the reiterate more implementation already presented by the national authorities. Similarly, it was pointed out that the ex post evaluation report for the 1989-1993 Spanish CSF was too descriptive and that it was not sufficiently analytical. The ex post evaluation of the 1989-1993 Northern Ireland CSF was judged to be incomplete. The report on the ex post evaluation of the 1989-1993 Italian CSF was considered to be badly structured and organised. Although the evaluation report for the 1989-1993 Greek CSF has chapter by chapter conclusions, there is no general conclusion. than 5. THE EX POST EVALUATION OF OBJECTIVE Nos 1 AND 2 CSFs (1989-1993) Introduction 5. 1. Assessors are often faced with the following difficulties: (a) initial programming that has often been carried out the hastily and without a prior appraisal of programmes, intrinsic in particular as economic fundamentals, especially with regard to Objective No 1 regions. regards (b) the absence of quantified objectives and result indicators; (c) gaps in the the statistics with socio-economic situation both before and after Objective No 2 SF intervention, in particular with regard to the industrial fabric, investments and gross regard to C 347/10 EN Official Journal of the European Communities 16. 11. 98 regional product. The statistical tools used do not always correspond to the area selected for analysis (boroughs, districts, employment catchment areas); (d) the fact that certain measures have not been completed; (e) the existence of exogenous factors that help explain the development of the socio-economic situation; (f) where the allocation of aid is not linked closely to regional or local needs, the heterogeneous nature of the projects and their objectives; (g) it is hard to identify the intrinsic effects of the measures and programmes, because of the barrage of measures aimed at the same region or area; (h) the difficulty of measuring the indirect effects of a programme (in particular as regards employment), because no system of observation was set up for this purpose when the programmes were launched and no detailed inquiry has been carried out on this subject since their completion. The ex post evaluation of the Objective No 1 CSFs (1989-1993) 5. 2. The ex post evaluation of the Objective No 1 CSFs was coordinated by a central consultant and comprises the ex post evaluations of the CSFs concerned that were carried out by national consultants between January 1994 and June 1995. The consultant responsible for coordination developed the methodology that was to be applied, which made use of analytical methods, at both macroeconomic and measure/project levels. The approach followed aimed, firstly, to analyse the strategy and intrinsic logic of the CSFs, OPs and measures, secondly, to provide a preliminary evaluation of physical progress and impact, and, finally, to examine the implementation systems. This procedure is summarised in Annex I. consistent, 5. 3. Although generally the proposed methodology does have shortcomings. Thus, it provides that analysis of the intrinsic logic and strategy of the CSFs, OPs and measures should be based on a uniform structure of objectives, linked to a standard classification of activities. This approach might allow comparison between Member States, but it does not necessarily allow objectives to be listed in order of importance on the basis of both the Member States’ priorities and the criteria of hierarchy, complementarity, substitutability and synergy. Nor does it allow the identification of all inconsistencies in the strategies specific to each of the CSFs and OPs. The methodology is supposed to provide answers to relevant questions about intentional and unintentional effects, but, in many cases, the physical and impact indicators given in the methodological paper are unable to provide them. 5. 4. The Court’s audit also shortcomings: found the following (a) as regards the development of the socioeconomic environment of the CSFs and OPs, a comparison was not always made between the initial situation upon intervention was to act which the Community (1988-89) and the situation at the end of it (1993-94). Data were sometimes out of date, particularly those concerning regional differences; (b) there was often insufficient analysis for the ranking of objectives, which is required by the methodological guidelines. Either it was not done at all, or it was only done partially —either just for those OPs and measures that were selected for specific examination or without systematically applying the structuring criteria. As a result, the ranking of objectives and instruments sometimes did not produce any improvement in the identification of problems of consistency and the relevance and relative importance of priorities; (c) adjustments are sometimes made to CSFs and OPs without a thorough analysis of the causes; (d) financial implementation, implementation systems and physical implementation are generally well dealt with in the evaluation reports. However, in certain cases, either the effects of programmes or measures were not examined except in terms of job creation, or the results were formulated in non-quantitative terms. is Sometimes the only quantification of effects provided by applying macroeconomic models and is concerned with increases in GDP, job creation and competitiveness. This shows the inadequacy of the data provided by managers in the context of monitoring systems; (e) bottom-up analyses did not allow conclusions to be drawn in terms of impact at programme level; generally (f) Community added value is sometimes only expressed in qualitative terms (for example, the taking into account of the programme logic, the implementation of an integrated strategy or the effect on the organisation process). Furthermore, it was not always investigated systematically; 16. 11. 98 EN Official Journal of the European Communities C 347/11 (g) anticipated long-term effects were not discussed or were discussed only in qualitative terms. 5. 8. The ex post evaluation of all the 1989-93 Objective No 1 CSFs carried out by the national assessors mostly concentrated on the improvement of the Member States’ monitoring and management systems rather than on a real exercise in the assessment of the impact of the SFs. 5. 5. The Spanish Objective No 1 CSF was the subject of both an ex post evaluation ordered by the Commission and another one organised by the national authorities and co-financed by the Communities. Both studies were drawn up during the same period and pursued the same objectives but were conducted independently and separately. 5. 6. With regard to results and impact, the ex post evaluations highlight the physical achievements of the Objective No 1 CSFs, the macroeconomic impact of the CSFs observed in certain national evaluations, and the progress made in the field of planning —although in this field, the programme-based approach encountered certain obstacles (10). They also stress the CSFs’ contributions in terms of innovative organisational structures, financial management, monitoring, systems, coordination and partnership, although with substantial progress still being needed at each of these levels. information 5. 7. The chief consultant’s 1995 summary included some useful recommendations that the Commission could not follow up completely. One of these was that, when programmes were being drawn up, a clearer distinction should be made between Community structural interventions designed to encourage endogenous growth factors and those aimed at the restoration of a regional equilibrium or a raising of living standards. Another concerned striking a balance between the priority given to investments and specific regional characteristics that needed development or finding an equilibrium between the fight against unemployment and the need to increase productivity and competitiveness. Finally, it recommended that regional policies should pay more attention to employment. infrastructure (10) The programme by programme approach was hampered by the following: i) the short planning deadlines implying, in particular, the absence of any prior appraisal; the lack of tangible objectives, quantified indicators for implementation, results and impact; ii) iii) administrative constraints, in particular where the sub-programmes of a given programme fall under the responsibility of different authorities, or the sectoral and regional approaches overlap; iv) weaknesses in the Community rules, resulting in particular, in projects rejected for the Objective 1 OPS being accepted in the context of the Community initiatives; the project by project approach of the Cohesion Fund, which also did not contribute towards a harmonious application of the programme by programme approach. v) Ex-post evaluations of Objective No 2 (1989-1993) 5. 9. The ex post evaluations of the Objective No 2 programmes for the 1989-1993 period were coordinated by an external assessor in the 60 eligible regions. A general analysis was made for each of the regions, while detailed studies were made of 22 regions. The main aim of these evaluations was to analyse the impact of these structural interventions on the industrial redeployment process in these regions and draw lessons for the next programming phase. A summary of this evaluation approach appears in Annex II. 5. 10. Although a common framework was envisaged, the approaches adopted are very heterogeneous. This concerns particularly the priorities adopted and the depth of the analysis, whether it is descriptive or critical, and the methods and techniques used. 5. 11. Analysing strategies and the appropriateness of the CSFs and OPs and the changes that they have undergone in relation to the socio-economic environment seldom leads to a reformulation of the structure of the objectives, sub-programmes and measures in accordance with the logic of regional development or redeployment and on the basis of the criteria of ranking, complementarity and substitutability. Whereas many strategic analyses based on the structure of the CSFs and OPs are partial and uncritical with regard to the strategy concerned, a reformulation of this kind should, after due elaboration, make it possible systematically to identify the strategy’s shortcomings in terms of consistency, the weighting of priorities, synergy, complementarity and effectiveness vis-à-vis the socio-economic environment. 5. 12. As for the impact of the SF interventions, the assessors are not able, on the basis of a sample of projects and an examination of their results, added value and additionality, to aggregate their observations in terms of individual measures and the socio-economic impact of the programme concerned. 5. 13. Few impact evaluations were based on a genuine combination of top-down and bottom-up approaches. in particular, used for the This combination was, in evaluation of the Objective No 2 interventions C 347/12 EN Official Journal of the European Communities 16. 11. 98 services intervention (infrastructures, Tuscany. In this case, the top-down approach used a bi-regional (Tuscany/rest of Italy) econometric model to estimate the overall impact of an increase in demand on production by sector, consumption levels and added value, both in Tuscany and the rest of Italy. This model also makes it possible to estimate the multiplier effects by type of for businesses, aid for private investment) and the effects on employment (11). The direct effects on employment were estimated using a technique of standard coefficients concerning the various types of intervention. This top-down approach therefore regarded direct short-term effects related to demand. The bottom-up approach, which is based on a sample of projects and measures, is more capable of reflecting long-term, more formative and indirect effects, relating especially to supply and external savings that enable reductions in production costs. Even though these effects are more difficult to isolate and this approach does not enable an quantify and aggregation of the effects in terms of individual measures and programmes (which, however, was the objective of detailed evaluation), qualitative considerations as to the effectiveness of the degree of the measures and additionality of Community interventions constitute a useful complement top-down approach. A the combination of both approaches can provide real added value in terms of analysis, in particular if it is based on the techniques of modelling or formalisation. to generally provided for in the evaluation procedures only seldom made it possible to obtain precise information on the physical implementation, results and impact of the operations. 5. 15. In the field of job creation, the results are not clear in terms of the durability of the jobs and the structural specific measures. the evaluation reports is sometimes contradictory. to the data produced contribution ascribable In addition, the in 5. 16. With regard to the impact of the Objective No 2 programmes, we can draw the following conclusions from the ex post evaluations: in the improvement (a) although the Objective No 2 OPs do not often bring socio-economic about an situation, they at least contribute towards avoiding further deterioration (in particular as regards the maintenance of employment (12)) and preparing the ground for industrial redeployment. Indeed, most of the regions were seeking to diversify their economies and their industrial bases and, where possible, cast off their reputations as exclusively industrial areas; (b) despite the integrated procedure, based on the programme concept, the OPs sometimes consist of a modification of national and regional policies that have already been set up and do not always constitute a programming tool; 5. 14. Impact evaluations that make use of systematic inquiries (such as a number of investigations concerning the additionality of measures or the application of a specific method for estimating net measure-related job creation by comparing an experimental group with a control group) are also rare. However, they do provide real added value in terms of impact evaluation. The effectiveness of measures and programmes would have been more accurately evaluated if assessors had actually conducted the statistical analyses of certain result indicators that they had sometimes said that they wished to carry out. Thus, it is not always possible to find data on the number of companies located in new industrial zones set up with the help of SF aid, or on technological transfers from research establishments to businesses. The meetings with measure and project managers that were (c) the simultaneous existence of the Objective No 2 OPs and the Community initiatives gave rise to confusion and to the overlapping of aid; (d) SF intervention had positive effects in terms of programming, implementation methods, partnership, synergy, coordination and evaluation. However, the lack of synergy and communication between ERDF and ESF managers and measures was often stressed; (e) as regards additionality, the SFs make it possible to tackle new target areas, carry out projects which would have remained unrealised without Community aid, speed up implementation, carry out more ambitious projects or give a formative character to initial projects. The additionality of different types of measure (aid for infrastructures, business services, aid for private investment) may vary considerably; (11) This does not, however, mean that these are job creations, as the econometric model is based on the hypothesis of full employment and companies with production overcapacity during down-turns. (12) The 8th annual report on the SFs mentions important leverage effects which refer to overall results, such as the creation of jobs or companies. 16. 11. 98 EN Official Journal of the European Communities C 347/13 (f) the SFs do not often act as a lever for the involvement of private money or, at least, the evaluations omit this aspect; measurement of results as regards employment and SMEs. (g) the evaluations stress that studies, preparatory negotiations, administrative procedures relating to the adoption and implementation of certain projects and the integration of new national policies all involve delays that are often underestimated. In addition, the Commission 6. 3. evaluations should be general and evaluations concerning one particular geographical area should be developed in a later phase. that thematic sector or suggested that techniques used 5. 17. With regard to management, the evaluations show the that certain programmes were inadequate, and that the internal changes made during the programming period mainly concerned the maximisation of the rate of absorption of appropriations rather than the revision of regional development priorities. for monitoring 6. 4. Certain Member States, in particular Spain (13), France (14), Ireland (15) and the Netherlands (16), also drew up frameworks for ongoing assessment. They all look at the feel that ongoing assessments should following: (a) the development of the socioeconomic context; (b) strategies and internal and external consistency; 6. ONGOING ASSESSMENTS (1994-1999) The development and integration of the principles of ongoing assessment in the 6. 1. Ongoing assessments are provided for chapter on implementing provisions, dealing with the improvement of monitoring (included in each of the CSFs, SPDs and other forms of intervention). It is involve a critical stressed that ongoing assessments analysis of the data collected during monitoring, explain any differences in relation to the original objectives and assess the intended results of the measure in question. They also contain an opinion on the validity of the current intervention and the relevance of its objectives. the operating conditions of 6. 2. In the letters that it sent to the Member States during the financial year 1995, the Commission stressed that ongoing assessments were to provide the Monitoring Committees with appropriate information as to whether the programmes were running smoothly, put forward possible reprogramming measures and present an independent, objective picture of the progress made. In the initial phase, it was also proposed that they should concentrate on the interventions and, in particular, on the indicators, the management abilities of the persons in charge and the procedures used for selecting projects. Subsequently, with a view to proposing necessary changes and adapting programming, evaluation reports were to look at the relevance of the strategies employed, the validity of the intervention objectives, programmes in terms of the work carried out, the initial results in relation to objectives, the efficiency of the programmes, the correct application of Community policies (environment, public contracts, etc. ) and the effectiveness of the (c) the implementation including programming and selection criteria, administrative organisation, the monitoring system and project financing channels; conditions, (d) implementation and the results achieved, taking into account intended results; (e) initial effects obtained, together with an appraisal of expected effects; (f) costs, with a view to measuring efficiency, i. e. the relationship between expenditure and results. It was also that, where necessary, ongoing assessment, should also lead to proposals for changes in respect of priorities or intended measures. felt 6. 5. The partnership system was introduced in 1988 as a means for ensuring the effectiveness of the SFs, but it has only played its full role in the system of decentralised management since the start of the ongoing assessment of all the 1994-1999 CSFs in 1995. On this occasion, it was decided that the Monitoring Committees should be responsible for establishing the objectives of evaluations and the required quality standards needed to guarantee the overall consistency of the process, adapting specifications to the characteristics of the regions in question whilst respecting the objectives of the exercise and selecting assessors on the basis of shortlists prepared by the persons responsible for evaluation (members of the (13) Evaluación intermedia de programas operativos regionales del objetivo no 2 (fase 1994-1996). Orientaciones sobre contenidos comunes a recoger en los pliegos de condiciones te´cnicas, 23 May 1996. (14) DATAR, Regulatory and administrative timetable. (15) General criteria to underpin Terms of Reference for Operational of Mid-Term Programmes/Community Initiatives. Evaluation (16) Evaluatiekader voor de programma’s van de Europese Structuurfondsen, Terp Advies- CBEA Pieter Van Run, December 1997. C 347/14 EN Official Journal of the European Communities 16. 11. 98 Monitoring Committee and Community rapporteurs) after a public call for tenders. Ongoing assessments have made a real contribution to strengthening the partnership system, by allowing it to participate in the causal analysis of structural interventions. Tables 5 and 6 show the operational implementation of ongoing assessments and the significant work carried out by Member States under the partnership system in terms of identifying the impact of the SFs. know-how but also the impartiality of conclusions about the legitimacy of the strategy, the consistency of socio-economic analyses and the objective quantification of the intended effects. Evaluation requirements in the area of the Structural Funds have created in the space of a few years a network of external assessors known to the Commission and to the Member States, which is advantageous insofar as it generates greater expertise in this area but which focuses on a specific market. 6. 6. The partnership approach to ongoing assessment has also strongly influenced the institutional structures and the national/regional traditions of the public bodies responsible. It has made for more widespread knowledge of the idea of multiannual programming, the evaluation of which is essential to drawing up critical assessments of structural interventions and preparing any necessary adjustments. In the field of evaluation, partnership is particularly well developed in connection with Objective No 1. to all involved, 6. 7. At present, the aim of ongoing assessment work is to supply overall solutions from programme designers to managers and political leaders. This work results in very bulky reports containing information from the programmes themselves, prior appraisals, annual implementation reports and ongoing or ex post to information gleaned orally during the audits, only some of the results and recommendations are of any real use in terms of assisting decision-making, managing projects, reprogramming or preparing the next generation of interventions. evaluations. However, according Implementation procedures 6. 8. Of the 389 intermediate evaluations undertaken, 320 concerned Objectives Nos 1, 2, 5b and 6 (17). The implementation of to significant delays in several Member States. these evaluations is subject 6. 9. The intermediate evaluation process was originally envisaged as comprising two distinct stages. The first stage involved an assessment of the scope for evaluation and the identification of related problems. The second consisted of the evaluation itself and the subsequent drafting of a report. In practice, these arrangements were seldom applied. 6. 10. As a rule, the Commission insisted on intermediate evaluations being carried out by external experts so as to individual guarantee not only the credibility of (17) 36 of which were examined on the spot and/or at the Commission. In practice, 6. 11. the Court’s on-the-spot checks highlighted the limitations on the independence of the external consultant. Given the limited time available to the assessors and the consultant’s de facto dependency on programme managers for the necessary information, the separation of the respective duties is not very rigorous. Too often, the results obtained in terms of performance merely reproduce and update information from the monitoring system while presenting it as an external contribution. 6. 12. Furthermore, the usual procedure of an open invitation to tender based on publication in the Official Journal was not always applied with a view to selecting a consultant to carry out assessments. In the Netherlands, the procedure consisted of asking a limited number of consultants on a list of potential assessors to submit a bid. The selection was then made from these bids. 6. 13. Although the Commission regularly participates in monitoring committee meetings and is able to win acceptance for its criteria for selecting consultants, its involvement in this process is inconsistent. Thus, in Belgium, Greece and Portugal the Commission is actively involved in the procedure for selecting assessors within the framework of the technical groups or the ad hoc technical accompanying committee which prepare the choice of consultant and monitor the corresponding implementation and evaluation. By contrast, in France the Commission is officially informed only once the two bids deemed to be the best have been selected; these latter are accompanied by a final proposal in respect of which the Prefect of the Region, in his capacity as contracting authority, asks for any observations to be brought to his attention. As regards the Netherlands, the Commission, the national which accompanying ante, intermediate and ex post evaluations of the regions covered by Objectives Nos 1 and 2 and in the work of the monitoring committees, reacted only belatedly to the content of the draft intermediate evaluation report on the Flevoland Objective No 1 SPD. committee which pilots the work of takes part ex in 16. 11. 98 EN Official Journal of the European Communities C 347/15 Priority given to evaluation objectives with regard to synergies between ERDF and ESF activities. the regards implementation of ongoing 6. 14. As evaluations, the priority given to evaluation objectives varies, with the exception of the examination of the physical and financial results of the measures, which is an important aspect of any ongoing evaluation. Priority developments often correspond to a genuine regional need, such as in Corsica, where the emphasis is placed mainly on dealing with problems of management, monitoring and links between evaluation and monitoring. Moreover, the pursuit of certain evaluation objectives is hampered by technical problems, such as the measurement of evaluation and impact indicators. the 6. 15. Very often, ongoing evaluations do not accord sufficient importance to critical appreciation of the regional development strategy chosen, to the viability of the projects financed by the Structural Funds, to the criteria for selecting projects, and to the effects and impact of specific and general objectives. the programmes’ 6. 16. Community added value, the analysis of which was one of the objectives of the ex post evaluations of the Objective No 1 and Objective No 2 CSFs for the 1989-1993 programming period, is covered only in exceptional cases by ongoing evaluations. 6. 17. Changes in the socio-economic context, where Community assistance takes effect and in respect of which evaluation contracts generally provide for monitoring and updating, are dealt with systematically in certain cases (e. g. by means of very detailed progress charts in Wallonia) and in an unsatisfactory manner in other cases, which has a direct influence on the quality and updating of the analyses of strengths and weaknesses which are generally carried out as part of ongoing evaluations. (the objectives of 6. 18. The extent and maintenance over time of the internal and external coherence of all of a programme’s components the programme, sub-programmes and measures, and the degree of complementarity and synergy between objectives and measures) are examined only rarely in a systematic manner. Thus, in the case of the Brandenburg OP, the financial commitments in respect of priorities 1. 1 (productive investments) and 1. 2 (infrastructure) account over for 138 % and 242 % respectively of the initial estimate, and the assessor provided no justification for the regional development strategy that was implemented. Synergies are sometimes subject to analyses, which are often incomplete. This aspect warrants more systematic treatment, in particular the 1994-96 period of the reorientation a 6. 19. Similarly, where programme’s priorities and of the allocation of resources is an explicit objective of the evaluation, the evaluation does not always produce a consistent overall presentation the proposed amendments or an explanatory of the the memorandum. Moreover, extensions or curtailments proposed in respect of the measures are not always specified. implications of 6. 20. Many of the evaluations examined showed that the assessor had not been able to achieve all of the objectives envisaged in the contract, in particular as regards impact. Moreover, the time and resources available were not always proportionate to the objectives. the evaluation of the measure’s Approaches, methods and techniques in general 6. 21. As contracts often formulate evaluation objectives and approaches the choice of approaches, methods and techniques is often left to assessors, who during implementation to the difficulties they have encountered. Thus, the assessors occasionally make indicators out of the only information at their disposal. sometimes terms, adapt them consensus to evaluating 6. 22. Techniques and methods have not always been developed to meet evaluation objectives. Moreover, no the appropriate exists as genuine operational methodology regional for development; the Commission merely circulates ‘best practices’ in general terms without actually coordinating or harmonising their implementation in a concrete way. This imprecision, together with disparities between the Member States as regards programming, implementation and evaluation, results in evaluation reports of variable quality and makes it very difficult to compare results at a European level. 6. 23. Measuring the impact of measures thus reveals shortcomings in the techniques applied, in particular as regards the characteristics and complexity of the effects to be evaluated: (a) a detailed search for even more relevant indicators has not always been carried out, and the envisaged indicators sometimes serve only to measure physical and financial results; (b) impact indicators have not always been given substance; (c) evaluation of the socio-economic or regional development impact appears to be hampered by the increasing number of measures, sub-programmes and measures (Objective No 1), whilst many programmes C 347/16 EN Official Journal of the European Communities 16. 11. 98 implemented for assisting the conversion and/or revitalisation of industry (OPs, Community initiatives and national programmes) may jointly. be Furthermore, Community and national measures sometimes differ in their application, as do the dates on which measures are launched. Lastly, objectives are often unquantified and are formulated too generally (18). In this context, the links between the diagnosis of the socio-economic problems to be resolved and the objectives of the series of measures, sub-programmes and projects implemented, as well as the links between the objectives of this series, do not always appear obvious, especially when one remembers that the physical and impact indicators attributed to the various levels of a programme do not wholly reflect the programme’s evaluation requirements. Similarly, it is difficult to identify the results of each programme and to separate them from the results of other private, regional or national measures; or on (Portugal) (d) models for estimating the expected effects and the impact of measures on the basis of mid-term or anticipated results are rarely used. The ongoing evaluation of certain national CSFs was based on an a input-output model macro-econometric model (Spain, Ireland), whilst the use of regional models continued to be very marginal. The example of the development model for the province of Hainault (Belgium) used in the context of intermediate evaluations is worthy of mention in this connection. The establishment of models describing the effects of the measures is not very widespread either, except in the field of transport infrastructure. Mention should be made of the application of the input-output model the Expo 1998 project (Portugal) as part of prior and ongoing evaluations, although the estimates should be considered with caution; to (e) estimates of the effects of measures on employment are also still precarious. The application of models and other analysis and enquiry techniques does not supply adequate information concerning, on the one hand, the net number of jobs created (taking into consideration, in particular the number of jobs that would have been created without the measure or the effects of job substitutions), and, on the other hand, the degree of permanence of the jobs created. Moreover, the application of such models and techniques usually provided no details about new jobs which could not be attributed to the implementation of the programme. (18) E. g. the revitalisation of the countryside, the strengthening of structural the fabric of modifications related to regional development. region and economic the 6. 24. Some evaluation contracts provide, as part of the bottom-up approach, for enquiries in respect of local economic development partners and final beneficiaries, including companies which receive Community aid, with a view to making a quantitative or qualitative assessment of the contribution made by the Community programme. The enquiries’ findings are often unrepresentative in the light of the number of participating companies and partners, and are of limited reliability insofar as they are not always subject to scrutiny. 6. 25. As regards result and impact indicators, it was observed that: (a) the time taken to obtain data and statistics is often long; (b) the existing data cover geographical areas which do not correspond to the areas in question; (c) data on companies are not always recorded by the establishments concerned, a fact which distorts local statistics if a single establishment is present in the area but the company itself has premises all over the country; (d) many statistical bodies are unable to subdivide overall data with sufficient precision to target an area or a given series of problems. 6. 26. Where the assessor provides more adequate indicators with a view to measuring impact, whether to compensate for a shortcoming or to replace those originally envisaged, he does not substantiate them and leaves the situation as to their operational nature uncertain. Furthermore, he does not always mention the enquiries still to be carried out which will give concrete form to the indicators, nor does he estimate their cost. It should also be stressed that it is difficult to quantify indicators concerning the programme’s aims in the absence of information on historical series. Implementation of Objective No 2 programmes (1994-1996) 6. 27. The results not only of the ongoing evaluation of SPDs, CSFs and OPs during the 1994-1996 period but also of the monitoring procedure highlight the imbalance between the financing originally envisaged and the funds actually used to implement the programmes. Despite numerous adaptations of the programmes during the last programming phase (1994-1996), in order to absorb the financial aid, appropriations were transferred from the 1994-1996 programming period to the 1997-1999 one at an average rate of 12 %. The risk of this situation is that it may lead to an overriding emphasis being placed on the 16. 11. 98 EN Official Journal of the European Communities C 347/17 need to use up appropriations and an underestimation of the role of evaluation, in particular as regards consistency between objectives, resources allocated and the impact of structural measures. 7. IMPACT OF THE STRUCTURAL FUNDS AND EVALUATION MODELS Impact of CSFs and programmes Results of the models as regards the impact of the Structural Funds 7. 1. Given their specific characteristics, the models used for macroeconomic evaluation in the field of the Structural Funds (see Annex III) make differentiated assessments of the impact of the Structural Funds as regards: (a) the implementation of the Structural Funds’ objectives in terms of GDP growth, actual convergence of per capita GDP, employment and unemployment; (b) observance of the conditions for implementing the Structural Funds, such as additionality and durable development with particular reference to controlling public debt and the balance of payments, to income distribution and to respect for the environment; (c) the effects in terms of supply and demand; (d) the effectiveness of Structural Fund instruments, such as public investment, investment in human resources and aid for private companies. GDP growth 7. 2. As Table 7 shows, the results of the principal models used to evaluate the impact of the Structural Funds (HERMIN, OGM and DIO) indicate a positive increase in GDP for Cohesion Fund countries and consequently greater convergence. Thus, cumulative growth in the economies of the Cohesion Fund countries from 1994 to 1999 as a result of structural measures under the CSFs appears, in terms of real GDP, to be 5,5 % for Portugal, 5,9 % for Greece, 2,6 % for Spain and 3,3 % for Ireland. These results assume that the principle of additionality is fully observed; they may thus be modified where the principle is not observed (see paragraphs 8. 11 to 8. 13). Furthermore, the various macroeconomic models produced sufficiently consistent results for the impact of the 1994-1999 CSFs on real to be considered plausible. annual GDP growth Nevertheless, demand-related effects continue to be predominant in the short term, as the effects on supply become apparent only much later. Similarly, the HELM regional model used for the Hainault region evaluates the impact of the SPD in terms of annual growth in total added value as accounting for an increase of 1,7 % to 2,5 % over the 1994-2002 period. By contrast, in the case of the Italian Mezzogiorno, the impact of the Structural Funds on growth seems very limited, or even close to zero. The Commission’s QUEST model, by introducing hypotheses on the expectations of businesses and consumers, produces lower estimates of the effects on growth and employment than was the case with the previous models. The gains shown by this model with regard to the potential for growth and the reduction of unemployment only become apparent in the longer term. 7. 3. As regards the average acceleration in annual growth due to structural measures implemented as part of CSFs, Table 8 shows that 1 % of growth in GDP generated by the multiplier effects of the CSFs leads, over the 1994-1999 period, to additional annual growth of 0,2 % to 0,3 % for demand and of no more than 0,2 % for supply. Economically speaking, the predominance, as highlighted by the HERMIN model, of the effects on demand, which would disappear if Community aid were discontinued, contributes to achieving the objectives of Article 130A of the Treaty only if the effects on supply are lasting, thereby increasing the productive capacity of the regions concerned. Insofar as the effects on supply become apparent much later than those on demand, the corresponding multiplier effects may be underestimated. Real convergence in terms of per capita GDP 7. 4. The models used within the framework of the Structural Funds show that convergence is not always guaranteed and that some CSF programming may at best reduce divergence rather than develop convergence. 7. 5. In the case of Ireland, Bradley (HERMIN 1994) and Honohan (HERMIN 1997) envisage the possible effect of aid under the 1989-1993 and 1994-1999 CSFs on per capita GDP. Bradley considers that the 1994-1999 CSF’s contribution to bringing the Irish economy closer to the EU average is modest whereas, in Honohan’s view, the CSFs play a significant role in the convergence process. Beutel (1997) predicts that Ireland will catch up with the rest of Europe even without the 1994-1999 CSF, which would increase the annual growth rate by 0,6 %. Similarly, the annual growth rates of Spain, Portugal and Greece would increase for the 1994-1999 period by C 347/18 EN Official Journal of the European Communities 16. 11. 98 0,4 %, 1,1 % and 1 % respectively due to the Structural Funds, which, in any case, has had a positive effect on convergence in terms of per capita GDP. 7. 6. Forecasting simulations based on the OGM model of convergence in real per capita GDP in relation to the EU average provide estimates that stress the fact that, even if the trends projected in 1995 are too pessimistic in the case of Ireland (see Table 9), the CSFs’ net contribution to convergence remains modest for the Cohesion Fund countries. 7. 7. In the case of Hainault, the HELM model takes aspects of real convergence into consideration for the purposes of ongoing evaluation and considers that the province’s estimated annual growth rate would be approximately 1,4 % to 1,6 % higher than the European average because of the Structural Funds; these figures would indeed make it possible for Hainault to catch up with other regions. 7. 8. These estimates highlight the fact that, given the differences between Objective No 1 Member States’ regional or sectoral productive structures, Structural Fund aid produces variable effects. They also confirm that Structural Fund programming must be improved in order for the corresponding assistance to improve the economy’s productive to convergence. capacity with a view Employment and unemployment 7. 9. Table 10 summarises the findings of evaluations of the impact of the Structural Funds on employment and unemployment. All of the studies conclude that the CSFs generate net jobs. These new jobs are often created in the non-traded services sector, whereas the productivity effects of the CSFs are mainly concentrated in the industrial sector. Lastly, some studies note a reduction in the number of industrial jobs due to an increase in productivity. Effectiveness of CSF instruments 7. 10. The HERMIN evaluations for Spain, Ireland and Portugal show that aid for public investment, especially in infrastructure, is more productive in terms of growth than aid for investment in human resources (except for Spain) and for private investment. This trend is illustrated in Table 11 which shows, for each instrument, the increase in GDP due to the 1 % increase in total GDP generated by the CSF over the 1994-1999 period. In the case of Greece, the HERMIN study also concludes that investment in ‘hard infrastructure’ covered by the CSFs contributes more to GDP growth than investment in human resources. The use of the OGM model (Pereira 1991) to evaluate the effects of replacing aid for private investment or for investment in human resources by additional aid for investment in infrastructure shows an improvement in GDP growth and in the convergence process as a result of the Structural Funds. Additionality and joint-financing requirements the public that, without financing. Thus, Honahan 7. 11. The aforementioned models also made it possible to evaluate the significance of additionality and of public (HERMIN, 1997) joint considers joint-financing requirement, the growth rate of the Irish economy would have been 1 % lower and employment would have experienced a smaller increase. By contrast, Honahan envisages a more appreciable reduction in public debt as a proportion of GDP and in the balance of payments surplus if the public joint financing requirement did not apply. In the case of Portugal, the joint financing requirement leads, according to the HERMIN evaluation, to a total additional gain of 1,3 % in terms of GDP, but the development is less positive in terms of public debt. 7. 12. The HELM evaluation for Hainault attaches great importance to additionality. According to the evaluation, the average rates of growth in value added for the 1994-1999 period based on the three envisaged scenarios (regional policy including the SPD, the continuation of regional policy without Community aid and regional policy from which the SPD has been deducted) are 2,5 %, 1,7 % and 0,1 % respectively, and the growth rate in terms of employment would fall from 0,8 % to 0,2 %. The also deteriorate situation would considerably. The HELM evaluation also concluded that, if the rate of subsidy varies, aid does not stimulate investment to the same extent and that considerable investment would take place if levels of aid were lower. industrial Durable development 7. 13. Table 10 shows that the effects of the Structural Funds on the balance of trade, the current-account balance or public debt are not always evaluated. Moreover, the various studies reach different conclusions. Thus, the economic expansion generated by the CSFs leads, in certain cases, to the improvement of public financing through taxation whereas, according to other models, the CSFs create budget deficits. 16. 11. 98 EN Official Journal of the European Communities C 347/19 Role of macroeconomic evaluation in the Structural Funds’ decision-making process the tend effects on 7. 14. Given the short-term effects of the Structural Funds on demand, ongoing evaluations to over-estimate the difficulties experienced by the political in using up Community funds and to authorities supply. longer-term underestimate Reorientation or reprogramming of the OPs or CSFs may also be biased. The fact is that measures or projects which do not use up Community appropriations quickly enough may be discontinued, especially in the fields of innovation or new technology where implementation is complex, in favour of existing structures which are much less the macroeconomic aspects of sustainable development are not always considered. productive. However, cases such in In this 7. 15. context, additional macroeconomic evaluations which take account of sustainability, growth or convergence are essential before any decision is taken regarding initial programming and any necessary changes. However, on several occasions in Spain, Greece and the new German Länder, measures were rescheduled on the basis of ongoing evaluation without serious overall macroeconomic analysis. With regard to Objective 1 regions in Germany, no macroeconomic analysis of the impact of the measures was performed when the intermediate evaluation was made. above evaluate the effectiveness of the CSFs’ principal instruments (public investment, private investment and investment in human resources) only in terms of their respective contributions to GDP growth. Expenditure on infrastructure seems to increase the yield of the private capital which it complements, and would appear to speed up convergence more than other forms of investment (see Table 11). the terms of 7. 19. Nevertheless, the choice and financing of measures or sub-programmes which make up the programme are not neutral impact of regional in development on lasting endogenous growth. The evaluations performed by the Commission or by the Member States in the context of their partnership as regards sub-programmes or measures are also insufficient to make it possible to examine the consequences of a specific measure on other areas or sectors or to compare similar objectives. it with other measures with Furthermore, analysis of the impact of the measures or sub-programmes must also take into consideration the initial attempts to offset growth against inter-regional equality which determined the choices made, so as to compare expected gains in efficiency with actual results. Impact of sub-programmes, measures and projects 7. 16. Very few evaluations are made of sub-programmes and measures, even though they act as the driving force of Structural Fund programming. The fragmentation, lack of definition and inconsistency of many sub-programmes and measures, and the inadequate quantification of their specific objectives, may help to explain the difficulty of measuring their impact. 7. 17. This shortcoming complicates any attempt to identify and measure synergies and the degree of complementarity between individual measures or between one programme and another. Furthermore, the links between a programme’s general objectives and the specific objectives of sub-programmes or measures are very often unclear. Thus, in several cases, problems of consistency in the Objective No 1 1994-1999 CSF programmes were noted in the ongoing evaluation reports. 7. 20. In principle, the projects’ relative costs and advantages are analysed with a view to approving financing. Pursuant to Article 5 of Regulation (EEC) No 2083/93 (ERDF), this analysis is compulsory in the case of major projects the total cost of which is more than 25 Mio ECU for investment in infrastructure and 15 Mio ECU for productive investment. These prior assessments of major projects are part of the examination of applications for Community aid and determine the amount of financing. However, ex post evaluation of them is not provided for in the SF Regulations and may not be required by the Commission for the purpose of establishing whether the socio-economic advantages the Community resources actually obtained reflect originally provided. This deficiency in the process for evaluating major projects renders the very concept of evaluation in this area ineffectual, as a prior assessment can be justified only if an ex post evaluation is actually made. Furthermore, these microeconomic and sectoral studies could also consider the potential impact of the Structural Funds on productivity and thus become an effective instrument for analysing this impact in the short or medium term. 7. 18. Furthermore, techniques are seldom fashioned to measure the impact of a sub-programme or specific measure. Thus, the macroeconomic models described 7. 21. Synergies and complementarity must also be taken into consideration when projects are evaluated. Current evaluations in this connection are still inadequate or C 347/20 EN Official Journal of the European Communities 16. 11. 98 superficial. The harmonisation of synergies and complementarity between projects is essential so as not to weaken their desired impact. 8. 4. The same is true of statistical bases, the weakness of which casts doubt on the validity of the indicators and has a major influence on the monitoring and evaluation process. evaluations often 7. 22. Although ongoing tackle problems relating to programme management, they do not often take account of the criteria which determined the selection of projects, such as their sustainability or their contribution to regional development. However, the programme management system must be able to guarantee that the criteria for selecting projects are applied in such a way as to ensure that the programme produces the best possible results. In particular, projects providing very little added value in terms of development should be avoided even if the fact that they consume appropriations very quickly is an advantage. The project selection system must be more transparent for the purposes of inspection and evaluation. 8. 5. The current plethora of indicators of varying relevance is detrimental to the quality of evaluation and prevents it from becoming operational. A small number of impact indicators which are essential for justifying or reorientating a structural policy should be carefully selected bearing in mind the constraints imposed by the regionalisation of data and the material possibilities for gathering information. The reliability of these indicators should be checked. The comparability of indicators and their aggregation on a European scale should be improved. 8. CONCLUSION 8. 6. In order for full account to be taken of evaluations in the decision-making process and in the operational management of the Structural Funds, steps should be taken to ensure that the evaluations are transparent and objective, to requirements and that the finalisation date is adequate. correspond objectives their that 8. 1. The evaluation of Community structural measures is a legal requirement stemming from Council Decisions and is currently extensively implemented in partnership with the Member States. Evaluation has become a system of aid for the design and management of intervention measures, and contributes to improving understanding of their effects and usefulness. For the 1994-1999 period, more than 250 mid-term evaluations were completed by Monitoring Committees (19) in 1997. 8. 7. In order for evaluation findings to be objective, assessors must remain independent of the managers and public authorities concerned. 8. 8. The use of a common frame of reference would make it easier to read and compare the evaluation reports and to share practical experience. 8. 2. With the revision of the reform of the Structural Funds, evaluation is entering a phase of maturity which should be consolidated without delay by improving the methodology. Much work still needs to be done to determine the most appropriate evaluation methods for measuring the socio-economic impact of the programmes within the framework of regional development, as the MEANS programme has been unable to provide a satisfactory solution. 8. 3. Particular attention should also be paid to indicators at all levels as regards their relevance, the scrutiny of information and the attendant costs. In addition, qualititative aspects should be formalised. Without further progress in these areas, the evaluation process may very well stagnate. (19) The Monitoring Committees include representatives of the national authorities concerned and of the Commission. indicate 8. 9. For each evaluation, more operational objectives should be defined. The terms of reference and the technical specifications, which are generally approved by the Commission, should clearly the most appropriate methodology for dealing with regional, sectoral or national aspects or for specifically covering short- or long-term effects. The strengths and weaknesses of any other methodology should be compared with the Commission’s methods. Similarly, the frame of reference for each evaluation should include a minimum list of indicators proposed by the Commission for measuring the performance of the programmes’ objectives so as to guarantee a degree of consistency which is essential for the purposes of analysis. Other socio-economic and to structural complement and refine this analysis. indicators may, of course, be used 8. 10. Given the diversity of approaches to evaluation, the Commission must require assessors to explain their approach much more explicitly when calls for tenders are being issued, in particular as regards the type of effect or 16. 11. 98 EN Official Journal of the European Communities C 347/21 impact which is supposed to be measured and the period concerned, with a view to assessing the exhaustiveness of the methods used. In addition, evaluations should contain a detailed analysis of the links between the programmes’ objectives and the priorities and measures implemented to achieve them, and of the corresponding inter-relationship between the objectives and results of the programmes, priorities and measures. 8. 15. Macroeconomic models make a fundamental distinction between the effects on demand in the shorter term and structural adjustments in terms of supply. The programming of Structural Fund assistance should place greater emphasis on improving the productive capacity of the economy so as to ensure that the effects of assistance are permanent and to avoid excessive dependency on demand, which would to a situation where lead Community aid would remain essential in the long term. 8. 11. In this context, the sub-division of evaluations between the beneficiaries concerned needs to be improved increase efficiency and to meet specific so as to expectations, in the area of ongoing evaluations. Similarly, greater control over costs should be sought, as the Commission is not always aware of the volume of expenditure allocated to evaluation within the partnership framework. in particular 8. 16. As the effects on supply become apparent only over a longer period, ongoing evaluations generally pay more attention the effects on demand. The Commission should aim to develop the instruments needed to monitor and evaluate effects on the structure of the economy. to indicate which 8. 12. The Commission should also particular aspects of regional policy are to be evaluated: additionality, convergence, partnership, concentration, employment, endogenous economic development or social welfare. The per capita GDP should not be the only criterion for judging the effectiveness of structural measures. 8. 17. Macroeconomic models also show that a policy of economic growth is not always compatible with durable development. The relationship between the impact on growth and the effects on durable development should be analysed in greater detail and systematic account should be taken of this duality in the programming of Structural Fund assistance. 8. 13. It should be possible to base estimates of the programme’s effects, impact and Community added value on national and regional development models and on models of the impact of the individual measures. 8. 18. The effects of the additionality and public joint-financing requirement are not easy to discern and are not always examined. More systematic and detailed analyses should be carried out in this area. 8. 14. The macroeconomic results of structural measures should be refined by taking account not only of the imperfections of the underlying models but also of the side-effects of the measures and their cumulative effect on social welfare. 8. 19. Evaluation per se does not profess to replace a political decision-making process which involves broader considerations, but aims to provide an essential tool for the sound and effective management of Community structural measures which the corresponding socio-economic results. takes account of This report was adopted by the Court of Auditors in Luxembourg at its meeting of 23 September 1998. For the Court of Auditors Bernhard FRIEDMANN President Cost of evaluation funded directly by the Commission (DG XVI) for the first CSF Table 1 Country Objective 1 1989-1993 Objective 2 Community initiatives Thematic Others Total Prior appraisal Ex post Prior appraisal Ex post Prior appraisal Ex post (in ECU) Belgium Denmark Germany Greece Spain France Ireland Italy Luxembourg Netherlands Portugal United Kingdom EC 35 000 31 000 42 000 35 000 35 000 35 000 53 000 150 172 207 372 97 662 109 572 150 552 193 022 72 572 17 500 35 000 17 500 70 000 35 000 35 000 17 500 35 000 122 500 69 040 25 520 138 570 117 090 253 185 96 580 15 880 47 925 131 500 14 815 65 963 10 514 143 480 526 347 739 274 197 228 201 915 395 955 15 128 46 816 452 706 223 428 98 400 35 395 25 000 593 350 260 977 104 371 464 064 390 156 25 000 625 169 626 694 77 500 151 845 30 000 151 845 151 845 35 000 550 256 187 898 96 034 299 550 1 456 714 1 455 713 881 291 810 551 1 255 088 48 508 154 741 1 340 897 1 229 694 740 971 Total 266 000 980 924 385 000 910 105 3 117 154 3 227 676 1 070 791 9 957 650 Source: the European Commission. C 3 4 7 2 2 / E N O f f i c i a l J o u r n a l o f t h e E u r o p e a n C o m m u n i t i e s 1 6. 1 1. 9 8 Cost of evaluation directly financed by the Commission (DG XVI) for the second CSF up until 1 April 1998 Table 2 Country 1994-1999 Objective 1 1994-1996 1997-1999 1994-1999 Objective 2 Objective 2 Cls Thematic Cohesion Studies Prior appraisal Ongoing Prior appraisal Ongoing Prior appraisal Prior appraisal (En ECU) Methodological development and others Total Belgium Denmark Germany Greece Spain France Ireland Italy Luxembourg Netherlands Austria Portugal Finland Sweden United Kingdom EC 24 000 352 527 38 940 108 303 68 995 86 841 56 000 83 809 19 000 17 671 60 000 71 774 49 150 1 030 037 4 171 941 1 382 117 481 092 1 067 557 1 514 311 37 499 11 827 1 631 604 339 214 281 295 70 885 26 530 23 312 94 784 85 102 186 122 105 287 9 143 46 431 33 000 16 750 27 000 154 591 127 300 611 348 9 278 128 450 209 078 807 462 722 690 61 741 104 924 171 201 326 614 247 636 10 600 9 600 29 200 19 000 45 600 31 169 10 600 10 600 30 000 1 559 39 702 23 286 29 725 36 091 4 326 33 185 460 4 406 245 35 247 35 845 4 314 127 986 2 576 126 19 417 33 357 62 217 55 167 48 367 56 967 52 317 49 417 14 597 46 817 38 417 52 122 90 000 4 030 438 1 045 981 75 547 1 551 316 4 358 697 1 842 384 1 700 175 1 180 200 2 539 868 96 541 269 677 233 944 1 765 268 682 578 555 931 415 217 6 877 328 Total 684 483 12 371 906 935 352 3 400 422 196 369 248 391 2 704 112 619 179 4 030 438 25 190 652 Source: the European Commission. 1 6 1 1 9 8. E N O f f i c i a l J o u r n a l o f t h e E u r o p e a n C o m m u n i t i e s C 3 4 7 / 2 3 C 347/24 EN Official Journal of the European Communities 16. 11. 98 Table 3 Control, Monitoring and Evaluation Means RR Achievements RR Impact Input Output Outcome rr Control RR rr Monitoring (of management) RR rr Evaluation (of effects) RR Source: MEANS Table 4 Judging the value of a programme Points of Reference Compliance with administrative texts Standards of sound management Public satisfaction Impact qq QQ QQ Society Needs What is at stake QQ QQ Results qq QQ QQ RR Means Resources RR Interventions’ achievements QQ Specific overall objectives Relevance Efficiency Effectiveness Usefulness Public action Evaluation Source: MEANS 16. 11. 98 EN Official Journal of the European Communities C 347/25 Table 5 Number of ongoing assessments by country and objective Country Objective 1 Objective 2 Objective 3 Objective 4 Objective 5a Objective 5b Objective 6 Total Belgium Denmark Germany Greece Spain France Ireland Italy Luxembourg Netherlands Austria Portugal Finland Sweden United Kingdom Total 1 8 28 27 6 11 24 1 1 25 12 144 4 2 9 9 27 11 1 5 4 1 5 13 91 5 1 13 12 1 1 1 1 1 4 1 1 5 1 1 2 1 1 1 1 1 4 1 42 19 1 1 1 1 1 1 1 1 8 3 1 8 7 20 13 1 5 7 2 5 11 83 19 6 40 28 58 55 11 50 4 13 15 25 13 14 38 389 1 1 2 At CSF level, five ongoing assessments were undertaken — Spain, Greece, Ireland, Italy and Portugal — and a summary of the evaluation work was drawn up for Objective 1 measures in Germany. Source: the European Commission. Table 6 State of progress of mid-term evaluations of Objectives 1, 2 and 6 Member State Monitoring Committees Calls for tenders Body selected Number of reports delivered or planned Obj. 1 Cohesion Fund Country Spain Greece Ireland Portugal 1 CSF Committee 12 regional subcommittees 7 ERDF sectoral committees 1 regional/central (Cantabria) finalised 12 finalised 7 finalised 1 finalised 1 external 12 external 7 external 1 external 1 report planned for 07/97 12 reports planned for 05/97; 1 report delivered 7 reports planned for 05-06/97; 1 report delivered report planned for 05/97 1 CSF committee 13 regional committees 17 sectoral subcommittees selection underway 13 finalised 7 finalised 3 published 6 offers received 13 external Nomination underway — 13 reports delivered during 06/97 — 1 CSF committee 9 sectoral committees launched in 1996 1 CSF committee 7 regional subcommittees 3 multi-regional subcommittees 7 sectoral subcommittees finalised finalised finalised finalised 1 external 9 external 1 external 7 external 3 external 14 external 1 mid-term report delivered during 03. 97 9 mid-term reports delivered during 03. 97 report planned for the end of 97 7 reports delivered during 1st quarter of 97 3 reports delivered during 1st quarter of 97 12 reports delivered during 1st quarter of 97 — 1 report planned for 07/97 — 1 report planned for 07/97 Obj. 1 Other countries Germany 1 CSF Committee 6 regional subcommittees 3 ESF-EAGGF-FIFG subcommittees 1997 (restricted) 1996 — 1 external 6 external — 1 final report planned for 08/97 6 reports delivered during 05/97 — Belgium 1 SPD committee 03/09/94 1 external final report delivered during 05/97 C 3 4 7 2 6 / E N O f f i c i a l J o u r n a l o f t h e E u r o p e a n C o m m u n i t i e s 1 6. 1 1. 9 8 Member State Monitoring Committees Calls for tenders Body selected Number of reports delivered or planned France 6 SPD committees 6 finalised 6 external Corsica: mid-term report delivered during 12/96 Hainault: report planned for 07/97 Martinique/Guadeloupe: reports planned for 06/97 Re´union/Guiana: report planned for the end of 97 Italy 1 CSF Committee 8 regional subcommittees 12 multi-regional subcommittees short list available, selection planned for 11/97 1996 publication underway, selection planned for summer 97 0 internal evaluation: report planned for 07/97 external evaluation: report planned for 06/98 2 out of 8 external 0 report planned for beginning of 98 — The Netherlands 1 SPD committee 03/96 1 external report planned for the end of 1997 United Kingdom 2 SPD committees 1:02/96 (Merseyside 2: 1996 (H&I) 1 external 1 external 1 report delivered at the beginning of 97 1 report planned for the summer of 97 Northern Ierland 1:06/96 1 CSF committee 9 sectoral subcommittees May 96 1 external 9 external report delivered in May 97 9 reports delivered in the first half of 97 Austria 1 committee finalised 1 external report planned for autumn 1997 Obj. 6 Finland 1 SPD committee finalised 1 external report planned for 10/1997 Sweden 1 SPD committee finalised 1 external report delivered during 06/97 Source: European Commission. 1 6 1 1 9 8. E N O f f i c i a l J o u r n a l o f t h e E u r o p e a n C o m m u n i t i e s C 3 4 7 / 2 7 C 347/28 EN Official Journal of the European Communities 16. 11. 98 The impact of the second CSF programmes (including national cofinancing) on GDP growth Table 7 CSF 1994-1999 as a % of GDP HERMIN (1) (Average % annual growth rates for 1994-1999) demand supply Total OGM (2) (Average % annual growth rates for 1994-1999) DIO (3) (Average % annual growth rates for 1994-1999) Greece Spain(4) Portugal Ireland 5,9 2,6 5,5 3,3 0,9 0,6 1,6 0,5 0,9 0,3 0,2 0,2 1,8 0,8 1,8 0,7 0,6 0,2 0,4 0,1 1 0,5 1 0,6 (1) HERMIN: Econometric model based on the original model HERMES (Harmonised Econometric Research for Modelling Economic Systems), but with the special feature of being minimum size (Min), hence (HER)MIN. See Chapter 7 and in particular points 7. 2, 7. 5, 7. 10, 7. 11, and Annex III point 5. (2) Optimal Growth Model (OGM) by Pereira (1991 and 1995). See Chapter 7 and in particular points 7. 2, 7. 6, 7. 10, and Annex III point 7. (3) Dynamic input-output model (DIO) by Beutel (1995, 1996 et 1997). See point 6. 23 d) and Annex III point 9. (4) Regarding Spain, not all regions are eligible for Objective 1. Source: Evaluations based on the macroeconomic models concerned, cofinanced by the Commission. Annual growth multipliers of the CSF programme in 1994-1999 Table 8 HERMIN demand supply Total OGM DIO Greece Spain Portugal Ireland 0,2 0,2 0,3 0,2 0,2 0,1 0 0,1 0,4 0,3 0,3 0,3 0,1 0,1 0,1 0 0,2 0,2 0,2 0,2 Source: Evaluations based on the macroeconomic models in question, cofinanced by the Commission. Real convergence in GDP per capita in the OGM approach Table 9 9 0 90 8 0 80 7 0 70 60 6 0 50 5 0 40 4 0 e g a r e v a U E f o % Greece G r e e c e I r e l a n d Ireland P o r t u g a l Portugal S p a i n Spain 30 3 0 1993 1 9 9 3 1995 1 9 9 5 1997 1 9 9 7 1999 1 9 9 9 2003 2 0 0 3 2007 2 0 0 7 Note: With regard to these forward simulations, which were drawn up on the basis of relatively old data, only the trend should be noted, and not the absolute values of the projections, which, particularly for Ireland, turned out to be too pessimistic. Source: Evaluations on the basis of the OGM model (Pereira, 1995) 16. 11. 98 EN Official Journal of the European Communities C 347/29 The impact of CSF programmes on employment and sustainability in the 1989-1999 period Table 10 Authors Employment and unemployment Sustainability Public finances Balance of payments Bradley et al [HERMIN] Honohan (ed. ) [HERMIN] Herce and Sosvilla-Rivero [HERMIN] Christodoulakis and Kalyvitis [HERMIN] Sema Group [HELM] — Total employment increases — Public borrowing Trade balance deteriorates requirement increases at the start of the programme and decreases afterwards when supply effects are present — Growth effects with unchanged public sector borrowing requirement are smaller — Employment in manufacturing declines partially due to productivity increases — Employment in the non-traded sector increases — Total employment increases — See previous study Balance of payments improves as long as EU funding continues — Debt/GNP ratio declines substantially — Total employment increases — See study by Bradley et al Trade balance deteriorates due to the demand effects — Unemployment declines — Debt/GNP ratio declines — Total employment increases — Primary deficit increases Not considered — Urban unemployment falls — Total employment increases Not considered Not considered — Manufacturing employment declines — Employment in commercial services increases Pereira (1995) [OGM] Full employment model Primary public deficit as a % of GDP increases Current account deficit increases except for Spain in 1997-1999 Lolos and Zonallas [BBM] Employment increases and unemployment declines Fiscal deficit increases from 1989-1993 and declines in 1994-1995 Current account deteriorates Beutel [DIO] — Employment increases in Portugal, Greece and Not considered Spain Exports-imports increase in Portugal and Greece but decrease in Ireland — In Ireland the first CSF decreases employment while the second CSF creates new jobs FIDIMS Employment increases in Northern and Southern Italy Not considered Not considered C 347/30 EN Official Journal of the European Communities 16. 11. 98 The accumulated % gain in GDP (GNP) generated by a one % point increase in the share of CSF in GDP (1994-1999) Table 11 Physical capital Human capital Private capital Spain Ireland Portugal 2 1,4 1,9 3,3 1,2 1,3 1,5 1,1 Not available Source: HERMIN evaluations. 16. 11. 98 EN Official Journal of the European Communities C 347/31 ANNEX I Summary of the methodology to be applied as part of the ex post evaluation of Objective 1 CSFs (1989 to 1993) 1. The explicit aim of the ex post evaluation of the Objective 1 CSFs (1989 to 1993) was to: (a) assess the strategy and objectives selected in the CSFs and to draw the appropriate conclusions for future action. To analyse the consequences for the initial objectives of redefining and adjusting interventions during the implementation period; (b) take provisional stock of the concrete results of the CSFs and OPs as regards significant measures and make an initial assessment of their impact; (c) analyse the effectiveness of the implementation mechanisms and draw the necessary conclusions with a view to making future adjustments. 2. Methodological approaches were prepared by the principal consultant, who was responsible for coordinating evaluations made by other consultants on behalf of the Member States concerned: (a) for the first objective, in order to evaluate the logic intrinsic to the CSFs and OPs and to the adjustments made to the CSFs and OPs since 1989 — and, more specifically, to analyse the causes of these adjustments, the Community added value, internal consistency, external consistency, the relevance of the strategy to existing problems and the appropriateness of financing — the following tasks, based on a top-down approach, were to be carried out: — collection of information and key data: financial data, ex ante objectives, anticipated output/effects, target groups and areas, main instruments used, number of projects/measures approved and number of projects completed, — description of the environment and development of the programme (initial situation —current situation); identification of external factors influencing the development of the programme and its success/failure; analysis of the basic problems and causes; formulation of key regional development indicators, including indicators of changes in disparities, — identification of policy instruments (loans, subsidies and technical assistance), — prioritisation of objectives, identification of levels of priority and of the links between objectives and instruments; (b) for the second objective, the analyses to be performed were based on top-down and bottom-up approaches: — analysis of the output and effects of OPs, measures and projects, account being taken of ‘net losses’, and substitution and multiplication effects, — analysis of the relative effectiveness of measures and combinations of measures; analysis of the causes of successes/failures, — comparison of output and input, the aim was also to formulate key and optional indicators; (c) for the third objective, the aim was to analyse the effectiveness of CSF and OP implementation measures and of a small number of measures, and to identify existing and planned improvements in the planning, implementation, monitoring and evaluation processes; for each Member State, the examination would cover decision-making, planning, financial control, implementation, monitoring and inspection mechanisms, financial channels, partnership, additionality, fund coordination, and greater flexibility and transparency in the simplification and harmonisation process; implementation and monitoring would also be analysed in depth for a small sample of measures/major projects; (d) ad hoc samples of programmes, measures and projects were chosen with a view to substantiating a small number of physical and impact indicators and to carrying out in-depth analyses. C 347/32 EN Official Journal of the European Communities 16. 11. 98 ANNEX II Summary of the ex post evaluation approach in respect of Objective 2 (1989 to 1993) The ex post evaluation of Community assistance under Objective 2 for the 1989 to 1993 planning period, which generally included ERDF and ESF components, comprised: (a) general comparisons in all Objective 2 regions and areas of the socio-economic situation prior to Objective 2 Structural Fund assistance and of the socio-economic situation after assistance; analysis of the strategy employed in terms of the prioritisation of objectives, priorities and measures, and of indicators, planned and unanticipated effects, synergies between programme components and the utilisation of the various funds. These analyses were to be based on the content of the CSFs and OPs, on annual implementation reports or reports submitted to the Monitoring Committee and on prior and ex post evaluations, where available; (b) detailed studies covering 22 regions, the aim of which was to: — study in depth the CSFs, OPs and types of measures adapted, — analyse institutional structures and programme management procedures, — evaluate the impact and added value effects attributable to the CSFs and to the OPs, — evaluate the cost and effectiveness of these effects, — study the Objective 2 programme’s contribution to the regional development process. These in-depth evaluations employed the bottom-up approach, the aim being — using an examination of operations covering the selection process, typology, management process, the impact and effect in terms of added value, and synergies and additionality — to work upwards from one level to the next, compiling information first about individual measures and ultimately about the whole programme. ANNEX III Characteristics of the macroeconomic models used to evaluate the Structural Funds 1. The principal models used in the macroeconomic evaluation of the Structural Funds comprise: (a) econometric models, such as HERMIN (1) and HELM (2) and QUEST II (3); (b) calculable general equilibrium models, such as OGM (4) and BBM (5); (c) input-output models, such as DIO (6) and FIDIMI (7); (d) structural-residual analysis. (1) HERMIN: Econometric model based on the original HERMES model (Harmonised Econometric Research for Modelling Economic Systems), but whose minimum size (Min) distinguishes it from this latter, whence (HER)MIN. (2) HELM: Hainaut Economic Lead-in Model. (3) QUEST II: a Multi-Country Business Cycle and Growth Model — Economic Paper No 123 — October 1997. DG II — Economic and Financial Affairs. (4) Optimal Growth Model (OGM); see Pereira, 1991 and 1995. (5) Bourguignon, Branson and de Melo model (BBM); see Lolos and Zonzilas, 1992. (6) Dynamic Input-output model (DIO); see Beutel, 1995, 1996 and 1997. (7) FIDIMI Consulting Company, Evaluation of the macroeconomic impact of the Objective 1 CSF (Italy) and of the implementation mechanisms. 16. 11. 98 EN Official Journal of the European Communities C 347/33 The characteristics of the application of these models in the cases examined are shown in diagram form in Tables III 1 and III 2. Table III 3 reproduces a model of the characteristics of the CSFs. 2. Furthermore, by simulating the evolution of the economies of Objective 1 countries over a period of more than 10 years and on multi-sectoral bases, it is possible to perceive the structural change brought about by the Structural Funds and to evaluate the reliability of the catching-up process which Community assistance is supposed to support and consolidate. 3. As part of its enquiry, the Court had an analysis performed (8) to assess the appropriateness of these macroeconomic techniques for measuring the impact of structural measures and to establish whether they may contribute effectively to the decision-making and strategic aspects of programming. 4. The analysis showed that explanations of the endogenous or exogenous factors upon which the success of the structural programmes depends vary according to the methods and conceptual approaches employed. The fact is that macroeconomic models aim to describe the mechanisms of the economic system of a country or region and to capture or simulate the effects of an exogenous shock such as the assistance provided by the structural programmes. However, no single description of the economic facts is definitive. 5. The HERMIN econometric model takes into account the specific characteristics of each Member State and aggregates effects at a national level. It assumes that economies actually utilise their full production capacity and that the accelerator and multiplier effects related to increases in investment consequently create an expansion in production and employment. This model seems to give satisfactory results in terms of effects on demand. With regard to the effects on supply, taking the Phillips curve into account, the model suggests that a fall in unemployment will bring about an increase in wages whilst limiting the effect on inflation. Also, the effects on productivity are explicitly taken into account by modelling the external saving effects that increases and improvements in human capital and public and private physical capital have on productivity. They are, however taken into account differently from one country to another. The model provides only partial information on durable development, and, in particular, does not consider variations in exchange rates aimed at guaranteeing the stability of the balance of payments or variations in fiscal transfers. 6. The HELM macro-econometric model, which was used for the ongoing assessment of the 1994 to 1999 Objective 1 SPD for the Province of Hainault (Belgium), is one of the few to take the regional development approach specifically into account by considering the effects of endogenous growth linked to long-term productivity, although improvements in productivity due to SF intervention are largely limited to the industrial sector and only concern labour productivity. On the basis of a similar assumption to that of the HERMIN model with regard to the utilisation of productive capacity and the multiplier and accelerator effects of investment on demand, it gives the overall and aggregate effects of the SPD on the Hainault economy, such as overall investment generated, added value or employment in the manufacturing industry and commercial service sectors for the period between 1994 and the years 2002 or 2005. It should be noted that the HELM model does take regional exports into account, but not imports from other regions, thus risking an overstatement of the effects on demand. The figure for demand is all the higher because neither wage adjustments related to the fall in unemployment nor higher costs or capacity constraints were taken into account. The HELM model also gives little information as to the trend of unemployment or the demand for labour. Lastly, it is based on historical statistical data, but these were not cross-checked against any other series of relevant data. 7. The computable general equilibrium models apply more particularly at a national level. They show the long-term effects of Community structural measures on supply. The OMG model, which uses a context of inter-temporal endogenous growth, assumes that the production factor markets are in equilibrium and that economies are working at full production capacity. Changes in demand have no effect on (8) The Court has had the services of an external assessor who was highly specialised in the field of macroeconomic and European economy models. C 347/34 EN Official Journal of the European Communities 16. 11. 98 production and employment and the SF programmes can only stimulate growth by increasing productivity. These productivity effects are specified clearly in the production function. The objective of the model is to determine an optimum situation for the country by maximising a function of inter-temporal social well-being. On the basis of the hypothesis of a situation without uncertainty, economic agents, anticipating perfectly the amount of money that will be allocated to them and the period over which aid will be spread, are supposed to take the best possible consumption and investment decisions in relation to these data. Lastly, the OMG model provides for long-term stability of the balance of payments and budgetary equilibrium. 8. For certain sectors, the BBM model expects that labour markets will be in equilibrium and that, consequently, demand will not influence production and employment, while, for other sectors, an expansion in demand is supposed to generate growth. This model also includes technological change — related productivity effects that result in reductions in inputs. 9. The DIO input-output model tries to show the links between all the resources available to an economy and all final consumption. In this way it identifies interdependencies between the various sectors of a given country, linked together on the basis of intermediate consumption. The input-output approach maximises the effects of structural interventions on demand and ignores those on supply (effects on productivity and wages and inflationary adjustments). Durable development is not taken into consideration. The FIDIMI model is a combination of a general equilibrium model and an input-output model which can be used to analyse the effects of Community interventions in Southern Italy on the North of the country. 10. Among other existing techniques for measuring the effects of regional policies, Eurostat uses structural/residual analysis to evaluate the impact of the SFs in certain areas eligible under Objectives 1 or 2. This analysis attempts to identify the residue resulting from the difference between the actual situation and the counterfactual situation produced via the application of estimated national tendencies. However, carried out in this way, the calculation of impact mixes the results of the programme under evaluation, other national and European political measures and the path previously followed by regional policy. Moreover, the studies carried out by Eurostat in 1997 include data for indicators, the most recent of which date back to 1993. This considerably reduces the value of their conclusions. 11. The QUEST II model attempts to analyse the economies of the Member States and their interaction with the rest of the world, in particular with the United States of America and Japan. The model is concerned more particularly with the transmission of the effects of economic policy at the national and international levels. It was initially devised to serve as a simulation tool rather than for predictive purposes. The QUEST II model now pays more attention to the effects of demand than its predecessor and takes into greater consideration interactions in terms of stocks. However, it gives little or no attention to the implications in terms of changes in the behaviour of economic agents. Table III. 1 Fundamental characteristics of the macroeconometric models Authors Model Country or region Demand effects Sustainability Wages Productivity Supply effects Bradley et al. (1994) HERMIN Ireland and Portugal — Standard Keynesian set-up investment demand stimulates aggregate demand — Unemployment augmented Phillips-curve set-up links employment, wages and inflation — CSF-related investments increase sectoral total factor productivity in Ireland Balance of payments: — Demand effect increases imports — Labour markets are not automatically in equilibrium — Inflationary impact of demand expansion is limited Honohan (ed. ) (1997) HERMIN Ireland — Economy does not — Rising wages cause labour operate at full capacity inflow in Ireland — Monetary sector and — Investment in human exchange rates are not modelled capital does not have wage effects — In Portugal no productivity — Productivity effect effects of private investment and certain infrastructure expenditures improves exports and attracts foreign direct investment in Ireland — Modelling of productivity effects is debatable in particular for human capital — Exogenous exchange rate does not allow full evaluation of balance of payments — Productivity effect may lower employment Public finance Herce and Sosvilla-Rivero (1994) HERMIN Spain See model for Ireland but no labour migration See model for Ireland See model for Ireland but: — Human capital investment increases labour productivity instead of total factor productivity — Aid to private investment decreases the user cost of capital — Limited modelling of public sector — Public sector borrowing increases due to CSF — Simulation with unchanged public sector borrowing is considered See model for Ireland but no link between productivity effect and foreign direct investment. 1 6 1 1 9 8. E N O f f i c i a l J o u r n a l o f t h e E u r o p e a n C o m m u n i t i e s C 3 4 7 / 3 5 Authors Model Country or region Demand effects Sustainability Wages Productivity Supply effects Christodoulakis and Kalyvitis (1995) HERMIN Greece — See model for Spain See models for Ireland — Exchange rate is modelled Sema Group (1997) HELM Hainaut — Keynesian/Kaldor set-up No wage and price effects — Investment demand stimulates aggregate demand — Labour demand is modelled — Labour supply and unemployment are not modelled — Imports from other regions are not modelled See model for Spain and Ireland but: — Human capital investments decrease unit labour cost — Aid to private investment See model for Spain but: — Nominal exchange rate adjusts to maintain a constant real exchange rate does not have productivity effect — Fiscal rule is assumed to stabilise public debt — Investment in ‘soft infrastructure’ raises public employment — CSF-related programmes raise primarily labour factor productivity in the manufacturing sector — In the reporting of the results no distinction between supply and demand effects is made No evaluation of consequences for public debt or balance of payments C 3 4 7 3 6 / E N O f f i c i a l J o u r n a l o f t h e E u r o p e a n C o m m u n i t i e s 1 6. 1 1. 9 8 Fundamental characteristics of the computable general equilibrium and input-output model Table III. 2 Authors Model Country or region Demand effects Sustainability Wages Productivity Supply effects Pereira (1991) Optimal Growth model (OGM) Portugal — No Keynesian demand effects — Wages adjust to — All factor markets are in equilibrium clear labour market Productivity effects because public, human and private capital are in production — Intertemporal welfare maximisation guarantees dynamic optimal solution Pereira (1995) Portugal, Greece, Ireland and Spain — The economy operates at full capacity Lolos and Zonzilas (1992) Bourguignon, Branson and de Melo model (BBM) Greece — Keynesian demand effects — Wages adjust to — Labour markets clear in some sectors and not in others — Fixed and flexible exchange rate are considered clear labour markets in some sectors — Unemployment augmented Phillips-curve in the other sectors — Balance of payments and public finance constraints are satisfied — No constraint imposed on public borrowing — Measures of income inequality are considered — Investment in public and private capital increases sectoral total factor productivity in a debatable way — Investment in human capital increases skilled labour input and public employment Beutel (1995, 1996, 1997) Dynamic input-output (DIO) Greece, Portugal and Spain Keynesian demand effects No wage effects No productivity effects No modelling of balance of payments and public finance constraints. 1 6 1 1 9 8. E N O f f i c i a l J o u r n a l o f t h e E u r o p e a n C o m m u n i t i e s C 3 4 7 / 3 7 Honohan (ed:) (1997) Herce and Sosvilla-Rivero (1994) (HERMIN) Christodoulakis and Kalyvitis (HERMIN) Sema Group (HELM) Table III. 3 The modelling of the characteristics of CSF programmes Authors CSF programme Sectoral focus Instruments Co-financing and additionality Starting point and end game Bradley et al (1994) 1994 to 1999 — Four sectors: traded, — Physical infrastructural (public) — Some studies compare full non-traded, public and agricultural sectors — Traded sector is key sector for supply effects — Non traded and public sector are key for demand effects investments — Human capital investment — Investments and production aids to the private sector domestic co-financing with no or partial domestic co-financing — Forecasts for 1991 to 2020 except Honohan (1993 to 2010) — Hard and soft landing scenarios are considered 1994 to 1999 Same as previous studies — Hard infrastructure — Soft infrastructure — Aid to productive investment — Education and training — Simulations assume projected co-financing rates are realised — Forecasts for 1991 to 2010 — Hard landing scenario 1994 to 1999 — Six sectors, agriculture, manufacturing, energy, construction, commercial services, non-commercial services — Investments in public and private — Scenario where European Union — Forecasts for 1994 to physical capital — Investments in human capital — Investment in R&D capital and regional aid of 1989 to 1993 is continued afterwards 2005 — Scenario where additional European Union aid and domestic co-financing for 1994 to 1999 is added — Scenario where no European Union aid is given in 1989 to 1999 Lolos and Zonzilos (BBM) 1989 to 1993 Nine sectors: agriculture, energy and mining, food industry, light industry, intermediate industry, construction, trade and commerce, transport and telecommunication services, other services (including Government) — Public investments in selected sectors Additionality is not discussed but, presumably, imposed on the model — Human capital investments increase public employment, the number of skilled workers and generate transfers to households — Support to private firms — Forecasts for 1988 to 1995 — Hard landing C 3 4 7 3 8 / E N O f f i c i a l J o u r n a l o f t h e E u r o p e a n C o m m u n i t i e s 1 6. 1 1. 9 8 Authors CSF programme Sectoral focus Instruments Co-financing and additionality Starting point and end game Beutel [DIO] 1989 to 1999 25 sectors based on input-output tables FIDIMI (1991) 1989 to 1993 16 sectors based on input-output tables — Sectorally focused modelling of Additionality is imposed on the model — Forecasts for 1989 to public, private and human capital investments — Investments in training affect wages and salaries in the economy 1994 — Sectorally focused modelling of — Additionality is a key part of the — Only total effects are CSF programs study — Scenarios differ in the amounts of funds that would be invested in Southern Italy with and without CSF programmes reported without a yearly break-up — No discussion of hard and soft landing scenarios Pereira (1991) (OGM) 1989 to 1993 Two sectors: traded and non-traded — Investments in public infrastructure — Co-financing requirements are — Forecasts for 1990 to — Investments in human capital — Support to private investment imposed 2006 — Model allows an evaluation of — Soft and hard landing whether additionality will be met scenario’s are considered Pereira (1995) (OGM) 1994 to 1999 and comparison with 1989 to 1993 See previous model See previous model — Co-financing requirements are — Forecasts for 1993 to imposed 2007 — Simulations compare, cases of: — Hard landing scenario (i) no-additionality (ii) official definition and (iii) intertemporal definition of additionality. 1 6 1 1 9 8. E N O f f i c i a l J o u r n a l o f t h e E u r o p e a n C o m m u n i t i e s C 3 4 7 / 3 9 C 347/40 EN Official Journal of the European Communities 16. 11. 98 COMMISSION’S REPLIES I. GENERAL COMMENTS The Commission accepts that the evaluation process, being something entirely new for many parties, is still rather rough and ready as regards its scope and the use that can be made of it, but insists that remarkable progress has been made in the space of just a few years. Conscious of the difficulties that still have to be overcome, the Commission has redoubled its efforts on the organisational and methodological front to achieve the most satisfactory situation possible, engaging in a permanent dialogue with all those in the different countries of the Union who are studying and carrying out research in this area. There are three main phases in the development of the assessment process for Structural Fund intervention, corresponding to the three programming periods: 1989 to 1993 period The regulations on the first reform of the Structural Funds were not adopted by the Council until December 1988 and the national plans submitted at the beginning of 1989 only partly reflected the requirements for strategic vision, choice of objectives to be achieved and quantification of expected results. The ex post evaluations therefore attempted to remedy these shortcomings by comparing the results achieved with the reworked objectives. This considerable task suffered from its own limitations, which are reflected in the reports dealing with the interventions in Objective 1 and Objective 2 areas. Nevertheless, it represents the first attempt at coherent analysis of the impact of this reform on economic and social cohesion. 1994 to 1999 period At the request of certain Member States and of the Commission, this first experiment was followed by a tightening of the rules on monitoring and evaluation and the identification of three stages: prior appraisal, ongoing assessment and ex post evaluation. There can be no doubt about the success of the ongoing assessment. Going beyond what was strictly required by the regulations, the mid-term assessments were organised for the first time by and at the level of each Monitoring Committee, thus bringing assessor and the beneficiaries of the assessment closer together. The reports produced were presented to and debated in each committee, with the result that the process served an educational and operational purpose that studies organised and carried out by the Commission could not hope to fulfil. There may have been some discrepancies between the methods employed, the choice of assessors, the cost of the work and the level of detail of the reports, even though coordination and general supervision were provided by the Commission’s evaluation coordination unit and its national contacts in the technical evaluation group. However, despite certain differences connected with the discretion allowed to each Member State under the subsidiarity principle, the Commission believes that this process played an essential role in propagating a genuine climate of evaluation. It proved to beneficiaries that this was an operational tool for improving the management of interventions financed from both national and Community sources. Future evaluations can only benefit from this attitude in that the quality of their conclusions depends as much on the quality of the information in the possession of and supplied by the project managers as on the expertise of the independent assessors. 2000 to 2006 period The new proposals for regulations aim in particular to reduce still further the factors limiting the scope of the evaluation process, out of concern primarily to increase the efficiency of the deployment of Structural Funds. The following proposals are made in this spirit: — the prior appraisal should be properly incorporated into the plans which Member States are required to submit containing a strict quantification of the objectives to be attained, — the results of evaluations already carried out in the previous period, either on a particular topic (thematic evaluations) or by a specific programme, must be taken into account, — there should be a close correlation between mid-term reprogramming and the conclusions of the ongoing assessment. In a bid to ensure greater effectiveness in the use of Structural Funds, the Commission has proposed that 10 % of the Structural Fund aid be disbursed at mid-term, depending on performance, the criteria for assessing performance being closely linked to the quality 16. 11. 98 EN Official Journal of the European Communities C 347/41 of the programming, follow-up, prior appraisal and ongoing assessments. II. SPECIFIC COMMENTS Finally, with regard more specifically to methodology, the Commission plans by the autumn to publish the work carried out under the MEANS programme (1) in order to ensure greater harmonisation of evaluation within the Union. This will involve: — a detailed description of specifications and the methodology for prior appraisal, ongoing assessment and ex post evaluations, — a survey of the literature on different assessment methods and the conditions for their application, — an analysis of systems of indicators by area of intervention. Harmonisation of concepts and a list of suggested indicators, — methodology and organisation of subject-based evaluations, — a multilingual glossary of the main concepts of evaluation, — an outline of the most innovative national or Community projects. 2. THE SCOPE OF THE AUDIT 2. 3. The Court did not include in its audit countries like Germany and the United Kingdom that have a very different attitude to evaluation. 3. SUMMARY OF THE MAIN OBSERVATIONS 3. 2. The Commission believes that the evaluations for which it does not have the figures on expenditure are marginal in terms of costs. Because ongoing assessment is a new and growing field and is carried out at a decentralised level, the volume of expenditure devoted to it is changing. Expenditure on evaluation is broken down by country, by objective and by type of intervention. The figures are also published in the Commission’s annual evaluation overview. The Commission will continue its efforts to obtain a systematic overview of the amount of expenditure set aside for evaluation out of the sums awarded in technical assistance and managed under the partnership system. It will be clear in what follows that the Commission endorses the Court’s comments in some cases. But it is important to take account of the difficulties and limitations inherent in any evaluation exercise, even if these are being gradually overcome or superseded. 3. 3. Under the terms of the regulations and in deference to the subsidiarity principle, the Commission has no authority to dictate a precise framework for evaluation indicators and methods. Nor should we lose sight of the fact that evaluation is first and foremost a technical aid to decision-making and management; it is not intended to take the place of political decision-makers. It serves to shed light on the consequences of choices to be made or decisions already taken. Moreover, there is a need for a degree of flexibility as to the methods used in order to allow for the different contexts from one evaluation to the next. Finally, the Commission would point out that since the start of the Fund reforms in 1988, the respective responsibilities of national administrations and the Commission for Structural Fund management in general and evaluation in particular have been constantly shifting. The Member States are increasingly required to assume prime responsibility and to carry out their own evaluation. (1) MEANS = Methods for evaluating activities of a structural nature: a multi-annual programme to analyse and propose methodologies of evaluation. In an effort to develop national practice and improve the quality of evaluation, the Commission has: — set up the MEANS programme and distributed eight booklets on methodological questions, — organised three European conferences on evaluation techniques and practice, — distributed documents such as the frame of reference for prior appraisals and a guide for evaluating major projects, — set up Technical Evaluation Groups in which the Member States participate and organised several seminars. is an objective problem to 3. 5. There coordinate the ex post evaluation with the different stages of planning structural interventions. Ex post evaluations should really be carried out two or three years after the end of programming. trying in C 347/42 EN Official Journal of the European Communities 16. 11. 98 In order to solve this problem the Commission has promoted ongoing assessments and thematic assessments. The thematic assessments carried out in 1992 and 1993, for example, produced material that could be used when preparing the programming for 1994 to 1999. The ongoing assessments will be available for the preparation of programming after 2000, because they have already been completed, and evaluation overviews are currently being introduced for each objective to supplement and update the ongoing assessments. Four thematic evaluations have also been started. The Commission has introduced this provision into its proposals for amending the Regulations. should be made between 3. 6. A distinction the objectivity of the evaluations and the adequacy and quality of the data. The question of data is often highlighted in evaluations. The reports almost always pinpoint this problem. However, there has been a distinct improvement in the situation as regards data, largely thanks to the demands of evaluation. The Commission is not satisfied with the current situation as regards data, but this should not be regarded as a criticism of evaluation but of the monitoring systems. The Commission would stress that the problem of the relevance of monitoring data and indicators is a matter primarily for the follow-up audit. The Commission is continuing its efforts to improve the situation and will act on the Court’s comment about a need for more systematic evaluation of the reliability of data. The work carried out in connection with the audit and control will also help to consolidate the improvement in the situation which has been apparent in the field for several years. 3. 7. The departments concerned comply with the conditions for awarding contracts prescribed in the relevant Directives and with the procedure laid down by the ACPC. However, it is not possible to negotiate the remit before signing the contract. This sometimes means that clarifications have to be made which upset the timetable, but the problem is marginal. basis, some reports in a country may be excellent while others are unsatisfactory. 3. 8. The Commission shares the Court’s views about the difficulties experienced by the first evaluations organised in a climate of indifference in the Member States. The ex post evaluations of Objectives 1 and 2 were carried out by consultancies with an international reputation and by highly qualified experts using a common methodology. The second part of paragraph 3. 7 justifies both the measure and the evaluation exercise, despite these constraints. The Court’s criticisms refer to the evaluations carried out in 1993 to 1994, but on the Commission’s initiative a great deal of work has been done since then on developing methodologies. The Court’s criticism of the ex post evaluation of Objective 2 is also valid in part. However, a more realistic view should be taken of the possibility of devising a precise impact or Community quantitative measurement of added value. 3. 9. The ongoing assessment took the form of an analysis of the implementation of the interventions. Effects are more likely to be assessed in the context of the evaluation overviews and ex post evaluations. These evaluations, however, do not always involve the use of econometric models — which tend to be very expensive and are not necessarily the most appropriate way of measuring impacts. Evaluation is a management tool and, as such, just one of several aids to the decision-making process. 3. 10. The Commission found that dividing Objective 2 programming into two periods did not correspond adequately to the needs of the areas in question and implementation of programmes more made complicated. The Commission’s proposals in March 1988 for a reform of the Structural Funds provided for a single programming period for Objective 2, as is the case for the other objectives. the In order 3. 11. — 3. 12. to avoid having an unmanageable number of evaluations of programmes or selective approach may sub-programmes a more sometimes be needed in certain key areas. The Commission alone cannot be blamed for this problem. The inadequacies of some evaluation reports may also be due to incompetence on the part of the consultants or the poor quality of the data available. Even when the same terms of reference are used as a The Commission shares the Court’s opinion on the need to cut the number of measures and sub-programmes or even programmes. It made a number of specific proposals to this effect in the course of the Fund reform. 16. 11. 98 EN Official Journal of the European Communities C 347/43 4. GENERAL ISSUES Introduction 4. 1. On several occasions the Commission expressed its regret at the failure of certain Member States to quantify their objectives adequately. The fact that national public expenditure is not administered by objective also hampers the development of management systems based on indicators. Consequently it is less a matter of a defect in the methodological or analytical framework than a problem of programme design and the sophistication of national statistics. 1 CSFs and 15 Objective 2 SPDs at the beginning of the period. In addition to the thematic assessments, which also relate to the period 1989 to 1993, the Commission carried out an ex post evaluation of all CSFs and SPDs for Objectives 1 and 2 in the period 1989 to 1993. 4. 11. The question of knowledge of costs was raised in paragraph 3. 2. The only evaluations that might not be included in the financial overview of evaluations drawn up by the Commission are the small number initiated solely by the Member State and funded from technical assistance. The amounts involved are marginal compared with all the other work. 4. 3. The Commission would point out that: Frame of reference — in 1994 it sent Member States its common ongoing on monitoring and guidelines assessments, (consisting — in December 1994 it presented the Technical of those Evaluation Group responsible for evaluation in the Member States) with a framework document on the purpose and content of and procedure for ongoing assessments. This initial meeting was followed by four further meetings of the same group, — in 1995 to it distributed the various departments copies of a MEANS document on how to organise ongoing assessments in the context of partnership. Three of the five deal with chapters formulating the specification and invitations to tender, selecting assessors and supervising the quality of the evaluation, document this of — in 1996, before receiving the first evaluations, it distributed to the departments involved a MEANS document on assessing the quality of evaluation a detailed schedule. including reports, It should be recalled, however, some 4. 5. evaluations were carried out under partnership arrangements even in the period 1989 to 1993. Thus Member States and the Monitoring Committees actively participated in several thematic and ex post evaluations initiated by the Commission. that 4. 12. — 4. 13. The MEANS documents were intended as educational tools for administrations which are often still developing their technical expertise. The priority for the Commission was to raise the average standard of evaluation, not the level of experts who are already specialised. Statistical bases and information concerning objectives 4. 14. — 4. 15. The Commission shares the view of the Court. Indicators efforts to develop acknowledges 4. 21. The Court the 4. 16. — Commission’s indicators. The regulations were tightened up in 1993, and an exercise to quantify disparities was launched with each Member State. The question of quantifying objectives has been a priority for the prior appraisal and some progress has been made: the 1994 to 1999 interventions are distinctly better quantified than their predecessors. The first task of the ongoing assessments has been to improve the indicators and their effectiveness. 4. 10. The Court comments on the small amount of money earmarked for Objectives 1 and 2 evaluations in the first programming period. Because almost none of the Member States had carried out an evaluation, the Commission itself took the initiative of conducting a first evaluation of all Objective The Commission would stress that the problem is not just one of increasing or reducing the number of indicators but the difficulty the monitoring systems have in regularly quantifying the agreed indicators that already exist for the programmes. Nor is it a question of a lack of coherence in programming, for it is the right of the regional and local authorities, under the principle of subsidiarity, to set their own priorities and determine the scope of programming in their countries. C 347/44 EN Official Journal of the European Communities 16. 11. 98 Progress is, however, being made, under the pressure and demands of evaluation. The objectivity of evaluations The Commission’s view is that a simple methodological framework is not enough and it therefore intends to take more far-reaching steps as part of the Fund reform. In these paragraphs the Court does not 4. 28. — 4. 30. make sufficient allowance for the complexity of the partnership system, despite the positive comments in paragraphs 6. 5 to 6. 7. Taking evaluation into account the that agrees 4. 23. The Commission dual programming cycle did not make coordinating evaluation and programming any easier and for this and other reasons it proposed a single programming period for the future. Nevertheless, the 1997 to 1999 programming drew on an ex post evaluation of the 1989 to 1993 period and on a prior appraisal of the SPDs carried out on the Commission’s initiative and relating to matters under its authority and priorities for the Union. As for the ex post evaluation of the 1994 to 1996 period, the Commission believes that the evaluations carried out in the context of reprogramming and the thematic evaluations of SMEs and R&D cover much of this ground. The Commission is currently discussing with the Member States the launch of an evaluation overview in the second half of 1998 covering the period since 1994. This overview would be used for future programming. 4. 24. — 4. 27. The Commission is currently drafting a report describing, for each country, the quality of evaluation and its contribution to modifications to interventions. A summary will cover the key points of the mid-term review. The information available to the Commission does not support the Court’s comments relating to widely divergent situations and hundreds of programmes. The Commission will assess the results when the report on the mid-term review is presented. For the time being, the Court’s position seems to be premature. The Commission would also point out that evaluations are useful not only for managers but also in the interests of greater accountability to the budgetary authority, the European Parliament and the public. to the is more There issue of objectivity and independence of evaluations than the existence of a precise frame of reference for applying the methodologies and checking data on the spot. The quality of the evaluation relates more to degree of professionalism, professional ethics and the interplay with the interests of the Member States. The Court fails to mention that three of the eight criteria in the MEANS brochure No 5 on assessing the quality of evaluation reports explicitly refer to the appropriateness of the methodology, the reliability of the data and the credibility of the results. The Commission is constantly striving for improvement in the standards and requirements, bearing in mind the reality on the ground. It will be turning its attention more to assessing the operation of evaluation systems on the ground. The evaluation contracts awarded by the Commission 4. 31. — 4. 32. The Commission applies the standards and procedures for awarding contracts laid down by the ACPC. Experts are chosen on the basis of selection and award criteria published in the Official Journal. Their application and the results of this exercise feature in the report to the ACPC. As regards the degree of precision of the contracts it should be borne in mind that the Commission is paying for expert advice and has every interest in ensuring that the contracts are drafted in such a way as to enable the experts appointed to make the best use of their skills. for Objective 1 is not 4. 33. The method used appropriate for Objective 2, which comprises 60 programmes. Given the expenditure involved and the number of teams, it would lead to a fragmentation of the invitation to tender that would go against the letter and the spirit of the Directive on the procedure for awarding public works contracts. 16. 11. 98 EN Official Journal of the European Communities C 347/45 The Commission complied with the rules in these cases and was careful not to lose control of the practical arrangements for implementing the evaluation. To this end a steering committee was set up to accompany the evaluation and meetings were organised periodically with the evaluation teams. The reports were subsequently checked with the departments in charge. 4. 34. The details of the specifications depend on the needs and information requirements of the requester. The requester drafts them and the evaluation units carry out checks. The terms of the contract, with the exception of the description of work to be carried out are standardised by the ACPC. As pointed out earlier, there is no generally accepted methodology for programme evaluation. This being the case, and in order not to lose its control over the approach adopted, the methodology, its relevance to the questions raised and the applicability of the techniques proposed. The general methodological approaches form part of the specification, but the precise details and techniques are left open to discussion. the Commission discusses 4. 35. In this specific case the Commission, with the the agreement of evaluation, tried to improve the quality of the report by organising a meeting on the content and the lessons to be learned with all the units and officials concerned. the consultants who produced The final report was submitted in May 1995 (three months after the scheduled date), accepted after official consultation in June 1995 and payment was made in July 1995. The report criticised by the Court was in fact a clarification by the consultant involving no additional budget. that 4. 36. The Commission stresses terms of three principal, structured around reference were well-defined questions: the relevance of the strategy, the results and effectiveness in terms of output and the effectiveness of the implementing strategies (pages 2 to 4 of the document). the The Commission explained its position on the question of methodologies various departments are consulted and the lead department decides whether or not to accept the report. in paragraph 4. 12. The 5. THE EX POST EVALUATION OF THE OBJECTIVES 1 AND 2 CSFs (1989 to 1993) Introduction 5. 1. The Commission endorses the Court’s comments on these evaluations the difficulties experienced with connected with first-generation programmes. the design of the The ex post evaluation of Objective 1 CSFs (1989 to 1993) 5. 4. Some of the Court’s comments on this point should be qualified. (a) If data on regional disparities were out of date this is a criticism of the statistical office rather than the methodology of the evaluation. (b) The assessors used the sampling techniques generally employed for evaluations, combined with case studies for greater detail instances. These techniques, properly used, can provide useful lessons. specific in (e) The Court states that bottom-up analyses do not generally allow conclusions to be drawn in terms of impact on impact at programme improving the Community policies, management, transparency, etc. cannot be top-down or based on econometric models. impact on level. The (g) The Court criticises aspects that were not within the remit of the evaluation. 5. 7. The Commission confirms that the evaluations in question were fully used in the context of its duty of accountability to the budgetary authority and as part of the work on the report on Economic and Social Cohesion. 5. 8. One of the three objectives of the remit for the evaluation was to suggest ways of improving the monitoring and management system. The relevance of the strategy was also analysed and a performance assessment drawn up which enabled the effectiveness of the interventions to be analysed. The ex post evaluations of Objective 2 (1989 to 1993) 5. 13. The systematic use of econometric models is not justified in all cases, either because the level of aid is low or because the necessary regional statistical series are unavailable. Most of the assessments of the policy appear in the report on Economic and Social Cohesion. The Commission has suggested modifications as part of its new proposal for the reform of the Structural Funds from the year 2000. C 347/46 EN Official Journal of the European Communities 16. 11. 98 5. 16. The Commission has already included a number of the recommendations of this evaluation in its proposals for the new regulations. Bottom-up methods are used for evaluating effectiveness and efficiency. In the case of large CSFs, where the consequences of reprogramming on global impacts had to be determined, use was made of econometric modelling techniques in Ireland, Spain, Greece, Portugal and Italy. 6. ONGOING ASSESSMENTS (1994 to 1999) The development and integration of the principles of ongoing assessment 6. 1. — 6. 20. The Commission shares the opinion of the Court in this part of the report. It would clarify a number of points: — Paragraph 6. 15: The Commission believes that critical appreciation of the development strategy, the impact in terms of objectives, and project viability are matters more specifically for the prior appraisal or the ex post evaluation. Paragraph 6. 16: The purpose of the ongoing assessments is to adjust the interventions and improve management. Analysis of the Community added value will be part of the evaluation overview or ex post evaluation. However, the evaluation of the impact of Community priorities on the modification of interventions will in several cases contribute valuable information about Community added value. summary of in countries with Paragraph 6. 18: As far as possible, the Commission tried to obtain a picture via the ongoing assessment of the synergy between operations financed by different Funds, particularly separate programming, as in the case of the German Objective 1 SPDs. To this end the assessors appointed for ERDF projects produced a the evaluations carried out under each Fund. The Commission agrees that the regional evaluation cited by the Court was particularly weak, but it is not representative of all the evaluations carried out in that country. It is correct to state, for example, that the ongoing assessment did not analyse the changes in the indicative funding plan resulting from a request by Brandenburg to change its OP. As for the high rate of absorption of funds allocated in the case of the projects referred to, a number of changes were made to the measures in the Monitoring Committees of 12 June 1996 and 5 August 1996. Approaches, methods and techniques 6. 21. The Court criticises the vague formulation of interim objectives in the contracts. The Commission gave guidelines in certain cases, but these were not followed at the decentralised level. 6. 22. The Commission drew on the best expert opinion in Europe and the United States of America to identify the most effective ways of evaluating programmes. Objective methodological difficulties and, in many cases, inadequate monitoring systems, limit the comparability of results at European level. 6. 23. The aim of ongoing assessments is not usually to measure the impact of programmes. This is a deliberate choice because the impact is not quantifiable in the first phase of the intervention, the sequence being first implementation, then results and finally impact. The question of the evaluation of the overall impact of large CSFs was discussed at paragraph 6. 21. 7. IMPACT OF THE STRUCTURAL FUNDS AND EVALUATION MODELS Impact of CSFs and programmes In this part of the report the Court 7. 1. — 7. 15. presents the results of different econometric and macroeconomic models but without any explanation of their scope or nature, which could lead to confusion. These data are presented in a structured manner in the report on Economic and Social Cohesion, with appropriate background information. Impact of sub-programmes, measures and projects 7. 16. — 7. 19. The Court appears to criticise the lack of evaluation at the level of sub-programmes or measures, yet is generally bottom-up, is necessarily based on the evaluation of measures and sub-programmes. the programme evaluation, which Similarly, the current thematic evaluations that have been launched by the Commission are concerned more specifically with certain categories of measure and/or sub-programme across the board, as in the case of measures for small businesses, R&D, the environment, etc. 16. 11. 98 EN Official Journal of the European Communities C 347/47 8. CONCLUSION 8. 1. — 8. 19. The Commission believes that the Court’s conclusions provide constructive criticism on issues which the Commission has been working on for years and will continue to develop. been made. The steps taken by the Commission under the aegis of the Structural Funds, reinforced by the SEM 2000 exercise, are helping to develop a culture of evaluation which did not exist before. There are few regions that have not carried out evaluations and that have not committed themselves to this process. The Commission admits that the Court has detected weaknesses but it has also noted the progress that has The subsidiarity principle and the difficulties and heterogeneity of geographic locations and subject areas are both factors to be taken into account