Communication from the Commission to the Council and the European Parliament - Independent external monitoring and evaluation of community activities in the area of research and technological development
\s COMMISSION OF THE EUROPEAN COMMUNITIES Brussels, 22. 05. 1996 COM(96)220 final COMMUNICATION FROM THE COMMISSION TO THE COUNCIL AND THE EUROPEAN PARLIAMENT INDEPENDENT EXTERNAL MONITORING AND EVALUATION OF COMMUNITY ACTIVITIES IN THE AREA OF RESEARCH AND TECHNOLOGICAL DEVELOPMENT SUMMARY The Commission is continually seeking to demonstrate and enhance the credibility and efficiency of its efforts in relation to evaluation of Community RTD actions. In the light of the recent legislative Decisions on the fourth Framework Programme and the related specific programmes, which prescribe a complex evaluation scheme and an active role for independent external experts in this scheme, the Commission decided to take a fresh look at Community RTD evaluation efforts. The Scientific and Technical Research Committee (CREST) was consulted for advice; subsequently, CREST established an Evaluation Sub-Committee. The Council's Research Group and Atomic Questions Group as well as the European Parliament's Committee on Research, Technological Development and Energy (CERT) have been informed of this development. This Communication summarizes the evaluation required by legislative Decisions and outlines a rationalized approach, taking into account the CREST advice. The approach will produce results in a timely, efficient and cost-effective manner. It concentrates on the continuous monitoring of specific programmes and the Framework Programme and their five-year assessment. Regular reporting will be made once a year, correlated with the publication of the Annual Report requested under Article 130p of the Treaty on European Union. The approach has been introduced on an experimental basis in 1995 and will be reviewed after three years of implementation. The approach builds upon the solid foundation of experience already acquired by the Commission from past evaluations and takes into account a recent initiative on concrete steps towards best evaluation practice in the Commission. The main feature of the approach is a further development towards coherent monitoring and evaluation of Community RTD actions. In particular, the approach: (i) (ii) (iii) (iv) introduces continuous monitoring and develops through programme evaluation to S&T policy evaluation; spells out the implementation principles; involves appropriate assistance from independent external experts; and should produce results in time for discussion on future programme Decisions. The Commission is responsible for the monitoring and evaluation of Community RTD efforts. However, the involvement of outside experts and bodies and the openness of procedures will enhance the independence and transparency of monitoring and evaluation. In addition, a new level of accountability is introduced through publication of the Commission's responses to recommendations produced by the five-year assessment panel of the Framework Programme. This will all enhance the credibility of Community RTD evaluation. Through this Communication, the Commission invites the Council and the European Parliament to take note of this rationalized approach which takes account of the current legislative requirements for monitoring and evaluation. 1. INTRODUCTION The Commission's White Paper "Growth, Competitiveness, Employment", COM(93)700 final, identifies increased RTD effort and its adaptation to new market conditions as one of the key means to improve the industrial and technological performance of Europe. Evaluation - both the evaluation of the programmes (their management, monitoring quality, efficiency of internal procedures, etc. ) and the evaluation of results (increase in scientific and technical knowledge, dissemination and optimization of results, industrial exploitation of RTD results and their relation to innovation processes, measurement of long-term economic and social effects, etc. ) - plays an important role in this regard by contributing to: the efficient implementation of RTD efforts; coherence with all relevant policies; and best use of public resources. The legislative Decisions on the fourth Framework Programme1 and its specific programmes, require the Commission to implement a complex evaluation scheme (see Point 3 below). In this Communication the Commission presents a rationalized approach which will produce results in a timely, efficient and cost-effective manner. 2. EVOLUTION OF COMMUNITY RTD EVALUATION EFFORTS The systematic evaluation of Community RTD activities was confirmed with the Plan of Action of 19832 and has since evolved further3. The use of external experts has been a key feature in promoting the independence of evaluation. In 1993, the Council and the European Parliament invited the Commission to continue to improve the credibility of the evaluation of Community RTD programmes while ensuring its independence. Subsequently, the legislative Decisions on the fourth Framework Programme included additional requirements concerning evaluation. In 1994, the Commission Services involved in the implementation of the Framework Programme, prepared a reflection document as a first step in response to the Council and the European Parliament in this regard. The document was transmitted to CREST, CERT and the European Science and Technology Assembly (ESTA). In 1995, CREST produced advice4 which included the establishment of a CREST Evaluation Sub-Committee. The Research Group and the Atomic Questions Group (Council), as well as CERT (Parliament), have been kept informed of this development. On 21 March 1995 the Commission approved a programme5 for "Sound and Efficient Financial Management (SEM 2000)", divided into three phases: consolidation within the In this document, the term "fourth Framework Programme" comprises Community activities resulting from two Decisions: J^ Decision N° 1110/94/EEC concerning the 4th Framework Programme of the European Community activities in the field of research and technological development and demonstration (1994-1998); and, 2L Decision N° 94/268/Euratom concerning a framework programme of Community activities in the field of research and training for the European Atomic Energy Community (1994-1998). Council Resolution of 28 June 1983 Communication to the Council, concerning a Community Plan of Action relating to the Evaluation of Community Research and Development Activities for the years 1987 to 1991, COM (86) 660 final, 20 November 1986 CREST advice to Council and the Commission on the monitoring and evaluation procedures for Community research programmes; adopted at its meeting of 18 May 1995. Recommendation n° 4 of the Memorandum from the President, Mrs. Gradin and Mr. Liikanen, SEC (95) 1301/4 of 22 July 1995; and Communication to the Commission on Evaluation (concrete steps towards best practice in the Commission). SEC. present framework, reform of financial management culture, and partnership with Member States. In adopting the second phase of this programme the Commission introduced, among other things, the requirement for systematic evaluation for all Community programmes and actions and clarified that it is primarily the responsibility of operational Directorate Generals to carry out the task of evaluation. 3. EXISTING SITUATION The legislative Decisions on the fourth Framework Programme and their specific programmes6 require the Commission (see Table 1): - - For the Framework Programme: - to continually and systematically monitor, with appropriate assistance from independent external experts, the progress in relation to its initial objectives (Articles 4. 1); and to have an external assessment conducted by independent qualified experts into the management of and progress with Community activities carried out during the five years preceding this assessment and communicate the assessment, accompanied by the Commission's comments, to the European Parliament, the Council and the Economic and Social Committee prior to presenting a proposal for the fifth Framework Programme (Article 4. 2); For the Specific Programmes: - to continually and systematically monitor, with appropriate assistance from independent external experts, the progress within the programme (Article 4. 1); to have an external assessment conducted by independent qualified experts of the activities carried out within the domains covered by the programme and their management during the five years preceding this assessment (Article 4. 2); - and, on completion of each specific programme, to provide an independent final evaluation of the results achieved compared with objectives and to forward this evaluation to the European Parliament, the Council and the Economic and Social Committee (Article 4. 3). To implement these legislative requirements with the existing methodology would lead to: i) a need to conduct 2 major evaluation exercises (one mid-way through the fourth Framework Programme and the other on completion), in each case for about 20 programmes; ii) too short a time interval between these 2 evaluations to be able to detect significant progress; iii) final evaluations being required before significant S&T results are achieved and their impacts become apparent; and iv) a requirement to clarify the roles of independent external experts who would provide 'appropriate assistance'. Council Decisions: 94/801/EC or 23 November 1994; 94/572/EC of 27 July 1994; 94/802/EC of 23 November 1994; 94/571/EC of 27 July 1994; 94/803/EC of 23 November 1994; 94/911/EC ofl5 December 1994; 94/804/EC of 23 November 1994; 94/912/EC or 15 December 1994; 94/9I3/EC of. 15 December 1994; 94/805/EC of 23 November 1994; 94/806/EC of 23 November 1994; 94/914/ECof 15 December 1994; 94/915/EC of 15 December 1994; 94/807/EC of 23 November 1994: 94/9 IT/EC of 15 December 1994; 94/916/IIC of 15 December 1994; 94/920/1-ura loin of 15 December 1994; 94/799/Euratom of 8 December 1994; and Council Decisions implemented by the Joint Research Centre: 94/918/EC or 15 December 1994; 94/919/Euratom of 15 December 1994. 4. APPROACH FOR RATIONALIZATION The following outlines a rationalised approach which the Commission intends to implement. It takes into account the CREST advice and, where appropriate, the principles of the SEM 2000 initiative. 4. 1 How to rationalize The CREST advice recognises that the complex evaluation requirements above can in practice be satisfied by a rationalized approach, while avoiding bureaucracy and evaluation for its own sake (see Table 1). Programme monitoring and evaluation requirements will be satisfied by two kinds of actions relating to the Framework Programme and its specific programmes: i) continuous monitoring, with appropriate assistance from independent external experts, involving annual reporting; and ii) five-year assessment mid-way through implementation. This assessment will be conducted by independent external experts and will incorporate the final evaluation under the previous Framework Programme. The final evaluation for any ongoing specific programmme will be incorporated in the subsequent five-year assessment. Thus, the final evaluation (Article 4. 3) will be independent from the five-year assessment (Article 4. 2) for any one programme. These two activities, monitoring and evaluation, are distinct and separate exercises. the Framework Programme, For the Commission's comments, will be communicated to the European Parliament, the Council and the Economic and Social Committee prior to presenting a proposal for the next Framework Programme (Article 4. 2). the five-year assessment, together with Moreover, the CREST advice suggests that the issues to be addressed will include: the consistency of the selection of projects with the objectives of the specific programme and the work programme; the extent to which selected projects or clusters of projects are fulfilling the wider policy objectives of the Community, in particular in areas of relevance to the specific programme concerned; the progress and output of projects against the original targets set; cases where the independent monitoring teams consider the results will have a important impact, or where poor performance requires further particularly examination; the efficiency and transparency of the management of the programme, including the development of calls, the assessment and selection process, contract negotiation and disbursement of funds, as well as internal Commission co-ordination; the use of special measures and support activities (e. g. to support SMEs, improve dissemination of information, etc); the changes that may be needed to the balance of the programme or to the strategy for implementation in the light of experience and changes in the wider environment. The CREST advice further suggests a set of input and output performance indicators for the monitoring and evaluation processes (see Point 5. 1 below). 4. 2 How to maximize benefit An integrated monitoring and evaluation approach, with the assistance of external experts, is outlined below where continuous monitoring reports will in time provide significant supplementary information to the annual report required under Article 130p of the Treaty on European Union7. The five-year assessment of programmes will provide a major input to discussions on future Community RTD activities. In this move to continuous monitoring and five-year assessment, results will be available at the time when decisions have to be taken and it will be possible to have a global overview of the state of implementation at regular intervals. In addition, evaluations will form a 'coherent ensemble'. Lessons from evaluations will be able to be drawn not only for specific programmes, but also for the Framework Programme and the whole Community S&T policy. Appropriate involvement of outside experts and bodies (notably CREST and Programme Committees), and openness of procedures, will enhance the independence and transparency of evaluations whilst recognizing the need to respect confidentiality of certain data. The external experts will discuss their reports with the Programme Managers and the Programme Committees. This process should facilitate implementation of the experts' recommendations. In addition, a new level of accountability is introduced through publishing the Commission's responses to recommendations produced by the five-year assessment panel of the Framework Programme. 5. EXTERNAL MONITORING 5. 1 Specific Programmes The. primary functions of external monitoring will be to assess the progress of programmes in order to assist Programme Managers in adapting specific programmes to developments, and to contribute to the five-year assessment of such programmes. Monitoring will be a quick response mechanism providing an annual picture of programme development. Monitoring will be based on continuous and systematic collection of data by Programme Managers about the progress and results achieved in the specific programmes. With the assistance of CREST, a set of suggested performance programme indicators have been identified which primarily relate to shared-cost programmes under Activity 1 of the fourth Framework Programme. The other specific programmes, such as the dissemination and optimization of results and the thermonuclear fusion, follow different implementation procedures for which different performance indicators may be appropriate. The Joint Research Centre (JRC) participation in specific programmes will be considered like any other participant, while the JRC direct-action activities will be reported on through the "Observations of the Board of Governors on the JRC Annual Report" which will directly constitute an input to the overall Framework Programme monitoring exercise. Treaty on European Union, Article 130p: "At the beginning of each year the Commission shall send a report to the European Parliament and the Council. The report shall include information on research and technological development activities and the dissemination of results during the previous year, and the work programme for the current year. " The relevant and available data for each specific programme contained in the performance indicators will be analyzed by a small panel of independent experts for each specific programme who would provide an external view. The monitoring panels will not duplicate the functions of Programme Committees, nor will they conflict with them, but add an extra dimension, i. e. of independent assessment concerning the overall performance and achievements of each specific programme. The monitoring panels will assess programme output against. programme objectives and against the progress in implementation. Each panel will present the results of its analysis to the relevant Programme Management which will take appropriate actions. 5. 2 Framework Programme The Framework Programme monitoring will assess the year-on-year implementation of the overall programme and will contribute to its five-year assessment. An independent external experts panel will review the outcome of examinations of the specific programmes and assess the overall progress in relation to objectives, priorities and progress in implementation of the entire Framework Programme. In assessing the whole Framework Programme, the contribution to the objective of strengthening the scientific and technological bases of Community industry and encouraging competitiveness at international level, as well as the contribution to the implementation of other Community policies shall be taken into consideration. The panel will present the results of its analysis to the Framework Programme Management which will take appropriate actions. 6. EXTERNAL FIVE-YEAR ASSESSMENTS OF SPECIFIC PROGRAMMES AND FRAMEWORK PROGRAMME For specific programmes and the entire Framework Programme, five-year assessments timed to provide input to discussions on. future legislative decisions on Community RTD activities, will be produced by panels of independent external experts. The annual monitoring reports, detailed above, will provide an input to the five-year assessment which will include a more strategic dimension. (For example, long-term projects which have been developed in successive Framework Programmes may be assessed over a longer time scale. ) These assessments will be ready when the proposal for the next Framework Programme is discussed. The five-year assessments, incorporating the final evaluations under the previous Framework Programme, will include three principal elements, each looking at distinct, but inter-related features: i) ii) assessing relevance, i. e. whether the initial objectives are still valid against evolving S&T, industrial and socio-economic conditions; assessing efficiency, i. e. whether the objectives have been pursued in a cost- effective manner; and, iii) assessing effectiveness, i. e. whether the initial objectives have been achieved: contributions to strengthening the scientific and technological bases and the competitiveness of Community industry, as well as contributions to all relevant Community policies. Special attention will be paid to the development of reliable methodologies for the evaluation of RTD project results. To this end, an ad hoc Working Group has been created which includes representatives from industry. It will recommend, with the help of external contributions, a common methodology to be used for the evaluation of results of RTD projects, particularly those with industrial relevance. The five-year assessments might pay particular attention to the coherence between Community and national S&T policies with a view to enhancing their mutual consistency8. For the specific programmes, as the five year assessment will incorporate the final evaluation of the previous specific programme, the assessment reports will be forwarded to the European Parliament, the Council and the Economic and Social Committee, as required in Article 4. 3 of the relevant Decisions. For the Framework Programme the five-year assessment report, accompanied by the Commission's comments, will be forwarded to the same bodies as above prior to presenting a proposal for the next Framework Programme, as required in Article 4. 2 of the relevant Decision. 7. IMPLEMENTATION The Commission is responsible for the monitoring and evaluation of Community RTD efforts. Appropriate consultation of CREST, Programme Committees and other relevant bodies will be sought. Independent external experts will be contracted to undertake the tasks of monitoring and five-year assessment and preparing reports. Monitoring panels for the specific programmes will normally comprise 3 experts with appropriate experience and stature: one from industry, one from academia, and one with experience of RTD programme evaluation. Monitoring panels for the Framework Programme will normally comprise 6-7 experts, the larger number reflecting the wider scope of their work. A similar balance of industrial, academic and programme evaluation experience will be sought. For the five-year assessments, the panels, while maintaining a minimum number of experts, will be slightly larger than for monitoring. The size of the panels for the specific programme assessments will be determined in relation to the objectives and scope of the respective programmes. A panel of 6-8 experts will be required for the Framework Programme assessment. The five-year assessment panels might include a few experts from the related monitoring panels to ensure continuity between the two different exercises. The experts will be drawn from a variety of backgrounds, appropriate to the programme under consideration, and will be expected to have extensive experience in order to fully examine horizontal issues. Special attention will be paid to ensure the coherence of monitoring and five-year assessment and to maintain the highest possible degree of transparency for the exercise. Regular reporting will be made once a year, correlated with the publication of the Annual Report requested under Article 130p of the Treaty on European Union. Treaty on European Union, Article I30h: "The Community and the Member States shall co-ordinate their research and technological development activities so as to ensure that national policies and Community policy are mutually consistent". 8. TIMING Every effort will be made to implement this scheme as quickly as possible. The 1995 monitoring exercise has been completed. The external experts' reports on the specific programmes and on the overall Framework Programme are available. In respect of the forthcoming five-year assessment, the first panels will be expected to start their work during Spring 1996 with specific programme reports expected by Summer/Autumn 1996 and the report on the overall Framework Programme before the end of the year. The rationalized approach, suggested above, will be reviewed after three years of implementation. 9. CONCLUSIONS The series of regular annual monitoring reports and five-year assessment reports will provide a comprehensive overview of the management of programmes (Framework Programme and specific programmes), their implementation and benefits derived. The involvement of external experts and the openness of procedures will enhance the independence and transparency of monitoring and evaluation. The publication of Commission's responses to recommendations produced by the five-year assessment panel of the Framework Programme will introduce a new level of accountability. Meetings between the experts and Programme Managers and the Programme Committees on the results of monitoring and evaluation will ensure mutual understanding and facilitate the implementation of appropriate recommendations. All these features will further enhance the credibility of Community RTD evaluation efforts. The five-year assessment reports will be given to the European Parliament, the Council and the Economic and Social Committee thus providing input to future legislative decisions on Community RTD activities. Through this Communication, the Commission invites the Council and the European Parliament to take note of this rationalized approach which takes account of the current legislative requirements for monitoring and evaluation. TABLE 1: EXISTING SITUATION vs. SUGGESTED SCHEME 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 Y E AR FP 3 : 1990-1994 1. LEGISLATIVE OBLIGATIONS: FP3 Specific programmes Review ! FP3 2) FP4 State of implementation Specific programmes f { FP4 { 2. SUGGESTED SCHEME: FP 4 : 1994-1998 FP 5 : 1998-2002 Final evaluation Final evaluation 5 year assessment h-L Final evaluation Continuous monitoring State of 5 year implementation I assessment n (*) « (*> (*> , Continuous monitoring, reporting at the beginning of each year (*) 5-year assessment, (incl. final evaluation of FP3) approx. period covered: 1991-1995 5-year assessment (incl. final evaluation of FP4) approx. period covered: 1995-1999 JFP3 + F P4 | (+ F P5 +. '. ) { Specificprogramine, j (*) (*) (*) (*) (*) (*) Continuous monitoring, reporting at the beginning of each year (*) Note : arrows indicate the approximate periods during which evaluation reports should be issued. ISSN 0254-1475 COM(96) 220 final DOCUMENTS EN 15 Catalogue number : CB-CO-96-231-EN-C ISBN 92-78-04209-9 Office for Official Publications of the European Communities L-2985 Luxembourg