Evaluating socio-economic programmes : volume 1, Evaluation design and management.
Evaluating socio-economic programmes Evaluation design and management EC Structural Funds Evaluating socio-economic programmes Evaluation design and management Volume 1 European Commission MEANS is a programme of the European Commission. It is aimed at improving Methods of Evaluating Structural Policies. The MEANS Programme and Collection was supervised by Phillipe Goybet and Miguel Angel Benito Alonso, Unit for the Coordination of Evaluation in the Directorate for Coordination and Evaluation of Operations of DG XVI Coordination-evaluation@fmb. dg16. cec. be The realisation of the Programme was entrusted to the Centre for European Evaluation Expertise (C3E) under the direction of Eric Monnier and Jacques Toulemonde. Responsibility for the opinions expressed in this boon lies solely with the authors of the text: C3E - 13 bis Place Jules Ferry - F - 69006 Lyon - France - c3e@c3e. fr A great deal of additional information on the European Union is available on the Internet. It can be accessed through the Europa server (http://europa. eu. int) and on the Inforegio Website (http://inforegio. cec. eu. int). Cataloguing data can be found at the end of this publication. Luxembourg: Office for Official Publications of the European Communities, 1999 ISBN 92-828-6626-2 © European Communities, 1999 Reproduction is authorized, except for commercial purposes, provided the source is acknowledged. PRINTED ON WHITE CHLORINE-FREE PAPER Printed in Italy Foreword is directly Developing good evaluation practices has become a priority area for the relevant Commission and to the effective use of Community resources. Indeed, the sound financial management principles which the Commission seeks to promote via its SEM2000 (Sound and Efficient Management) Programme rely, specifically, on the regular evaluation of the Community activities. The Directorates General responsible for structural policy have considerable expe­ rience in this area. Since 1988 they have in the process of made evaluation an integral part of each stage Community Structural intervention. Thus, the programme development stage includes a prospective evaluation; the implementation stage a strengthening of the monitoring systems and, at the end of the intervention, evaluation establishes what has been achieved in terms of programme impacts. However, the capacity of evaluation to contribute to the effectiveness and transparency of structural interventions rests ultimately on the integrity of the methods adopted to verify the credibility of results. This is why, four years ago, on the initiative for the Directorate General for Regional Policy and Cohesion (DG XVI), the MEANS programme was initiated as a response to the demand for developing evaluation methodo­ include practices combining both general scientific logy. This also requirements and the specific requirements of Community interventions whilst, at the same time, remaining accessible to a large audience. One of the more tangible results of the MEANS programme has been the publication of this six volume set by a group of independent experts, focu­ sing on the principal elements of socio-economic programme evaluation. A short summary of this work can be found at the beginning of the present volume. This collection covers the various methodological approaches and innovations that have taken place ¡n the context of structural funds eva­ luation. It will enable those responsible for evaluation to develop their approach in a flexible and independent manner and for the various programme partners to exercise greater control over the evaluation pro­ cess. Attending the conferences of evaluation specialists and "clients" convened in Brussels (1995), Berlin (1996) and Seville (1998), I gained some impression of the interest that new evaluation techniques can gene­ rate. I would stress that the development of evaluation needs to progress in tandem with efforts to incorporate evaluation findings in the decision making process. This implies evaluation as a means of establishing whe­ ther policies are equitable, effective and relevant to the identified needs. In satisfying these requirements, evaluation should be seen as a valuable instrument affording both Member State and Community authorities the necessary guarantees that Community funds are being deployed in the best interests of the EU citizen. This is a cost worth paying when the goals of Economic and Social Cohesion and the European Union enlargement are at stake. Finally, I wish to extend my thanks to the members of the Committee of Independent Experts established at the launch of the programme. Their expertise and standing have contributed greatly to the quality and relevan­ ce of the MEANS programme. I therefore thank: E. Chelimsky (USA), ex- President of the "American Evaluation Association"; K. Kennedy (IRL), Director of the "Economic and Social Research Institute" (ESRI); J. R. Cuadrado Roura (SP), President of the "European Regional Science Association"; C. Seibel (F), Director of the "Statistiques démographiques et sociales de l'Institut National de la Statistique et des Etudes Economiques" (INSEE); L. Senn (I), Director of the Department of Economie Regional of the University Bocconi; E. Stern (U. K. ), Director of the unit of evaluation of the Tavistock Institute; A. Sorber (t) (NL), Head of the "Policy Analysis Department of the Ministry of Finance"; L. I. Strom (SV), Statens Institut för Regionalforsking (SIR); L. Tsoukalis (GR. ), professor at the University of Athens; H. Wollmann (D), professor at the University of Humboldt - Berlin and founder member of the "European Evaluation Society". E. Landaburu : Contents The MEANS Collection Presentation of Volume 1 5 11 I Evaluation issues within the framework of the Structural Funds 15 The value added of evaluation The specificity of programmes and their evaluations Development of practice within the framework of European Structural Funds II Preparing an evaluation and defining its content Defining the general framework of an evaluation Deciding to launch an evaluation Ordering an evaluation III Performing an evaluation Analysing the logic and content of the programme Evaluating the impacts of an entire programme Analysing macro-economic impacts Carrying out more in-depth analyses IV Using an evaluation Optimising the use of an evaluation Assessing the quality of an evaluation V Guaranteeing the success of an evaluation Better preparation of an evaluation Opening evaluation to more partners Consolidating evaluation techniques Annexes 17 27 35 49 51 65 75 85 87 109 135 147 159 161 169 181 183 189 197 203 Annex 1 - Process of ex ante, mid-term and ex post evaluation 205 211 Annex 2 - Guide for drawing up the terms of reference 217 Annex 3 - Bibliographic references Index 219 The MEANS Collection in For several years the European Commission has been training and research, development, investing communication, with a view to improving the quality and utility of evaluations performed within the framework of economic and social cohesion policy. This work has taken place within a global programme entitled MEANS: Means for Evaluating Actions of a Structural Nature, initiated by the Directorate General XVI (Regional Policies and Cohesion). The MEANS programme is justified by the importance of evaluation in a domain which accounts for about one third of the Union's budget. The aim of the programme is to devise a coherent set of methods for addressing a wide range of evaluation problems encountered in the field of Structural Fund interventions. The also training seminars and The outputs of the MEANS programme comprise a body of methodological experiences, ideas and innovations which, until now, were presented in seven 50-60-page handbooks. published programme methodological reports and quarterly newsletters, and organised three European conferences (the most recent, in Seville, was attended by just under 600 practitioners involved in the evaluation of the Structural Funds). An effort was made to take full advantage of existing experience and recent innovations, particularly in the following areas: measurement of employment effects; development of systems of indicators; and evaluation of macro-economic effects. Initiation and training in evaluation practices was organised throughout the Commission in the Structural Funds sector. Through the evaluations initiated at the European level and the work that has gone into building on these experiences, sufficient knowledge has now been to produce a comprehensive publication acquired Evaluation design and management accessible to a wide public. The result is a collection covering all aspects of the evaluation of structural policies, and which takes full advantage of existing MEANS documents, revised and comprehensive to guarantee their coherence. It was decided to design the Collection without referring too directly to the regulations governing European cohesion policy for the period 1994-1999. Experience had shown that the regulatory framework changes frequently whereas technical problems and professional standards change very little. With a view to ensuring the relevance over time of the Collection, it was intentionally decided to focus on "socio-economic development programmes". This term, considered in its broadest sense, applies to interventions financed within the framework of European cohesion policy and to all other similar interventions which share the same essential characteristics, i. e. : a development aim, multiple objectives, a partnership dimension and a multi-sectoral content. In fact, the evaluation of such programmes requires special methods that are not readily found in the existing literature. This is the main justification for the MEANS Collection. An aid for those who manage and perform evaluations The Collection is intended for two types of readers: those involved in managing evaluations and those involved in performing them. Readers in the first category participate in the entire evaluation process, including decisions taken before and after the actual evaluation work. They are primarily local, national and European officials (in the latter case, desk officers of the geographic units at the Commission) responsible for managing programmes financed by the European Union. The Collection must offer them a complete set of information on the state of the art, particularly when they lack know-how in this domain, so that they are able fulfil their missions. The The MEANS Collection work of the MEANS programme has nevertheless led to more general methodological progress as well, which is likely to interest those responsible for commissioning evaluations in other domains. The second category of readers for whom this collection is intended consists of teams responsible for carrying out the actual evaluation work: public research centres, university firms, independent consultants and evaluation units within administrations. laboratories, private consultancy This Collection is, moreover, likely to interest teachers and students of academic programmes on regional economies, economic policy, employment policy, public management, etc. However, it does not have an academic vocation. In particular, the bibliography, limited to a few reference texts, is not intended to be exhaustive. General survey of evaluation in six volumes The complete Collection consists of six volumes designed to complement each other and to provide essential elements for professionals to perfect their evaluation practice autonomously. Moreover, each book is organised in such a way that it can be read and understood independently from the other volumes in the Collection. Apart from the present volume, the Collection is organised on the following lines: The second Volume covers the selection and use of indicators for monitoring and evaluation. It shows how indicators can be used to back political decisions, to reinforce management and to measure global performance of programmes. Volume 2 proposes a methodological framework, numerous examples of indicators, and a guide for assessing the quality of the indicators. Its purpose is to improve practices in terms of quantification of indicators and comparisons between programmes. Evaluation design and management The third Volume presents the principal evaluation techniques and tools adapted to socio-economic programmes. These tools are taken from six or seven different scientific disciplines: economics; sociology; political science; management; geography; etc. It is, in a sense, an encyclopaedia of tools presented in the form of information sheets. Each sheet consists of a general description of the technique, the principles of its use, a list of strengths and weaknesses, an example of its application, and a bibliography of references. Apart from giving an idea of the possible uses of each tool taken on its own, the document includes a guide for choosing tools and for their combination, in relation to the questions asked and the context. To our knowledge, there is no other multidisciplinary guide of this nature in the existing literature. It is certainly one of the most original contributions of this Collection. Technical solutions for evaluation within a partnership framework are the subject of the fourth Volume. The need for this book stems from the fact that socio-economic development programmes are unusual objects of evaluation. First, they always involve several public authorities within a partnership framework which recognises the existence of legitimately different points of view. They generally have numerous objectives which are not always clear and coherent enough to form a basis for evaluation. Secondly, the programmes concern multiple sectors, which increases the observation and analysis work considerably and complicates the writing of a the multiple components of the programme. Finally, since each public authority its citizens, each administrative level introduces into evaluation its own criteria and priorities. No complete solution to these problems can be found in existing evaluation manuals. Thanks to a number of pilot evaluations, the MEANS programme has tested and perfected four techniques which, used together, offer a complete methodological solution. The volume provides instructions for using transversal synthesis, covering is accountable to The MEANS Collection these four techniques: impact mapping, the cross- impacts matrix, the scoring card and multicriteria analysis. The fifth Volume deals with the transversal evaluation of impacts on the environment, employment and other intervention priorities. The evaluation of programmes financed by the Structural Funds cannot be limited to the examination of impacts aimed at directly; it has to analyse the extent to which European priorities as diverse as employment impacts, the maintenance of jobs, equal opportunities between men and women, the environment, the competitiveness of SMEs, etc. , have been taken into account. These priorities, or priority themes, are often referred to in operational documents without, however, being the object of mainstreaming. It is acknowledged that the evaluation of such transversal impacts, known as thematic evaluation, is difficult to achieve and that the fact of there being indirect impacts compounds the problem. This volume provides a methodological framework for identifying the potential into impacts of a programme, accountóincluding for measures which are not referred toóand identifying and defining appropriate indicators for empirically monitoring and analysing their effects on the publics concerned. It offers the elements required for the analysis and choice of the best methodological solutions for thematic evaluations. taking them Finally, the last Volume closes the Collection with a glossary of 300 concepts and technical terms. The writing of the Collection required an extremely rigorous effort to homogenise and define the terms used, and to write a specific index for each volume. The glossary draws together several hundred terms taken from the different indexes, with definitions, translations and descriptions. It meets a genuine need in so far as the quality of an evaluation partly depends on the quality of communication between the partners who commission it, and between the commissioners and the evaluation Evaluation design and management team. In the case of European economic and social cohesion, interaction is made more difficult because the evaluation is performed in an international and inter- sectoral context involving people whose mother tongues and professional references differ. Moreover, the recent development of evaluation means that many actors are unfamiliar with standard evaluation vocabulary which, in turn, has not always been firmly established. The resulting difficulties of communication can seriously compromise the quality and utility of evaluation work. This Volume broadly covers all the subjects addressed in the rest of the Collection. In the numerous cases where several definitions exist at the Commission and/or in the scientific community, a single, clear choice is made and the main alternative definitions prevailing in other sectoral or national contexts are given. Presentation of Volume 1 This first Volume in the MEANS collection serves as a "entry" both into the Collection and into the evaluation of socio-economic programmes. In addition to the issues traditionally considered in currently available evaluation guides (General Accounting Office, 1991; European Commission, 1997; Conseil Scientifique de l'Evaluation, 1994; see bibliography in Annex 3), it presents and treats in an operational manner, questions specific to socio-economic programmes, such as interaction between the different management levels (project, programme, policy), partnership between several government levels, overall evaluation of complex programmes and in-depth analyses of actions or specific themes. This book helps readers to discover and understand the implications and value of evaluation. At the same time, it shows what can and cannot be expected from an evaluation at a particular time and in a specific context. It provides the elements required for steering an evaluation from A to Z: defining responsibilities, specifying the scope and objectives of the evaluation, outlining the method, schedule and budget; assessing the quality of the work and, finally, supervising the use of evaluation conclusions. It is applicable, generally, to evaluations performed before, during and after a programme, that is to say, for the prospective analysis of needs, strategies and expected effects, and for the retrospective analysis of impacts observed in the field. The modes of organisation proposed are general in scope and their aim is not limited to current organisational schemas which are bound to evolve during the new generation of post- 1999 programmes. This first Volume, which is to serve as a general introduction to the evaluation of socio-economic development programmes and as a synthesis of the entire Collection, is intended for a less specialised public than the following volumes which are more technical and intended for more informed readers. This Volume has been set out so that readers may have easy access to information corresponding to their own specific interests. Box 1 and Box 2 propose several approaches which can be used to go directly to the most relevant sections. Evaluation design and management Box 1 - For those who wish to know more about evaluation by going directly to the essential For a rapid overview: > Scan the messages in italics heading each chapter, which briefly summarise the key points in the book - start on page 27 For the reader who is relatively unfamiliar with European structural interventions: > Carefully read Part I, "Evaluation issues within the framework of the Structural Funds" For the reader who has to commission an evaluation: >■ Move directly on to Part II, "Preparing an evaluation and defining its content" For readers who design, perform or supervise the collection and analysis of data: >· Read, in particular, Part III, "Performing an evaluation" For getting the most out of your evaluation: >· Advice and recommendations are presented in Parts II and IV For ideas on perfecting evaluation practice: >■ See the suggestions and recommendations made in the conclusion to the Volume, in Part V Presentation of Volume 1 Box 2 - For those who have to design and perform an evaluation Ex-ante evaluation: > Consult annex 1 - The evaluation process - column "ex ante" > Follow the symbols -^ heading certain sections or paragraphs Intermediate evaluation: >■ Consult annex 1 - The evaluation process - column "mid-term" > Follow the symbols M heading certain sections or paragraphs Ex-post evaluation: > Consult annex 1 - The evaluation process - column "ex post" > Follow the symbols ► heading certain sections or paragraphs I Evaluation issues within the framework of the Structural Funds The value added of evaluation The evaluation of public intervention consists of judging its value in relation to explicit criteria, and on the basis of information that has been specially gathered and analysed. Beyond this general, and necessarily, abstract definition, the growth of evaluation practice within the countries of the European Union has taken a number of forms: managerial evaluation aimed at improving management, democratic evaluation used for accounting to citizens and pluralistic evaluation that tries to bring about agreement between the partners involved in public action. These three concepts are to be found within the experience of evaluating the European Union's policies, with one of them predominating according to whether the evaluation concerned is ex ante, ex post or intermediate. ■ A recent approach, specific to the public sector The purpose of evaluation is to check the raison d'être of a public intervention, to confirm both reproducible success stories and failures not to be repeated, and to report back to citizens. Whereas private sector organisations have little need to justify or account for their actions, since they are judged by the market, public sector organisations are obliged to evaluate. Origins of evaluation Evaluation is related to the analysis of public actions and, more generally, to the social sciences. It systematically uses the methods of economics and sociology, but also those of social psychology, economic geography, anthropology, etc. As a tool for enhancing knowledge it developed along with the regular collection of socio-demographic and economic data (e. g. public records and census). By contrast, the evaluation of programmes as a decision-making aid is more recent. It appeared in the 1950s, when in the USA the effectiveness and neutrality of the administration were being called into question. At the time, this was part of the effort to rationalise resource planning and allocation. The PPBS (Planning, Programming, Budgeting System) was born and spread to most European countries (see Rist, 1990, for an historical analysis of evaluation). In the USA, evaluations started to be performed on a wide scale in the 1960s. They focused primarily on innovations in the fields of education, Evaluation design and management social services, public hygiene and crime prevention. For the U. S. scientific community many social programmes became real laboratories which were expected to produce knowledge and to help improve the actions of public authorities. to become aware of the restrictive and artificial nature of the "social experiment" type of approach and of the PPBS norm. In the 1970s, however, administrative officials started From then on evaluation was seen more as a way of managing necessary budgetary restrictions. This was the case in the UK, where the development of evaluation was linked to the concern to limit public spending following the public service reforms in the 1980s. Evaluation became part of a new, more decentralised and responsible concept of public management. It is the reflection of several concerns which can be defined in terms of three main objectives. First, there is the need to estimate as objectively as possible the effects of policies or programmes on society, and to understand the logic that governs how they function (cognitive aim). Secondly, there is the desire to help policy-makers and stakeholders to judge the value of these programmes (normative aim). Thirdly, there is the need to contribute in a practical way to the improvement of programmes by informing different protagonists on the meaning, conditions and consequences of their actions and decisions (instrumental aim). Evaluation brings together existing information in a coherent whole. Where relevant, it produces new information for enhancing programme managers' knowledge about the social and economic context in which their public interventions are situated. Moreover, evaluation is an instrument at the service of strategic and operational decision-making. Its practical usefulness can be defined as the linking up of objectives, resources, results and effects of interventions, with a view to improving the efficiency and effectiveness of a programme. Finally, evaluation introduces the notion of value and, a fortiori, of value judgement, into administrative practices. This dimension, which can be qualified as "political", is what makes evaluation so original. It implies the emergence of a new profession in its own right, distinct from others such as research. A necessity for the public sector The need for evaluation in the public sector can be understood, especially at the ex post phase when efforts are made to measure the impacts of public Evaluation issues within the framework of the Structural Funds activity, by comparing it to the role of the market for a private enterprise selling goods and services. By deciding either to buy these products or to go to a competitor, each customer calculates all the benefits, costs and disadvantages to be derived from the purchase in the short or long term. Customers' buying behaviour therefore indicates what the firm should or should not do. Finally, the firm does not need to justify its existence. It survives if it is financially viable, that is to say, if it is useful to its customers. By analogy with the private firm, a public organisation also produces goods or services. It offers (and sometimes imposes) them, not to customers but to addressees, users, etc. If the people participating in a public action are satisfied with it, that is generally not enough to say that the action is successful. On the contrary, public authorities often try to produce distant and global effects, such as sustainable socio-economic development, which are not synonymous with addressee satisfaction. Evaluation makes it possible to know and to judge all the effects of a given public action: direct and indirect, positive and negative. The sanction of the electorate is certainly an important signal for any government, but its main disadvantage is the non-discriminatory nature of the judgement thus expressed. Only direct democracy as practised in Switzerland allows, to a certain extent, for the expression of a judgement on each intervention (evaluation is no less advanced in Switzerland!). Because the democracies represented in the European Union are essentially indirect; evaluation plays a crucial role, both as a signal to show what should and should not be done, and as an instrument capable of justifying the existence of public action in the eyes of citizens. The development of evaluation in Europe Different conceptions in the European countries The development of evaluation in Europe started in the northern countries (UK, Sweden, Denmark, and the Netherlands), where it has now become standard practice in the administrations. All the European countries are concerned, and have adapted evaluation to suit their own cultural attitudes to the role of government and the relationships between their legislative and executive powers. The major policy decisions taken within European countries can be classified in three main categories. To the two traditional lines that can be Evaluation design and management qualified as "managerial" and "democratic", has been added a third that can be referred to as "pluralistic". These three directions or conceptions are different without being mutually exclusive. Box 3 describes them briefly by showing the principles and recommendations characterising each one (Duran, 1992). Box 3­ Three conceptions of < ¡valuation Managerial Democratic Pluralistic Optimisation of public resources Involve citizens and report to them Collective problem solving Global project Valuing Independent « Evaluators » (experts) Standards Objectiveness Judgement by elected representatives Publicising the approach Judgement by those involved in the action Impartiality The general idea of managerial evaluation is to ensure that limited public revenue is efficiently managed. The underlying question can be formulated as follows: how can trade­offs between the different sectoral policies be justified? The approach preferred here is that of "optimisation" of existing budgets, within administrations. The political dimension of the evaluation act is disregarded. The "evaluator", in official terminology, is above all a methodologist whose legitimacy is grounded in his or her scientific knowledge and approach. The deontology is based on two principles: the independence of the evaluator and the objectiveness of the method. The reference to scientific rigour constitutes, in a sense, the guarantee of the evaluation's independence vis­ à­vis policy. Example: The British " Value for Money " approach During the 1980s, many functions of the British administration were privatised or delegated to semi­independent agencies. At the same time, demands for the efficiency of these services grew, both at the central administrative level and among citizens. This tendency seemed even more obvious during this period within the framework of ε managerial culture in the administration. This culture was based on "Value For Money" evaluations and on the publication, in the "Green Book" by the Treasury, of technical guidelines for this type of evaluation. Evaluation issues within the framework of the Structural Funds The second, "democratic", conception shows the growing awareness of the crisis in traditional systems of representation (turning away from political parties, trade unions, etc. ). In this conception, public authorities must be fully accountable for their actions, to the citizens. The emphasis is thus on the need for evaluation to strengthen counter-powers to the administration and, more generally, the executive, accused of monopolising expertise. Evaluation, defined as a "judgement of the value of public policies", must feed into the democratic debate. It is recommended that these evaluations be managed by evaluation authorities consisting of democratically elected persons. Evaluation professionals are at the service of such committees but for technical issues only. The process must be publicised to ensure its democratic nature. From a general point of view, this model seems to be developing, particularly in a context of modernisation of legislative and political practices, where keen interest is currently shown in the notion of "guideline laws" and "sunset legislation". These are laws which need to be confirmed, by vote, in view of the results obtained after the first few years of application. Example: Consensus conferences in Denmark Among various types of democracy, a Danish practice has become a reference. "Public Consensus Conferences" have been part of the political scene in Denmark for over ten years. The Danish Board of Technology calls on ordinary citizens to express themselves and to debate with experts and representatives of various interests, on particularly controversial subjects (for example, genetic manipulation of animals, tests likely to be carried out during recruitment, or the signing of insurance contracts). These conferences, spread over three weekends, enable citizens to express themselves and to undertake a dialogue with experts. The reports produced in this way aim at informing public opinion and facilitating democratic debate. Without replacing political decision-making, they lay the foundations of debate between the executive and the legislative powers and become part of the decision-making process. The last type of evaluation is called "pluralistic evaluation", "participative evaluation" or, in the American literature, "stakeholder evaluation". This approach tries to find a compromise between the technical requirements justifying the first conception and the political requirements inspiring the second. In this conception, evaluation is part of a collective problem-solving process. On the basis of reliable information acceptable to all, value judgements are formulated by seeking consensus within a pluralistic evaluation authority Evaluation design and management composed of political and administrative officials, as well as spokesperson for the publics concerned. the Example: Pluralistic authorities for evaluation In France, ad hoc evaluation authorities are often formed to conduct evaluation work. Thus, within the framework of national evaluation of the prevention of natural hazards, the evaluation authority was composed of actors of diverse origins: ministries, local authorities, public agencies, community groups, etc. Within the framework of policies co-financed by the state and the regions, certain evaluations are steered jointly by political and administrative officials at both levels, along with representatives of the publics concerned. the 1950s. So-called A comparison can be drawn between the different conceptions stated above, and the "four generations" identified by publications in the U. S. These refer to the different stages of development of practices in this country since first generation evaluations concentrated on the description of the effects on society of public interventions. The second generation focused on an analysis of relations of causality and on the attribution of effects to interventions. The third generation started systematically to verify the achievement of these aims, which makes it very similar to managerial evaluation described above. Finally, fourth generation evaluations have developed in a pluralistic mode, that is to say, by associating the different protagonists with the evaluative approach (Guba, Lincoln 1989). Evaluation procedures at the Commission Evaluation at the European Commission dates back to the 1980s, in the fields of research and aid to developing countries. From the early 1980s European science and technology policy was based on a system of pluri- annual programming, and evaluation was rapidly the management of these programmes. A specific programme was, moreover, set up (Monitor, 1989-1992) to improve methods and indicators used in evaluating research. Evaluation in the field of aid to developing countries also increased significantly during the 1980s. The most commonly used approach was that of Project Cycle Management, partly discussed in this book (see page 148). Evaluation practice later extended to other fields, particularly that of the Structural Funds, with the establishment in 1988 of specific regulations on the monitoring and evaluation of programmes. included in Evaluation issues within the framework of the Structural Funds In 1991, a little over half the Commission's Directorates had an evaluation unit. This unequal development can be explained by the birth of European policies. Some policies were a priori legitimate, that is to say, recognised by the Treaties as the exclusive competence of European institutions (e. g. foreign trade, competition, and agricultural market policies). These policies were scarcely evaluated until very recently. The few existing evaluations have all been carried out internally. Thus, in the case of competition policy, for example, evaluation was for a long time carried out by European officials who relied on consultation with the industrialists concerned and on informal meetings with national administrations. The Commission, in a strong position because of the legal texts defining its competence in these areas, is able to conduct its own evaluations. interventions has been In other areas, the legitimacy of European constructed progressively over the years. These concern policies that have traditionally been the province of national authorities (e. g. scientific research, consumer rights, transport networks, etc. ). Despite the growing recognition of European initiatives, these policies still have to be justified. Interventions are recognised as being necessary, provided that the Commission can prove "the advantages gained from implementation of the programme, throughout the Community". In order to do so, it commissions numerous evaluations. Thus, the evaluation of research programmes is aimed at legitimising European policy, particularly vis-à-vis the Member States. Use is made of expert committees that are known to be independent and are recruited from the different countries. In the field of political cohesion, evaluation was born in 1988 and has since developed steadily. Evaluations are sometimes similar to the pluralistic conception described above, in so far as some of them involve European, national and regional officials, as well as socio-economic partners. This without being too removed from the other conceptions of evaluation. Moreover, on its own initiative (unilateral order), the Commission carries out thematic evaluations or evaluations of transversal programmes across several countries. Part I of this Volume features a detailed analysis of the evaluation of Structural Funds. Finally, in 1995 the Commission launched a programme aimed at improving its financial management: "Sound and Efficient Management - SEM 2000". This programme generalises evaluation, to include all policies which have an effect on its budget. It institutionalises the use of the managerial model in improving management at the of evaluation as a key element Evaluation design and management Commission. The Directorates are requested to establish on-going evaluation programmes and to create evaluation units. Support mechanisms have also been set up (network of evaluation services, manuals, training, data base of past evaluations and pilot projects). Any renewal of a pluri-annual programme is now subject to evaluation of its previous results. ■ Three good reasons to evaluate Verifying that public action replies to uncovered or insufficiently satisfied needs One of the aims of evaluation is to verify the raison d'être of public action. A public intervention can be justified only if it is intended to satisfy needs or solve socio-economic problems. Evaluation serves to clarify these needs or to verify the real existence of these problems, especially if it is ex ante. When it is ex post it serves to examine whether the needs or problems still exist. Furthermore, it is essential to rule on the appropriateness of the public intervention as such. A public programme can be justified only if the needs cannot be met by private initiative or by any other existing public initiatives. Thus, the inadequate nature of private initiative constitutes the first motive for public intervention, by virtue of the principle of subsidiarity, in the original sense of the term, meaning that public authorities act only when the private sector fails to do so. More particularly, for European institutions this concept of subsidiarity takes on additional meaning in so far as the legitimacy of the European Union must also be justified in relation to national sovereignty. The Maastricht Treaty reaffirms the importance of respecting subsidiarity in relation to the national level: "the Community acts when an objective can be met more satisfactorily at the European level than at the level of Member States taken in isolation". Improving interventions By observing the reality of results and impacts, the evaluation exercise provides an opportunity for feedback arising from actions underway. Certain lessons drawn from the experience lead to proposed improvements in the implementation of evaluation, without necessarily changing the basic Evaluation issues within the framework of the Structural Funds assumptions. For example, the budget for the least successful actions will be reduced. This is a case of direct feedback or simple learning (called a "single loop learning"). Evaluation may also propose strategic reorientation, for example by re-organising the implementation of actions so that they succeed better. In that case evaluation produces indirect feedback or "double loop" learning, in so far as it challenges the logic of the intervention. is commonly called the This evaluation aim is what instrumental contribution, that is to say, concrete assistance in terms of solutions to the problems encountered or adaptation to an unstable environment. Its function is qualified as formative, in the sense that it has the merit of providing all the elements required by the operational officials to be able to improve the programme: more rational distribution of human and financial resources, improved management of services responsible for implementing them, etc. The goal is to analyse the factors of success or failure, in order to help managers adjust their modalities of intervention and implementation of the programme. Accountability The third aim of evaluation consists of reporting to political authorities and citizens on results obtained and on the sound use of allocated resources. This aim is called responsibility, liability or accountability. The duties of the public authorities are the reverse side of their right to levy taxes and their power to compel. Evaluation then takes on its full political dimension in the functioning of democracy. The Commission, although not politically accountable to the European electorate, has progressively asserted a demand for transparency, vis-à-vis taxpayers, in the use of the funds it manages. Evaluation must help to explain in simple terms where public money was spent, what effects it produced and how the spending was justified. The addressees of this type of evaluation are obviously political officials and in fine, most often via the media, the citizens. An essential notion within the framework of European evaluations is that of "Community value added"; in other words, is the support given by the intervention of European institutions justified in terms of the progress made? The specificity of programmes and their evaluations Socio-economic development programmes must be evaluated by means of methods adapted these characteristics is that they are co-financed and involve joint decision­ making. This implies that evaluation must be carried out with the participation of several partners and must meet their different expectations. their specific characteristics. One of to These programmes are also characterised by multiple objectives and domains of intervention. The result is greater complexity of evaluation work. Finally, unlike those contexts in which evaluation is considered to be a purely retrospective exercise, the system chosen for European cohesion policy consists of programmes at three successive stages: ex ante, intermediate and ex post. Evaluation of a set of combined objectives From the outset, the purpose of European economic and social cohesion policy was to reduce economic and social disparities between regions in Europe and between their inhabitants. In the early 1990s, differences in levels of development and standards of living throughout Europe were still marked. The per capita Gross Domestic Product of the ten poorest regions in the European Union was still 3. 5 times lower than that of the richest regions. The unemployment rate of the ten regions with the lowest employment levels was as much as seven times higher than that of the ten regions with the highest employment levels. The reduction of these disparities, which cannot be achieved solely by free market forces, is a challenge for public interventions in the very long term. The interventions of the European Union reinforce those of each Member State by helping to enhance European solidarity. The objective of economic and social cohesion was introduced in 1986 in the Single European Act, triggering the 1988 reform which included a substantial increase in resources allocated to its financial instrument, the Structural Funds: ERDF, ESF, EAGGF and, more recently, FIFO The European Union Treaty, ratified in 1993, reinforced this orientation placing economic and social cohesion among the Union's main objectives with, as a backdrop, the objective of "economic convergence", so essential for the achievement of a Single Market but also of the economic and monetary Union. Evaluation design and management The long-term global objective of cohesion in the European Union can be broken down into several general objectives such as solidarity and mutual support. These, in turn, are expressed in concrete and measurable economic terms. Economic and social cohesion is officially defined in Article 130a of the European Union Treaty in terms of "harmonious development", with a specific geographical dimension: the lesser-developed regions, including rural areas, should be helped to catch up with the others. Moreover, this aim includes support to the most underprivileged social groups, particularly the unemployed and people whose jobs are threatened. Acting together, the Structural Funds have, until 1999, been concentrated on seven objectives. Objectives 1, 2, 5b and 6, all specifically regional, concern: the structural adjustment of lagging regions (Objective 1); economic conversion of declining industrial areas (Objective 2); economic diversification of vulnerable rural areas (Objective 5b) and structural adjustment of regions with very low population density (Objective 6). Objectives 3, 4 and 5a, covering the entire Community, aim at fighting long- term unemployment, the integration of young people into the professional world and people threatened with exclusion the job market (Objective 3), the adaptation of workers to changes in industry and systems of production (Objective 4), and the adaptation of agricultural structures and fishing (Objective 5a). from In order to enhance the visibility and effectiveness of the Structural Funds, the Commission has proposed (in Agenda 2000) to reduce these seven objectives to three: two regional objectives (lagging regions and regions undergoing economic and social reconversion) and a third, horizontal objective devoted to human resources. The global objective of cohesion and reduced disparities corresponds to several priorities defined by the Member States and the Council. These priorities must be respected within the framework of the objectives of each of the programmes and, in fine, in the selection criteria of the financed projects. These priorities correspond to the four transversal challenges which are: protection of the environment, competitiveness, job creation and equal opportunities between the sexes. Volume 5 of this Collection is devoted to taking these priorities into account within an evaluation. Consequently, the evaluation of programmes must simultaneously analyse several long-term objectives and an even larger number of medium-term objectives, without forgetting to supply information on the merits of the Evaluation issues within the framework of the Structural Funds programme in relation to the above-mentioned transversal policy priorities. Administrative officials and, to an even greater extent, evaluation teams, have to deal with considerable technical difficulties because of these inter­ linking objectives, resulting in multiple objectives that are difficult to observe and to analyse in detail. Evaluation of interventions in multiple domains With the strengthening of the process of European integration, the areas in which the Member States have decided to pursue a common policy have increased. Some specifically target enhanced cohesion while others are supposed to contribute to it, with their own objectives. Interventions can be put into various categories: interventions in agriculture and fishing; measures aimed at increasing competitiveness (research and development, industrial and trade policies), policies on networks (transport, telecommunications, energy); measures aimed at improving the quality of life (social policy, education and training, protection and improvement of the environment); etc. These are covered by the four Structural Funds (ERDF, ESF, EAGGF- Orientation, and FIFG): • The European Regional Development Fund (ERDF), created in 1975, aims at enhancing the economic potential of the assisted regions, supporting structural adjustments and promoting the growth of sustainable employment. It supports productive investment, infrastructure projects and actions to develop the economic potential of regions by co-financing operational programmes, assisting large projects, and providing global subsidies, technical assistance and preliminary studies. reformed in 1988, • The European Social Fund (ESF), created in 1958 with the Treaty of Rome, long-term but unemployment and improving the capacity of young people to find employment. Since 1993, incentives for adapting to industrial change have been added. The Fund helps to finance occupational training and measures for supporting employment and improvements in the educational system. is primarily aimed at fighting • The "guidance" part of the European Agricultural Guidance and Guarantee Fund (EAGGF-G), created in 1962 with the Common Agricultural Policy, is aimed at encouraging structural adjustment in agriculture through measures to modernise production and development in rural areas. Evaluation design and management • Finally, the Financial Instrument for Fisheries Guidance (FIFG), created in 1994 to replace various separate financial instruments in force since 1976, supports the structural adjustment in this sector. The Cohesion Funds and the European Investment Bank also contribute towards the cohesion policy through subsidies and loans for projects. The Cohesion Fund, instituted by the Maastricht Treaty, is intended to facilitate the preparation of Greece, Portugal, Ireland and Spain for entry into the economic and monetary Union. This Fund can be considered as supporting the weakest Member States, in exchange for their undertaking to satisfy convergence objectives and to control their budget deficits. i. e. the This Collection, and particularly this Volume, uses a typology of interventions based on three main domains of intervention covered by the policies implemented within the framework of the Structural and Cohesion Funds, (transport, communication, and public facilities in general), the development of human resources (through training actions and consultancy services to businesses and individuals) and support for the productive environment, including assistance to economic actors for expanding their capacity to produce, export, innovate, etc. (see page 137 and onwards). improvement of basic infrastructure This wide range of heterogeneous domains of intervention is frequent in socio-economic programmes. Yet, for programmes financed by the Structural Funds, this diversity reaches levels rarely equalled. This makes evaluation all the more difficult, which is why two volumes of the Collection provide possible methodological solutions from the point of view of indicators (Volume 2) and evaluation methods (Volume 4). Several levels and several stages of evaluation In the public domain, evaluation is a wide-ranging operation which may concern activities, services, organisations and practices. It is performed at three main levels: the project, programme or policy level. In North America, Northern Europe and at the European Commission, the notion of programme evaluation dominates. In the Southern European countries, evaluation literature more often relates to public policy. The term "programme" signifies (1) a coordinated set of different types of action (e. g. training courses; personalised advice; subsidies to employers, etc. ), of which (2) the operational objectives are defined precisely (e. g. a Evaluation issues within the framework of the Structural Funds programme for the conversion of the long-term unemployed) without implementation being defined entirely in advance and, finally (3) which has a limited time-scale and budget. The notion of "policy" refers to a set of programmes and/or measures which have the same general objective but not necessarily the same specific objectives, schedule and modes of management (e. g. an employment policy will refer to a set of measures which have accumulated over the years but are not always coordinated). The definition of the preceding two terms relates to that of a "project" which is a non-divisible action with a mode of management, a schedule and a budget that are all defined precisely from the outset. These three levels of evaluation are relevant for European cohesion policy and it is worth while linking them (see page 51). However, it is at the programme evaluation level that the most activity is concentrated. The MEANS Collection primarily concerns work at this level. In the evaluation literature, contradictory opinions are expressed on the timing of evaluation. Some authors consider that evaluation is only a retrospective exercise while others also attribute a prospective dimension to it. In the context of European cohesion policy, evaluation is clearly defined as an activity which takes place before, during and after a public action. The three steps or stages in the evaluation of a programme are related to its life cycle and consist of ex ante evaluation, mid-term evaluation and ex post evaluation. Given the uninterrupted sequence of programme cycles, there seems to be an overlap, in time, between evaluations performed at different stages and for different cycles. This necessitates a careful articulation so that the exercises reinforce one another without being redundant (see page 51). Partnership and evaluation Partnership Programmes financed by the Structural Funds are systematically co- financed by the Member States at the national and infra-national levels (the Regions or other local authorities). European financing is accompanied by a demand for additionality. The principle of additionally is that Community support for economic and social development must not be substituted for Evaluation design and management efforts by national governments; financing by the beneficiary State must remain at least equal to what it was before the Structural Fund contribution. In concrete terms, this means that any Community spending will be in addition to national spending. The application of this principle is verified nationally within the framework of financial control and not during evaluation as such. This verification still involves methodological problems (incomplete data, modes of calculation, etc. ). The combination of European and national financing has a decisive consequence for evaluation, i. e. the existence of joint decision­making and partnerships. The design and implementation of co­financed programmes requires the establishment of common objectives, a contractual framework of action and mutual trust. In the case of the Structural Funds, joint management is often taken to mean vertical partnership, that is to say, the fact that close collaboration must be established between the Commission and all competent authorities appointed by each Member State at the national, regional and local levels. There also exist several forms of horizontal partnership: geographical, sectoral, administrative, etc. Since 1994, partnership has spread to all competent organisations, including (as a rule) economic and social partners appointed by the relevant Member State. Box 4 ­ Partners in the European programmes / \ / \ ■Representatives^ / o f t h e E. U. \ ι ' N ν ι / \ \ — — *" ; , national x ι ^ ι ι representatives \ < 'ι „ ­ ­ » '. '' ι / s " * ·. \ ι. ' • * I, \ \ 1 I 1 ' ' regional \ ' ' ' ' representatives \ ¡('. J IJ. _ ­. }. 11. « Vertical » partners ,­r Decision • • >. Monitoring "*■ Evaluation Monitoring Committee A Other actors « Horizontal >· partners Evaluation issues within the framework of the Structural Funds Evaluation in partnership represents an opportunity In terms of evaluation, partnership and more particularly "vertical" partnership between the different administrative levels will generate opportunities that need to be exploited and constraints that must be the overcome. Vertical partnership development of pluralistic evaluations, known as "fourth generation" evaluations (see page 19), but also introduce a whole series of new constraints. Partnership policies produce situations in which evaluations are difficult to perform and in which their scope is reduced. Common objectives often lack precision and coherence due to the negotiations and compromises which governed joint their responsibility complicates resource management. As a result, it is difficult to define the object of evaluation and partners tend to fear that the negotiated modus operandi might be called into question. formulation. Moreover, for Because evaluation in partnership has to take into account the partners' diverse expectations and seek convergence between differing points of view, it is confronted with the fact that the stakes are not the same for all the partners. The existence of specific stakes is often overlooked, with the idea that what is good for the group is necessarily good for everyone. This implicit assumption results in the absence of differentiation between the questions asked and judgement criteria. Finally, an evaluation that is constructed without taking into account initial differences between partners, supposedly because of the partnership, is very likely to become a procedural construct without substance. From one situation of partnership to the next, there are profound de facto differences in the application of mandatory evaluation. In many countries and regions this obligation is reduced to a formal and relatively superficial exercise. In other cases, it has led not only to more effective use of the Structural Funds but also to a general evolution in administrative models. Thus, for example, in several Member States and regions which were recently declared eligible, Community requirements concerning evaluation framework of the have helped programming and the implementation of regional policies. to promote new practices within Moreover, the application of these regulations varies as much between the different Member States as between regions within the States themselves. It has been observed that evaluations are more likely to be of a high quality when relations between the partners are balanced, in other words, Evaluation design and management when one funding organisation is not too dominant. The richest regions are not necessarily those that comply with Structural Fund regulations most satisfactorily, probably because they do not contribute substantially to the budgets of these communities. The least privileged countries or regions receive a large amount of assistance but are not always the most active as regards evaluation, either because they already find it difficult to implement the available resources, or because they see evaluation as a threat due to their being indebted to the other partners. It can be assumed that each partner's desire to evaluate depends on their assessment of the gains as well as the costs and risks involved in the evaluation of their programmes. In the future, the European Commission will focus more on the choice of strategic priorities, leaving the national, regional and local authorities to steer programming and the management of programmes. This more decentralised and simplified management will require in return more extensive monitoring and evaluation. Development of practice within the framework of European Structural Funds The diffusion of evaluation within the framework of European cohesion policy has occurred within the last ten years. As it spreads throughout the Member States, it encounters widely diverse references and ways of thinking (macro-economic expertise, regional economics, political analysis, strategic analysis, etc. ) and is based on concepts that may vary from one country to the next. Whatever the vocabulary and structures adopted, the evaluation systems set up in the different Member States have helped to enhance the transparency and effectiveness of interventions financed by the Structural Funds and, in many cases, even of other public interventions in similar domains. From a situation of virtual non-existence of practice, considerable progress has been made in the quality of evaluation reports. A number of difficulties still exist, however, particularly in the analysis of impacts and the use of reports. Changes in Structural Fund regulations At the Commission, a demand for transparency vis-à-vis taxpayers, on the sound use of funds from the Member States, has progressively taken hold. This aspiration is reflected in a simple expression: "striving for effectiveness in European spending". The Single European Act, adopted at the beginning of 1986, introduced into the EEC Treaty a new section V containing article 130D. This article announced the reform of the Structural Funds, intended to improve their effectiveness with a view to enhancing economic and social cohesion in the Community. Similarly, in February 1992, the Maastricht Treaty included this imperative need for cohesion, and Article 130B stipulates that the Council is to define measures required for ensuring that the Funds are effectively used. In accordance with these requirements, the regulations relating to Structural Funds (those of 1988 for the first generation of Structural Funds and those of 1993 for the current generation) included specific articles on evaluation, particularly Article 6 of the framework regulation and Article 26 of the coordination regulation. Evaluation design and management Regulations applicable between 1988-93 Article 6, Paragraph 2, of the 1988 regulations stipulated that "in order to assess the effectiveness of structural impacts of Community actions must be evaluated ex ante and ex post. For this purpose, app ¡cations for support by the competent authorities must include the necessary information so that it can be evaluated by the Commission". interventions, the These first regulations, adopted by the Council on 19 December 1988, concerned regional development plans which had to be submitted by 1 January 1989. For many national and regional administrations, the need to establish pluri-annual programming was in itself a kind of revolution. Moreover, most competent authorities and the Commission services did not have the required evaluation skills, at least as far as evaluating Structural Funds was concerned. Consequently, evaluations carried out during the 1989-93 period were not of a very high quality, despite efforts by many actors, especially at the Commission. It was observed, in particular, that the framework defined by this first version of the regulations failed to create adequate conditions for the ex ante evaluation phase to take place in the normal way. Regulations applicable between 1994-99 Acknowledging this partial failure, at the instigation of the Commission but also through the strong impetus given by certain Member States (the Netherlands and the U. K. , in particular), the new July 1993 regulations considerably strengthened requirements concerning evaluation. This new regulation (Article 6 of the framework regulation) specifies the notions of prior assessment, monitoring and ex post evaluation. The Articles detailing the content of programming documents submitted by the Member States, introduce the notion of "specific quantified objectives, where possible". Similarly, Article 26 of the coordination regulations presents a precondition clause specifying that "support will be granted when a prior assessment has shown the socio-economic advantages to be gained in the medium-term, in relation to the resources mobilised". for ex ante evaluation, a In practice, this very strong requirement Community variant of the British notion of Value-For-Money, was not applied very strictly. By contrast, the introduction of mid-term evaluation which had not existed before was particularly successful, as our survey below shows. This strengthening of European legislation as regards evaluation was clearly a strong incentive for spreading the idea that evaluation was unavoidable if Community Funds were to be obtained. Evaluation issues within the framework of the Structural Funds Regulations of the third generation of programmes (2000-2006) With the third generation of Structural Funds the assessment of their effectiveness will be reinforced. Community actions will henceforth be the object of ex ante evaluation by the Member States themselves, of intermediate evaluation by programme managers, and of ex post evaluation, at the initiative of the Commission, for assessing their effects in relation to the objectives (their particular objectives and cohesion objectives) and for analysing their impact on specific structural problems. The effectiveness of the Funds is to be measured at three levels: the overall effect on the objectives of Article 130 A of the Treaty (particularly the strengthening of social and economic cohesion), the effect on the priorities proposed in the Plans and provided for in each CSF, and the effect on the specific priorities selected interventions. Complementary the evaluations, where relevant of a thematic nature, may be launched with a view to identifying experiences that are transferable from one programme to another. for In order to improve transparency, evaluation reports are to be made available to the public more extensively than in the past. Recent developments in evaluation practice1 Prior appraisal: mixed findings Evaluation of the second generation of programmes was stimulated by the revised and reinforced regulations of 1993 and given a rigorous framework. In particular, the prior appraisal phase has to specify the objectives of regional strategy and ensure that Community priorities have been taken into account. It thus provides the responsible services at the Commission with elements for judging the relevance of actions during negotiations. As a rule, prior appraisals are performed by national administrations. For the first time, they are now also systematically being performed by the Commission. However, prior appraisals have not always been as useful as ' This section is based on a survey carried out in 1997 within the framework of the MEANS programme. The survey consisted of telephone interviews in all the Member States and visits in five countries. Evaluation design and management expected. Due to a lack of time, analyses have often remained too superficial to produce value added, in respect of both the formulation of general and specific programme objectives, and the modalities of their implementation. Efforts have nevertheless been made by the Member States, particularly in the quantification of objectives. Programming Documents of the 1994-99 period are unquestionably of a better quality than those of the preceding period. They were enriched by debate around prospective analyses, which helped to clarify objectives, structure programme components and grade priorities. The spectacular development of mid-term evaluation Significant progress has been made in the area of the mid-term evaluation of the second Structural Funds programming cycle. Because this period was able to build on the experience gained during the preceding period, evaluations were systematically performed in most cases. From a quantitative point of view, the number of reports produced has increased five or six fold compared to the preceding period, and evaluation practice has developed spectacularly. Thus, in the fifteen Member States, for the period from 1996 to the beginning of 1998, over 380 mid-term evaluations were organised by Monitoring Committees for the six Objectives. Among them, there were no less than 250 evaluations for Objectives 1 and 2 alone, for the single year 1997. By comparison, for the period 1989-1993, an average of less than 40 evaluations per year were performed on all the programmes and for all forms of evaluation (ex ante, ex post and thematic). The number of programme evaluations within the framework of Structural Funds was therefore mu tiplied by 5 or 6 from one generation to the next. Systematic overall evaluations and sometimes in-depth thematic and geographic analyses have been performed for Objective 1 and 5b programmes. These mid-term evaluations aim at providing the first programme results and indicating adjustments to be made during the 1994- 1999 programme. For Objective 2 programmes, evaluations were performed in 1996-97, at the end of the first programming phase (1994-96), in the majority of countries. The new Member States also supplied their evaluation reports. In view of the delayed launching of programmes, the implementation period Evaluation issues within the framework of the Structural Funds taken into account in reports was shorter but the work was nevertheless useful. In Sweden, for example, the evaluation of Objective 2 programmes was recognised for drawing initial conclusions on implementation methods and the appropriateness of the measures. In general, the usefulness of the exercise from the managerial point of view is enhanced with experience. This is true to such an extent that many managers have done more than demanded by the regulations. Over and above the thematic and transnational evaluations performed at the initiative of the Commission services, numerous sectoral evaluations have been launched by national administrations or programme managers. Noteworthy improvements in the quality of evaluation reports The average quality of evaluations carried out during the 1994-99 period has unquestionably to the preceding period, particularly as far as mid-term evaluations are concerned. There is of course still some mediocre work, but this can be explained by the delay in launching certain programmes in some Objective 1 or Objective 5b regions. increased compared Most reports can be criticised for not being analytical enough and for providing conclusions that are too descriptive to be truly useful in decision making. There are nevertheless some practices which meet the best professional standards. Thus, evaluations (particularly in Belgium and Ireland) have provided credible conclusions on the tricky question of reallocating budgets: identifying measures to be reinforced or reconsidered. With more experience in the domain and better knowledge of the specific nature of evaluation, evaluators have been able to produce better reports. Many of them are based essentially on the analysis of secondary data (collected from data bases or existing documents) and monitoring data which have improved in recent years. On the basis of these data, the Member States have been able to comply with the regulations which require the examination of physical and financial data for evaluation. Significant work has been carried out in Italy, for example, where a system of standard indicators was established in the area of training, and in Portugal where proposed indicators were constructed for all sectors. It is in the collection and processing of socio-economic data that most improvements need to be made. Primary data, collected directly by surveying the publics concerned, remain rare and information provided by Evaluation design and management context indicators fails to take full advantage of national and regional statistical systems. Some of these statistical systems have not greatly developed (e. g. in Greece and Portugal) or, as in the case of the new German Länder, have not been harmonised in line with European systems (EUROSTAT). It is within the perspective of optimising existing data that Volume 2 of this Collection has been written. Multiplication of initiatives and increase in resources Remarkable efforts have been made by the Commission services, both on their own and in close cooperation with national administrations. The Community services can be credited with the creation and strengthening, as from 1994, of evaluation units and the diffusion of specialised know-how to all the operational services. Between 1994 and 1998, close to 260 officials in charge of Structural Funds attended two- to three-day evaluation training courses as part of the MEANS programme and SEM 2000 (i. e. : almost half the officials concerned). Depending on the Member State, the national administration has an essential role at the central level (e. g. the Irish Department of Finance; the German Treasury and Ministries of Agriculture or Transport) or the decentralised level (e. g. the Prefectures in France). In some countries, such as Spain, responsibilities are assumed by both types of authority (national and regional). In the Netherlands, for example, the national authorities formulated a methodological to guarantee the relevance and quality of evaluation reports carried out under the responsibility of the different committees. framework (see the example below) Example: The Dutch methodological framework In 1994 this country developed a common methodological framework for all the stages of evaluation and all Structural Funds. As a result, each evaluation report is set out in the same way: socio-economic situation of the region, strategy and coherence, organisation and monitoring, implementation and synergy, and effects. This framework, combined with the use of indicators common to the different regions, and with an efficient monitoring system, allows for an exchange of information and experiences owing to the comparability of tools used in the regions concerned by the Structural Fund programmes. Participation by regional authorities in evaluation exercises has increased substantially since 1994: programming, definition, selection of evaluators, Evaluation issues within the framework of the Structural Funds taking into account of recommendations, etc. This trend can be considered as a factor in the development of evaluation, especially when these regional authorities have a high degree of autonomy and are therefore dominant partners. In Belgium, the federal level does not intervene in the management of structural evaluations. Evaluation is consequently managed at the regional level. In Germany, the administrations of the Länder are primarily responsible for the implementation of Structural Funds and therefore for their evaluation. Significant differences nevertheless exist between the old and the new German Länder. The former, familiar with an evaluative approach, involve the national level only when this is essential, whereas the latter are systematically supported by the national administrations. In countries such as Portugal or France, ministries concerned with technical issues are not necessarily very active in the evaluation of Structural Funds. By contrast, the administrations which coordinate at the national level have a key role in constantly managing and monitoring evaluation work performed under the responsibility of each of the Monitoring Committees. The dynamics created by the Structural Funds The value added of partnerships Partnerships, by creating relationships of trust, have made evaluation credible and have thereby facilitated its implementation. The reservations and even hostility vis-à-vis evaluation which persisted in the early 1990s has gradually disappeared in most Member States. Naturally, this has greatly facilitated the implementation of evaluation. The management of Structural Fund programmes involves a large number of actors and coordination between European, national and regional levels. Partnerships, involving socio-economic partners, were built up around Monitoring Committees. In almost all countries Monitoring Committees have entrusted the responsibility for evaluation to Steering Committees (or technical or ad hoc political supervisory committees, etc. ) which are smaller and therefore more apt to address technical issues. These committees are usually composed of European, national and regional authorities and sometimes of experts (academics, statisticians, etc. ). They play a key role in the methodological conception and examination of the Evaluation design and management quality of evaluation work. Evaluation reports are discussed within these restricted groups before being presented to the Monitoring Committees in plenary sessions. The application of this organisation in Portugal is presented below. Example: Partnership, a key factor of success in Portugal Each Monitoring Committee constitutes an evaluation group, consisting of Commission desk officers and representatives of Portuguese national and regional administrations, responsible for relations with the evaluation team. This technical group meets at least three times a year (to validate the terms of reference, select offers, and discuss reports before submitting them to the Monitoring Committee). Good relations between partners within the technical groups and constructive interactions between commissioners and evaluation teams are two strong points which, according to national officials, have had a highly positive impact on the quality of evaluations. Partnerships with Commission services have also played a positive role, e. g. : contributions in terms of technical documents to the different phases, participation in the selection of evaluation teams and analyses of the quality of reports. Many initiatives have been launched by the Commission to disseminate knowledge and promote exchanges between Brussels and its partners. For example, the Commission has set up Technical Evaluation Groups (consisting of representatives of national authorities and Commission services) which meet several times a year. For the commissioning of evaluations it has, in partnership with several States, drawn up model terms of reference. Meetings are held regularly in the Member States (Greece and Finland, for example) to share and exchange information. The subject of partnership cannot be addressed in the context of Structural Funds without referring to one of the conclusions produced by the MEANS programme totally relevant today: although partnership is a driving force for the development of evaluation practice it can also undermine the quality of evaluations when too many compromises are made between co-decision-makers. in 1996, which remains The professionalisation of evaluation teams During the past five years the evaluation market has developed remarkably, as have the number of agents performing evaluation work. The supply of Evaluation issues within the framework of the Structural Funds evaluation services often remains external to administrations, except for the central administration in Ireland, France and Italy which have created their own specialised evaluation teams. When they exist, these evaluation units, whose independence is guaranteed by a special legal status and a position "outside the hierarchy", benefit from experience that outside practitioners do not necessarily have since evaluation accounts for only a part of their professional activities. The case of the Irish administration is presented below. Example: Permanent evaluation teams in the Irish administration In 1992 the Irish government and the Commission established specific Evaluation Units for Industry and Agriculture, as well as a specific Unit for the ESF and, among others, for the entire CSF, to act as a "horizontal" coordinator. These units, thanks to their independence and professionalism, contribute towards the smooth functioning and quality of the work produced. The units carry out thematic evaluations and in-depth analyses. Their work is planned by pluralistic Steering Committees. Most evaluations are conducted by the units themselves, although the Agriculture unit sometimes sub-contracts work. Reports on all these evaluations are published. The number of consultancy firms active in the field of evaluation has risen sharply in all countries. It is considered that entrusting evaluation to outside evaluators, whether private consultants or universities, is a guarantee of the independence of the team in relation to the authorities concerned. This sector has developed in different ways: the emergence of teams linked to the academic world and the creation of departments or subsidiaries of the major international groups. Thus, in Greece and Portugal a significant number of small firms (over thirty consultancy firms) are currently involved in the evaluation of structural programmes. In some cases, such as France, a phenomenon of concentration can be observed, i. e. a few firms share most of the market. The conditions for performing quality work are not always met. For example, the time allocated is often too short (from one to six months) because of scheduling reasons. In some countries such as Finland, Ireland, France or Germany, or for in-depth evaluations or certain specific cases, schedules are more reasonable (up to twelve months). The corollary of this is that the budgets allocated are too small (30 Keuros for certain evaluations in the Netherlands and Spain) to carry out proper field surveys among the individuals or businesses concerned. Evaluation design and management In general, the cost of evaluation work remains fairly low compared to the overall budget of the programme: 0. 02% on average and probably 0. 01 % in the Objective 1 regions. However, these ratios will have to be brought closer to international standards (see page 78) when the budgets for in-depth evaluations, either underway or ex post, in the future, are made known. The quantitative development of supply is also accompanied by qualitative developments generated by greater professionalism. Skills have evolved in recent years, owing more to accumulated experience the specialisation of evaluation service suppliers. Evaluation teams often have multidisciplinary competencies, although economists play a dominant part in the evaluation of regionalised programmes. than to It is increasingly frequent that consultants request the collaboration of academics who have more specialised and precise expertise in specific areas of public policy. In fact, many consultancy firms have been set up by academics, especially in the southern European countries (Portugal, Spain and Italy). Know-how in Germany, Finland and Belgium is based primarily on universities, institutes and research centres. Certain methodological progress made in recent years warrants attention. Know-how in in the modelling of regional economies has Germany, Italy, Ireland and, more recently, Belgium. Input-Output analyses have been skilfully implemented for conducting evaluations in Portugal, Germany, Italy and Spain. The analysis of relations of causality has developed and specialists are using comparative statistical analyses more often. The use of comparison groups, limited to Denmark and the U. K. until recently, has spread in Sweden, Germany, France and Spain, primarily in the domain of employment. increased The build-up of evaluation capacities within countries It has been noted that institutional peculiarities of each State result in significant differences in the implementation of evaluation. The same applies to the appropriation and use, by commissioners, of results in reports. Unsurprisingly, the European landscape, with regard to the use of evaluation, varies widely. Nevertheless, three phases in the level of utility attributed to evaluation, and hence in the quality of its organisation, can be distinguished schematically. 7s' phase: Evaluation is seen as an answer to regulatory obligations. It is therefore a constraint and an additional workload weighing on managers Evaluation issues within the framework of the Structural Funds who consider it, above all, as a demand from Brussels-even though regulations are decisions unanimously adopted by the Council. 2nd phase: Evaluation becomes a system to aid the design and management of interventions. A dialogue is established with the evaluators and the quality of the information gathered improves. In this highly operational phase, the evaluation approach is refined and progress is made rapidly. 3rd phase: Evaluation becomes a political act, the results of which are publicly debated. The aim is to inform public opinion on the effectiveness of the use of Public Funds, and on the demonstration of their efficiency in terms of obtaining the expected impacts (Value for Money). In a sense, evaluation becomes a tool of democracy by informing citizens, if not seeking their consent on State spending. These three phases are represented simply in Box 5 which shows the different Member States' use of evaluation work, in relation to each other. The use of evaluation is represented by the vertical axis, while evaluation practice within countries is represented by the horizontal axis. For each case, the diagram shows whether practice is limited to the Structural Funds or whether evaluation is also practised within the framework of other national or regional policies. Box 5 - Practice and use of evaluation Netherlands) ( Sweden ] Limited to the Structural Funds Also exists outside the Structural Funds Evaluation design and management This graph is only a transitory image (from a survey carried out in 1998). The dynamics of each country could also be shown. For example, Portugal has become largely familiar with evaluation practice and, in recent years, significant progress has been made in terms of quality and the use of evaluation work. The influence of Community regulations on the development of practices is significant in most of the Member States. Thus, in Objective 1 countries, for example, evaluation work is essentially focused on the Structural Funds. This influence, although perceptible, seems less marked in certain countries such as Ireland where recent reforms to improve the management of public actions stem mainly the performance management movement developed in the English-speaking countries. In certain States, European requirements governing the implementation of structural policies have prompted remarkable changes in attitudes and have led to systematic evaluation. Thus, in Germany and France, for example, evaluation was being institutionalised but Community regulations accelerated the existing process. from As far as the most recent members of the European Union are concerned, Community regulations have had particularly noteworthy effects because evaluation practice, skills and culture are still limited. The obligation to evaluate, imposed by the Community, has provoked the emergence of a new market accompanied by greater institutionalisation and a more assertive evaluation culture. This phenomenon can be found in Finland the relative is developing where evaluation practice integration of European structural policies into Finnish national policies. fast, owing to It is interesting to note that the democratic use of evaluation is promoted when a national evaluation culture exists independently of Community requirements. In other words, regulatory obligations can help to establish a managerial use but are probably not enough to induce a contribution to the public debate. By contrast, the number of countries in the bottom-left part of Box 5 reflect from an institutional point of view, as well as its influence on the development of practice in countries where the evaluation culture was previously weak. importance of Community regulations the International reflections on the building of evaluation capacity emphasise the importance of incentives and actions to promote awareness, as significant factors in the development of evaluation. Various initiatives can the creation of national evaluation societies or contribute, including Evaluation issues within the framework of the Structural Funds associations such as the recently founded German Evaluation Society and the Italian National Evaluation Society (Boyle, Lemaire, to be published). The particularly active British evaluation society (UKES) is briefly presented below. Example: The British professional evaluation community British teams in the private sector and universities have developed internationally recognised competencies and expertise in evaluation. The U. K. is one of the rare Member States to have evaluation research teams which work in this domain "transversally", without limiting themselves to a particular methodological aspect nor to a given political sector. The British evaluation society (UKES), with over 200 members, organises an annual conference and workshops for developing the exchange of methodology. Training is an essential tool in promoting awareness. Those countries in which administrative personnel have received evaluation-specific training remain rare. However, wherever these activities have been set up (seminars on evaluation organised in France by private firms or administrative services, or in Greece by Commission services) the results are positive. Moreover, a few courses on the evaluation of public policies exist or are being set up in universities, most often as part of training focused on specific sectors of the economy. Based on the above observations concerning recent developments in evaluation practice, it is possible to identify two major challenges to be met if the current dynamic of the evaluation of the Structural Funds is to be maintained: • the professionalism of practitioners must be enhanced to complete inadequate know-how among administrative officials and evaluation teams, and improve the quality of evaluation work; • effective action to increase awareness and provide incentives must be implemented; in other words, evaluation must move on from a response to an obligation to a voluntary approach. Preparing an evaluation and defining its content Defining the general framework of an evaluation Evaluation can have a number of forms, depending on the stage of the programming cycle at which it is performed (ex ante, intermediate or ex post evaluation), the level of decision-making concerned (evaluation of a policy, programme or project) and the scope of the evaluation (overall, thematic or in-depth evaluation). These different forms of evaluation, rather than being independent from one another, have to be linked together. It is therefore preferable to view evaluation initiatives in a general framework covering several programming periods and levels of responsibility. In particular, the succession of European Structural Fund programming periods imposes a schedule that creates overlaps between ex ante, intermediate and ex post evaluations. It is therefore necessary to take into account the complementarity between evaluation work, so that each evaluation can draw on preceding ones, contribute to future ones and avoid redundancy. the overlaps between Similarly, (political /programme / project) means that evaluations performed at one level produce conclusions relevant to other levels. In particular, programme evaluations take into account the objectives and priorities of the policy context in which they are situated. responsibility levels of Linking the different stages of evaluation In order to anticipate the usefulness of an evaluation, it has to be situated in the life-cycle of the programme considered as it is represented in the following diagram. Box 6 - Evaluation in the life cycle of a programme Design of the programme Ex ante evaluation Feasibility study Ex post evaluation Report Conclusion Implementation Monitoring outputs Mid-term evaluation Evaluation design and management Ex ante evaluation -4 Ex ante evaluation takes place at the beginning of the cycle before a programme has been adopted. This form of evaluation helps to ensure that the final programme is as relevant and coherent as possible. Its conclusions are intended to be integrated into the programme when decisions are taken. Ex ante evaluation focuses primarily on an analysis of the strengths, weaknesses and potential of the Member State, region or sector concerned. It provides the relevant authorities with a prior judgement on whether development issues have been diagnosed correctly, whether the strategy and objectives proposed are relevant, whether there is incoherence in relation to Community policies and guidelines, whether the expected impacts are realistic, and so on. It also provides the required foundations for monitoring and for future evaluations, by ensuring that there are explicit and, where possible, quantified objectives. It helps to specify selection criteria for the selection of projects and to ensure that Community priorities are respected. Finally, it helps to ensure the transparency of decisions by allowing for a clear explanation of choices made and their expected effects. Ex ante evaluations are performed at the time when public authorities are involved in discussions and negotiations on the future programme. They are therefore subjected to strong constraints: pressure of deadlines, vague formalisation of the proposed programme to be evaluated, amendments to this proposal while the work is underway, demands for confidentiality, etc. The evaluation team must therefore be able to intervene flexibly and rapidly, and be able to apply techniques for analysing needs and simulating socio­ economic effects. Mid-term evaluation M Mid-term evaluation is performed during the second stage of the programming cycle, during the implementation of the interventions. Depending on the conclusions of mid-term evaluation, adjustments may be made during the cycle. This evaluation critically analyses the first outputs and results of interventions. It also assesses the financial management of the programme and the quality of the monitoring and of its implementation. It shows how and whether original intentions have been carried out and, Preparing an evaluation and defining its content where relevant, checks whether de facto changes have been made to the initial objectives. By comparison with the initial situation, it highlights changes in the general economic and social context and judges whether the objectives remain relevant. Mid-term evaluation also examines whether the evolution of Community priorities and policies poses a problem of coherence, and helps to prepare adjustments and reprogramming, and to argue them in a transparent manner. Mid-term evaluation relies heavily on information drawn from the monitoring system, but also on ex ante evaluation and on information on the context and its evolution. It generally consists of short and exhaustive exercises focusing primarily on the results of the programme evaluated, without attempting an in-depth analysis of impacts that have not yet had the time to emerge. It is, however, possible and advisable to refer to in-depth or thematic evaluations of former programmes when such analyses do exist. Mid-term evaluation has a "formative" nature, that is to say, it produces direct feedback into the programme which it is helping to improve as far as its management is concerned. Ex post evaluation ► Ex post evaluation recapitulates and judges the entire programme, particularly its impacts. Its aim is to account for the use of resources and to report on the effectiveness and efficiency of interventions and the extent to which expected effects were achieved. It focuses on factors of success or failure, and on the sustainability of results and impacts. It tries to draw conclusions that can be generalised and applied to other programmes or regions. Ideally, the results of this evaluation should be available when the next programme is planned, that is, at least a year before the end of the programme. However, for the impacts to have been produced, ex post evaluation would have to be performed two to three years after the end of the programming period. While waiting for this period to pass, a provisional review is often requested shortly before the end of the programming cycle, in liaison with the ex ante evaluation of the following cycle. Impact analysis is always a large-scale exercise if performed systematically. Ex post evaluations therefore tend to involve surveys in the field and to take place over long periods lasting from twelve to eighteen months. Evaluation design and management Successive programming cycles The sequence of three evaluation phases in successive cycles creates overlaps that have to be organised as efficiently as possible to avoid any duplication of work. The basic principle is that of combining evaluation work, during a programme, with the use of conclusions of evaluation performed on the preceding programme. The relative continuity of actions programmed from one period to the next makes it possible to use conclusions from the recent past to judge the relevance of the new measures proposed. The following diagram shows that interactions are possible between evaluation work performed at the different phases of several successive programmes. Box 7 - Connections between evaluation and programming cycles Programmes (1) (2) (3) Results, impacts Evaluation mid-term (1) ex ante (2) ex post (1) mid-term (2) ex ante (3) Observation Feedback ■^ Thus, an ex ante evaluation that prepares the adoption of a future programme has to take advantage of the results of earlier work, i. e. : • the intermediate evaluation of the period drawing to a close. This evaluation will have produced conclusions on the first years of activity and on the ensuing programmes. It may have been completed by a final review of the outputs and results of the current programme, based on information from the monitoring system. Preparing an evaluation and defining its content • ex post evaluation of the period preceding the current period, possibly completed by thematic evaluations and in-depth analyses. These evaluations will have made it possible to observe and analyse the impacts of former interventions which are similar to the planned interventions and which took place in a partly similar context. H Since the role of intermediate evaluation is to check whether the objectives are still relevant and in the process of being achieved, it will be necessary to refer primarily to the monitoring system data but also: • to the ex ante evaluation and particularly to the diagnosis made in relation to the prevailing socio-economic context before the start of the programme, and which needs to be updated; • to the ex post evaluation of the preceding programme, of which the conclusions concerning the same areas of intervention could serve as references. ► Ex post evaluation is based on management and monitoring data and on surveys in the field which will help to observe and analyse the real and sustainable impacts of the interventions. It refers to ex ante evaluation in so far as it has to report on the attainment of objectives. It refers to intermediate evaluation, particularly to identify the success or failures which were identified at that stage. In so far as evaluation must draw conclusions from the experience of preceding programmes to improve future programmes, an interesting solution is to establish a pluri-annual evaluation plan. The idea is to identify the different possible evaluations and to establish their date and content in relation to the political schedules and deadlines for decision-making of the evaluation "customers" at various levels. ■ Performing evaluation at different levels Since the Structural Funds reform in 1988, the Commission has favoured the integrated programming of Community interventions. A programme is composed of numerous projects within the framework of specific objectives, which are themselves directed towards the global objective of the programme. The evaluation of a programme is neither the sum nor the synthesis of the evaluations of the projects comprising it. Some questions are not raised at the individual project level but concern the programme evaluation. That is, for example, the case when the evaluation of the effects of synergy is Evaluation design and management required. This point is considered below (see page 128) as well as in Volume 4 of the MEANS Collection. Evaluating projects at their selection stage The national and regional authorities are responsible for selecting projects. This is frequently accompanied by an ex ante evaluation, also called prior appraisal. The example described below is that of a regional development programme in the U. K. , a country where particular care is taken in project appraisal. Example: Selection of the projects within the Highlands and Islands programme. Within the framework of the programme under consideration, there are two cycles of calls for projects each year. All the applications for funding are first examined in relation to their eligibility. Projects considered to be eligible are given scores in terms of eight general criteria, common to the entire programme (relation to the objectives, probable effectiveness, respect innovation, relations with other projects, intervention of the private sector and support from local partners), and in terms of several specific criteria which vary depending on the measure. Since the criteria are weighted, it is possible to give a global score to each project. After comparing applications for funding with available resources, a standard for admission is established and a list of projects to be financed is recommended. The final decision is taken by the programme managers. the environment, additionality, for This example shows that project evaluation must be performed in relation to the programme objectives. Close ties with the objectives are one of the main selection criteria in the scoring system. Inversely, during the overall evaluation of a programme, the approach is often from the project to the programme level, i. e. bottom-up. In many cases analyses are initially focused on specific projects or measures, which are therefore micro, and progressively extended to the entire programme to evaluate hypotheses of causality which are, by nature, always macro. In this way project evaluation feeds into programme evaluation. The limits of the bottom-up approach lie in the fact that it causes the evaluation team to make increasingly fragile assumptions because the effects to be assessed are more and more distant and complex. The advantage of this articulation of project and programme evaluation is that it provides a detailed explanation of the sequence of short and medium term impacts. Preparing an evaluation and defining its content Example: evaluation of a project in its regional context. An evaluation team performed an ex ante evaluation of public support attributed to the establishment of a car manufacturing plant in southern Italy. The main question concerned the economic and social impact of this plant on local business and, in particular, on the reduction of unemployment. By examining the project in isolation, the evaluation team showed that the impact was weak. In fact, the plant remained largely dependent on resources from outside the region in terms of labour and suppliers. Thus, from a purely micro-economic point of view, the impact of the project was limited. On the other hand, by examining the project in its regional context, the evaluation team showed that it was likely to generate new activities, to develop local production of goods and services and, finally, to produce significant long-term effects, provided that macro-economic mechanisms were taken into account. Evaluation of large projects An exception is made of the evaluation of large infrastructure projects which are important instruments in the implementation of European Union regional policy. The major projects have been defined as "those of which the total cost [. ] generally exceeds 25 million euros for infrastructure investments and 15 million euros for productive investments"2. The amounts at stake justify the projects' in-depth and independent prior appraisal, by comparing their costs and benefits according to the technique described in Volume 3 of this Collection. These evaluations are the joint responsibility of Commission and Member States3. the For investments in infrastructure, an analysis of the project's socio­ economic advantages includes the expected rate of use, the potential impact on the development or reconversion of the region concerned, and the estimation of the consequences of the Community intervention on the realisation of the project. For productive investments, the analysis will focus on market prospects in the sector concerned, employment effects and expected profitability of the project. In the processing of individual applications for funding, the Commission takes into account the results of prior appraisals and evaluations. If the appraisal presented by the applicant is considered insufficient and 2 Article 16 of the Council regulation 2082/93 3 Regulation 2082/93. Evaluation design and management unconvincing, the Commission may require the applicant to expand the analysis or, alternately, may itself appraise the project by requesting, where necessary, the opinions of independent technical experts. In any case, the Commission's decision will be the result of dialogue and collaboration with the applicant, to ensure that the investments made produce the most satisfactory results. Evaluation of policy The results of programme evaluations are systematically used in syntheses produced at a national or European level. At these levels, they serve as a basis for reflection and support for decision-making in the area of economic and social cohesion policy. Box 8 lists a series of direct and indirect "clients" of programme evaluations at the European level. Box 8 - The "customers" of evaluations at the European level Programme evaluations are directly or indirectly useful: • to European governments, as a basis for strategic planning of cohesion policy and for preparing their proposals to the Council; • to the Commission's operational units, in the preparation of decision-making on the financing of programmes and their content; • to the representatives of the Commission, when they express the Community the point of view during decision-making on adjustments during implementation of the programme; • and, finally, to the services responsible for preparing annual reports on the implementation of Structural Funds and the tri-annual report on social and economic cohesion. A particularly interesting form of evaluation at the European level is that which analyses certain impacts considered as priorities at that level. Thus, an evaluation may address transversal questions such as employment impacts, the competitiveness of SMEs, equal opportunities, the environment, etc. These Community priorities are often recalled in programming documents without, however, being fully integrated or mainstreamed. Trying to evaluate impacts in relation to priorities amounts to designing what is called a thematic evaluation (see Volume 5 of the Collection). A thematic evaluation as such covers several programmes and may be carried out at the national or European level. Its main characteristic is that it treats a specific theme, generally a priority for the commissioner. For example, the European Commission performed an evaluation on the theme Preparing an evaluation and defining its content of equal opportunities, in about ten programmes in different States. Thematic evaluations serve to compare different programmes and to gain a better understanding of processes of impact production in a particular domain. Major transnational evaluations have thus been launched on the themes of the Environment, R&D and support for SMEs. Selecting t he different levels of analysis Evaluation experience of Structural Fund programmes shows a wide variety of ways of addressing evaluative questions. Fairly often, the evaluation team is expected to treat ten or more questions in a few months. Other evaluations, by contrast, devote a whole year to the in-depth analysis of a single question. The choice of studying a question in greater depth will be guided by the need to better understand a difficulty encountered, to show the exemplary character of a success story, to better understand the effectiveness of a measure, to identify the conditions in which an objective was met, etc. In the following example, the steering group considered analysing several priority questions in more depth. Example: Seven questions to consider in depth when evaluating a programme In 1997, within the framework of the evaluation of an industrial reconversion programme in Franche-Comté region in the North-East of France, the evaluation steering group devoted its first two meetings to the identification and grading of a series of questions. Seven fields of questioning were thus identified. These were then discussed in workshops in order to formulate, for each of them, a realistic and useful evaluative question. In the order of priorities established by the steering group, these questions were as follows: (1) Which categories of projects have the best impact on the attractiveness of the territory? (2) Has support to enterprise helped to create or maintain jobs that are firmly rooted in the area? (3) Have services for enterprises contributed sufficiently to the diversification of economic activity? (4) Does training serve to raise qualifications or rather to make them more relevant? Which of these two expected effects is the most necessary? (5) Is the probable impact of higher education and research projects sufficient? (6) What is the effectiveness of the different categories of projects in terms of creation and maintenance of employment? (7) Which categories of projects guarantee or improve equal opportunities between men and women? An in-depth study of all seven questions would have been too costly. It was therefore decided to limit the number of questions to study in depth to three. Certain questions were eliminated so that there would not be duplication with other evaluations underway in the region. Finally, questions (1), (4) and (5) were judged priorities. Evaluation design and management It is the combination of several criteria that leads to the selection of questions to be evaluated in depth. These may be: • the most costly measures (assuming the stakes involved in decision­ making increase in proportion to the budget); • the most innovative measures (considering that these parts of the to be validated by an programme are experiments which need evaluation); • the most strategic objectives (assuming that these parts of the programme are the most likely to gain from an informed debate); • the most controversial objectives (assuming that these parts of the programme are the most likely to gain from the clarification of judgement criteria), etc. • priorities which interest a specific decision-making level in particular, etc. Box 9 shows that depending on the object analysed. in-depth analyses may assume different forms, Box 9 - Three situations of in-depth ana ysis Measures Concerned groups S *^ 1 ^ ~ ~ ~\ ^ 2 SMEs Impacts Exports New business creators JT j " * "^ ^ "~ 3 ~~ Innovation Advice Investment incentives Training The first form of in-depth analysis (see circle 1 in the Box) concerns the impacts of a measure, an action or a major project. Depending on the nature of the public concerned, the impacts sought will be, for example, jobs created or maintained, investments induced, the consolidation of activities, the diversification of activities, etc. The addressees of the intervention are known and can be questioned. Primary data obtained in this way provide indications relative to gross effects. The analysis of Preparing an evaluation and defining its content causality may also be carried out by means of the methods and techniques described below (see page 147): the search for confounding factors, the estimation of deadweight, eviction or displacement effects, the constitution of a comparison group, and the construction of a counterfactual situation. In this situation, the evaluation produces conclusions on the success or failure of a given intervention. Where possible, these conclusions will be generalisable to other similar interventions (external validity). The second form of in-depth analysis (see circle 2 in the Box) focuses on a particular public, such as the long-term unemployed, creators of new enterprise, small farms, etc. The evaluation is performed from the point of view of the public concerned. It analyses this public's needs and attitude towards the programme and its judgement criteria. In this situation, the evaluation produces conclusions on the comparative impact of the different measures directly affecting this public. It can also show how these different measures reinforce or contradict one another (synergy effects). The third form of in-depth analysis (see circle 3 in the Box) concerns an impact or a category of impacts. The question then applies to all the programme measures and focuses on their respective contribution to the production of this impact. The particularity of this type of situation is that the public concerned is homogeneous, whereas the actions which impact on this public are heterogeneous and numerous. In this situation, evaluation leads to conclusions on the comparative impact of the different measures contributing directly or indirectly towards the impact. The notion of thematic evaluation calls to mind that of in-depth evaluation. Some thematic evaluations examine how the same impact (e. g. the competitiveness of SMEs) is obtained by several sorts of intervention (see Box 9 - situation 3) or how interventions financed in a specific domain such as Research & Development produce several kinds of impact (see Box 9 - situation 1). Thematic evaluations can be carried out within a single programme (the term is then equivalent to in-depth analysis) or transversally, on several programmes. These evaluations are described in Volume 5 of the MEANS Collection. Thematic evaluations are particularly useful for evaluating programmes from the point of view of Community priorities. They are grounded in the principle of "mainstreaming" whereby a priority or specific needs are systematically taken into account in all Community policies and particularly Evaluation design and management in the evaluated programme. This concept is used mainly for taking into account women's and men's needs in all policies, in order to promote the priority equal opportunities. This objective can be achieved if the effect of the intervention on the specific situation of women and men is taken into account from the design stage of the programme, through implementation, monitoring and evaluation, to the systems of indicators. Mainstreaming necessitates the organisation of society to guarantee a more equitable distribution of roles between men and women. the development of measures aimed at adapting Differentiating audit, monitoring a nd e v a l u a t i on The expectations placed on evaluation should not exceed what it can really contribute, and should not try to give it roles which it is not designed to play, such as audit and monitoring. Within the framework of structural policies, the Commission wishes to satisfy the requirements of both the Council and the Parliament as regards audit, monitoring and evaluation. The three exercises are similar in so far as they are judgements on public activity and can mutually enrich one another. However, any misunderstandings. These practices require different professional qualities, time frames and modes of organisation. distinguished they must to avoid clearly be An initial clarification concerns the point of view from which public action is judged. Audit verifies the legality and regularity of the implementation of resources. Monitoring verifies the sound management of the interventions and produces a regular analysis of the progress of the outputs. Evaluation juges programme implementation on the basis of the outputs, results and impacts it has produced in society. A second distinction concerns judgement criteria. Audit judges in terms of criteria that are known and clarified in advance (budgets, regulations, professional standards). Monitoring in terms of operational objectives to be achieved. By contrast, evaluation often has to start by choosing its judgement criteria. These are formulated on the basis of the objectives of the evaluated public action. However, experience shows, and the evaluation literature confirms, that the objectives of public action are rarely clear, quantified, verifiable and immediately usable as evaluation criteria. judges Preparing an evaluation and defining its content These basic differences distinguishing audit, monitoring and evaluation are presented in Box 10. Box 10 - Articulating audit, monitoring and evaluation Inputs Output Outcome / Outreach Control (of legality) Monitoring (of management) Evaluation of quality, management and effects References Compliance with administrative texts Good management standards > Meeting citizen's needs and solving problems Thus, the professional qualities and know-how required by the different exercises are not the same. Auditors have in-depth knowledge of pre- established criteria (legal or accounting knowledge, for example). They have authority and their behaviour inspires respect. They do not hesitate to use constraints and to publicly stigmatise errors and dysfunctioning for the purpose of dissuasion. Monitoring officials need to have recognised competence in terms of organisation, management and the establishment of information systems. They supervise and assist operators in so far as they provide tools and procedures that help them in their professional practices and also serve in a critical analysis of the conditions of implementation. Finally, because evaluation practitioners cannot rely on unanimously recognised criteria, they need the collaboration of public managers to collect relevant information, interpret it correctly and formulate a legitimate judgement. They cannot be effective if their interlocutors do not act willingly and implies a degree of confidentiality. faith, which often in good If the three exercises have intrinsic differences, they all make use of one another, owing to their complementarity. For example, audit and monitoring can identify apparent successes and failures, which evaluation will then try Evaluation design and management to confirm by means of an in-depth analysis. By contrast, an evaluation can show that the application of a legal or professional standard constitutes a particular factor of effectiveness or ineffectiveness. Interdependence exists between exercises of monitoring and evaluation, that monitoring indicators are useful for evaluation. in the sense Risks of conflict between the three approaches are nevertheless important. In particular, an evaluation must not be performed with audit methods or it could create contradictions which hinder the collection of information and the use of conclusions. From another point of view, no evaluation can replace audit or monitoring. For example, in case of doubt as to the regularity or professionalism of the implementation, it is necessary to carry out an inspection or an audit in order to take necessary decisions, and to wait for the situation to become less troubled before investing in an evaluation. Deciding to launch an evaluation Most evaluations which fit into the framework of European economic and social cohesion policy are of a mandatory nature. This should not, however, be cause for underestimating the role of the decision to launch an evaluation. This decision is an opportunity to frame the work to be undertaken, by means of a precise definition of its scope (i. e. what has to be evaluated), its schedule and, above all, its real motivations apart from the need to comply with regulations. formulating a clear and By the public authorities commissioning the evaluation will give it full political legitimacy. They appoint the people who are to participate in the evaluation process, by distributing responsibilities and setting the rules of the game. firm decision, When evaluation is mandatory, there is a risk of the commissioning authorities being tempted by the easiest solution, which consists of basing their decisions on established models, without enough thought to what they can really expect from the evaluation in the context of their programme and in the limits of the political schedule. As a result, evaluation could become a uniform, bureaucratic procedure. Several recognised authors in this field affirm that, on the contrary, each evaluation must be considered unique (Patton, 1986) because it has to meet expectations which change with time and the context, but also because experience has shown that there is no universal evaluation method. Defining the scope The decision to evaluate is an opportunity to define the limits of the programme in terms of institutional, temporal, sectoral and geographical dimensions. This is what is known as the scope of the evaluation or the "evaluanda". Defining the scope of an evaluation amounts to asking the question: "What is going to be evaluated?". The field of investigation must be delimited in at least four respects: institutional (European, national or local level), temporal (period under consideration), sectoral (social, industrial, environmental, rural, etc. ) and geographical (which part of the European territory, which region, town, nature reserve, etc. ). Within the framework of the Volume scope is, theoretically, delimited by financing of the programme to be evaluated, by the territory concerned and Evaluation design and management by the programming period. It is, however, useful to have first considered the following points: • Is the intention to limit evaluation to the funding of the programme or to include other national, regional or local funding which is, to a greater or lesser degree, directly related to the programme? • Is the intention to limit the evaluation to interventions in the eligible area or to extend observations to certain neighbouring areas which encounter similar development problems? • Is the intention to limit the evaluation to funding allocated to the programming cycle under consideration or to a certain extent to include funding of preceding cycles? It is necessary to stress the importance of a relatively strict definition of the scope of the evaluation. Experience has shown that during the evaluation process participants tend to want to examine, to a greater or lesser degree, everything concerning the programme. To reach conclusions, the evaluation must be reduced to an examination of the programme and its most essential interdependences with other public policies. ■^ The risk is particularly great in the case of ex ante evaluations which can turn into an exercise in forecasting or speculation which is far from the object of the evaluation. In this case, limiting the scope of the evaluation strictly to the programme proposal is recommended. A n n o u n c i ng t he m o t i v es a nd p l a n n ed use When an evaluation is launched, it is strongly recommended to specify the motives underlying the decision to evaluate, and to draw up a list of potential users. Even if the evaluation is compulsory, this exercise is a way of ensuring that no work is carried out unnecessarily. It is also important to specify how far the evaluation team is expected to involve itself. Are expectations limited to mere conclusions, or is there a hope of obtaining recommendations of an operational or strategic nature? The expectations are obviously different, depending on the stage at which the evaluation takes place (ex ante, intermediate or ex post). It is very important to note that, in order to obtain a useful evaluation, the decisions likely to be taken and which can be informed by the evaluation, must be stated as precisely as possible. Often commissioners, not wanting to influence the evaluation team too much, are reluctant to express in Preparing an evaluation and defining its content advance the changes they think should be made to the programme or their doubts concerning the effectiveness of a particular action. The intention is commendable: reveal nothing in advance to see whether the evaluation team reaches the same conclusions! Experience shows, however, that evaluation has little chance of documenting intended decisions if these are not known in advance by those who are going to collect and analyse data in the field. Socio-economic reality is in fact highly complex and the evaluation team is confronted with a large number of observations and possibilities for making improvements. It may therefore happen that attempts are made to verify hypotheses which may be interesting in the absolute, but are hardly realistic in the eyes of political officials or managers, given the existing margins for manoeuvre or emergencies. Distributing roles within the setting of an evaluation The evaluation setting refers to the roles and responsibilities within the evaluation process, and relations between evaluation, monitoring and management of the programme. Clear and fair rules of conduct contribute towards quality work and avoid blockages. The main choice relates to the steering of the evaluation, a role which includes, in particular, the drawing up of the terms of reference, the selection of the evaluation team and the validation of its report. Box 11 - The evaluation setting Commissioning public authorities Steering group 6. Decisions, directions communication Concerned groups Operators Evaluation team Evaluation design and management In the past, the steering of evaluations was often either entrusted to a single official or exercised collectively by the co-funders of the programme within the framework of formal meetings held infrequently and attended by In practice, the system numerous partners (Monitoring Committees). functions well when the responsibility is for steering informally, to a small group of officials from the entrusted, relatively administrations most directly concerned by the programme, sometimes supported by a few experts. Meetings must be held frequently (once a month at certain times) and the working procedures must be flexible and informal. the programme Schematically, (see Box 11), the setting of the evaluation can be reduced to the distribution of roles between the following five key players: the commissioners, the steering group, the evaluation team, the operators and the publics concerned. In some situations other entities need to be added. For example, in some cases the steering of an evaluation has successfully been organised on the basis of a global committee coordinating several sectoral committees (or committees created on a smaller geographical scale). The responsibilities of these entities are described by their respective roles during the six main steps comprising the evaluation process: • Step 1: The public authorities, commissioners and future users of the evaluation formulate their instructions to a steering group. This step is materialised at least by a paragraph in a programming document. It could, in due form, as recommended in the concluding chapter of this Volume. in the future, be set out in a mandate • Step 2: Tne terms of reference are drawn up by the steering group on the basis of the mandate given to it. This document usually serves in the call for tenders and in the selection of the evaluation team. The steering group will interact with the evaluation team as much as is necessary to ensure that the team does not diverge from the guidelines in the terms of reference and that it works towards answering the questions asked. • Step 3: The evaluation team collects secondary and primary information from the operators and publics concerned, and then analyses them. This is the longest step. • Step 4: A report which can be qualified as "technical" is written by the evaluation team. It includes conclusions and, where relevant, preliminary recommendations for the steering group. • Step 5: The conclusions are partly or entirely validated by the steering group. This group furthermore formulates recommendations to the public Preparing an evaluation and defining its content authorities in the evaluation report which either includes the technical report or presents it as an annex. • Step 6: The authorities use the evaluation by disseminating it among the public concerned and/or by deciding on new guidelines that the administrations will have to implement. This crucial step is often isolated from the evaluation process as such, because the steering group can be dissolved after submitting its report and is therefore unable to follow up on subsequent decisions. Nevertheless, it is possible to ensure that at least some follow-up occurs, by including a commitment in the initial mandate by which the authorities undertake to request, if not the steering group then at least its President, to present the conclusions and to attend the meeting at which decisions are taken. towards •^ It is important to emphasise that, in the past, evaluation settings gradually converged intermediate evaluations. This was not, however, the case with ex ante evaluations. Time constraints and the difficulty of mobilising protagonists with regard to a programme that did not yet exist, may explain the absence of a steering group, without necessarily justifying it. the above schema for ► The same shortcomings are found in ex post evaluation although the same difficulties do not exist. This type of evaluation is often requested by an administration without the necessary care being taken to clarify expectations at the most important decision-making levels or in the steering of the process. Formulating the evaluative questions List of possible questions Probably the most fundamental aspect in the design phase of an evaluation is the formulation of evaluative questions. It is these questions which constitute the core of the evaluation exercise. Possible questions include: descriptive questions intended to observe and measure changes (what happened?), causal questions which strive to understand and assess relations of cause and effect (how and to what extent is that which occurred attributable to the programme?) and, finally, normative questions which apply evaluation criteria (are the results and impacts satisfactory?). Ideally, any evaluative question must encompass these three dimensions and have the following qualities: Evaluation design and management The question must correspond to a real need for information, understanding and/or identification of new solutions, otherwise it will merely be an "alibi question", aimed, for example, at endorsing a decision already taken. If a question is of interest in terms of new knowledge only, without an immediate effect on decision-making or public debate, it is more a matter of scientific research and should not be included in an evaluation. The question concerns an impact, a group of impacts, a result or a need. That is to say, it concerns, at least partly, elements outside the programme, notably its addressees or its economic and social environment. If a question concerns only the internal management of resources and outputs, it can probably be treated more efficiently in the course of monitoring or audit. The question concerns only one judgement criterion. This criterion may sometimes be difficult to meet, but experience has shown that it is a key factor in the usefulness of the evaluation. Without judgement criteria clearly stated from the outset, the final report rarely provides conclusions. Example: Some evaluative questions relating to the European Structural Funds Has support to scientific and technical centres initiated the catching up of regional SMEs in terms of innovation? Has the programme created or maintained an adequate number of jobs compared to its cost? Do the outputs and results of the programme correspond to the specific needs and expectations of women? Which measures have the most positive (or negative) impacts on the environment? Evaluation criteria When they include a judgement criterion, evaluative questions fall primarily into one of the following four categories: those related to the relevance of the programme, those related to its effectiveness, those related to its efficiency and those related to its utility. These four main categories are represented in the following diagram. ■^ M The term "relevance", in the context of an evaluation, refers to the appropriateness of the explicit objectives of the programme in relation to the socio-economic problems it is supposed to address. In ex ante evaluation, questions of relevance are the most important because the focus is on the choosing the best strategy or on justifying the one proposed. In intermediate Preparing an evaluation and defining its content evaluation, the aim is to check whether the socio-economic context has evolved as expected and whether this evolution calls into question a particular objective. Box 12 - Main evaluation criteria Needs Problems Issues II || Society Economy Environ­ ment Programme Objectives Outreach ,—t- Ζ Outputs Evaluation Relevance Efficiency Effectiveness Utility Η ► The basic questions of intermediate evaluations and, more particularly, of ex post evaluations, concern the effectiveness and efficiency of the interventions implemented and of the entire programme. As far as effectiveness is concerned, the question is knowing whether the objectives formulated in the programme are being achieved, what the successes and difficulties have been, and how appropriate the solutions chosen or the influence of confounding factors outside the programme are. In the context of evaluation, efficiency is assessed by comparing the results obtained or, preferably, the impacts produced, and the resources mobilised. In other words, are the effects obtained equal to the inputs? The terms "efficiency" and "effectiveness" are, moreover, commonly used in a different sense by managers who will seek, within the framework of monitoring, to analyse outputs rather than results or impacts. In this respect it is important to emphasise that questions concerning the performance of a programme are increasingly frequent. This notion encompasses those of effectiveness and efficiency as defined both for monitoring and evaluation. Thus, it is a fairly large set of questions that will be grouped under this heading. The criterion of utility judges the impacts obtained by the programme in relation to society's needs and economic issues. Utility is a very particular Evaluation design and management evaluation criterion in so far as it makes no reference to the official objectives of the programme. It may be judicious to formulate a question of utility when the objectives are badly defined or when there are many unexpected impacts. This criterion must nevertheless be used with caution to ensure that the evaluation team's selection of important needs or issues is not too personal. Some North American authors have advocated this type of goal free evaluation. The four categories of questions can be characterised simply and concisely as follows: • Relevance: To what extent are the programme objectives justified in relation to needs? Can their raison d'être still be proved? Do they correspond to local, national and European priorities? • Effectiveness: To what extent have the expected objectives been achieved? Have the instruments used produced the expected effects? Could more effects be obtained by using different instruments? • Efficiency: Have the objectives been achieved at the lowest cost? Could better effects be obtained at the same cost? • Utility: Are the expected or unexpected effects globally satisfactory from the poini of view of direct or indirect addressees? These four criteria are not exclusive. For example, a typical question might be: Will the effects obtained last after the end of the programme, even if there is no more public funding? The criterion here will be that of the sustainability of the impacts produced. Other criteria such as equity, coherence, synergy, reproducibility, etc. may also be used. Probability of an answer Once the evaluative questions have been identified, their evaluability has to be considered. A prior assessment has to be made of whether the evaluative questions are likely to be answered, given the available resources. In concrete terms it is a matter of ensuring that an evaluation team, with the available time and resources and using appropriate evaluation tools, will be able to find all the elements needed for the construction of credible answers to the questions asked. In order to know whether it is possible to attain an acceptable degree of credibility, an evaluability study can be carried out. This point is considered in the last part of the book (page 199) devoted to possible improvements to existing practices. Preparing an evaluation and defining its content For each evaluative question it is recommended to check, even very briefly: • whether the concepts are stable, • whether explanatory hypotheses can be formulated, • whether available data can be used to answer the question, without any further investigation, • whether access to the field will pose major problems. A number of factors can make a question very difficult to answer, e. g. if the programme is very new, if it has not yet produced significant results or if the available data are totally inappropriate. These reasons may lead to the decision not to undertake the evaluation, to postpone it, or to ask more realistic questions. Selecting priority questions Because trying to answer all the questions in depth would be too costly in time and effort, it is always recommended that the evaluative questions be graded. Box 13 - Selecting priority questions Stakeholders Types of decisions to be taken Political context ^ ^ J Initial list of evaluative questions: Probability of answer Probability of use v^ f > Priority questions Evaluation design and management In order to do so, it is necessary to assess the issues associated with each of these questions (knowledge, negotiation, decision, and communication). The probabilities of finding answers to the questions and of using these answers must guide the choice of priority questions. This choice will be far more "successful" when potential users have been consulted and have appropriated the selected questions. It is obviously better to obtain slightly inaccurate answers to important questions than accurate answers to questions that nobody asks and which a fortiori will never be taken into account. As Box 13 shows, a question is considered a priority if the probability of obtaining an answer and of this answer being used is high. For this purpose, the following questions must be asked: • Will the conclusions be used? By whom? For what purpose (deciding, debating, informing)? When? • Is it politically appropriate to perform such an evaluation at this particular time or in this particular context? Is there a conflictual situation which could compromise the success of the exercise? • Has a recent study not already answered most of the questions? • Are evaluation priorities not too unstable? If necessary, these points can be studied prior to the evaluation (see page 185). Ordering an evaluation The terms of reference reflects the decision to launch an evaluation, in terms which are sufficiently precise so that several evaluation teams can compete and so that the work may be performed in good conditions. The drawing up of the terms of reference therefore requires particular care. The terms of reference recapitulates the origin, scope and aim of the evaluation, as well as the distribution of roles. It describes the content of the programme to be evaluated and specifies information which is already available and will be made available to the evaluation team. It states precisely a limited number of questions to which the evaluation must provide answers. It also outlines the evaluation method and sets the steps in the schedule. The document may indicate the budget earmarked for this purpose. The evaluation team is chosen after a call for tenders. Independence and competence are two essential criteria in the selection of a team. Carefully writing the t e r ms of reference The terms of reference is the document which serves as the basis of a contractual relationship between the commissioner of an evaluation and the team responsible for carrying out the work. Constructing the terms of reference clearly is a vital step when an evaluation has to be performed by outside consultants. This work is nevertheless equally important when part of the evaluation is performed in-house, although it is often believed to be unnecessary in such cases. The terms of reference may concern either the evaluation operation as a whole (when it is entrusted to a single team) or a part of the research work programmed in the evaluation project (in-depth analysis of an evaluative question). Example: an evaluation consisting of several terms of reference Within the framework of their development programme, the authorities of a region chose to divide up the work into separate batches according to the technical capacities required to carry it out: (1) questionnaire survey on a sample of addressees, (2) interviews with managers and operators and (3) economic expertise. Other evaluations are broken down by territory (e. g. rural development / urban development) or by sector of intervention (e. g. : technology / training / tourism). Note that this type of division means that the commissioner is fully responsible for methodological choices and for the synthesis of results. Evaluation design and management The terms of reference is a 5 to 20 page document, generally complemented by administrative annexes. Its contents are described in Box 14. Only the points which have not already been considered are explained below. Box 14 - Standard layout of the terms of reference 1 - Regulatory framework 2 - Scope of the evaluation 3 - Who is going to use the evaluation and what for? 4 - Evaluative questions and criteria 5 - Knowledge and available data 6 - Method (guidelines) 7 - Evaluating setting 8 - Schedule 9 - Suggested budget 10 - Required qualifications 11 - Structure of the proposal Regulatory framework of an evaluation For the benefit of the team responsible for the evaluation, it is necessary to recall the context in which it is situated. Although the legal and contractual foundations are set out in the regulations of the European Union and in each programming document, it is advisable to refer to them. The terms of reference must specify who initiated the evaluation project and, where relevant, who was involved in formulating the order for the evaluation, as well as the motives for the decision to evaluate, if they are not purely regulatory. Moreover, the aim of an evaluation, that is to say, the reasons for which an evaluation is requested, must be clearly stated. Is the intention a change of direction? Why? Is the intention to modify the monitoring or arbitration procedures? Is the intention to reallocate funds? Available knowledge The terms of reference contain a review of the current state of knowledge on the programme and its effects. The elements in this inventory include extracts or references of programming documents, lists of previous analyses and evaluations with important extracts, a description of the Preparing an evaluation and defining its content monitoring system, quantified indicators, and the various reports and data bases available from the services managing the programme. is essential for the candidate teams to adjust This inventory their methodological proposals in relation to the size and nature of the data collection and interpretation work which will have to be organised during the evaluation exercise. Principal outlines of the method Each evaluation uses an ad hoc method constructed with the commissioner's agreement, on the basis of the selected evaluation team's proposal. At the terms of reference stage, the choice is generally made not to describe the method completely, but to outline it, as shown in the diagram in Box 15. The aim is not to confine the candidate teams to strict methodological constraints, but to state a small number of requirements that the commissioner considers essential for obtaining answers to the evaluative questions. Should an inventory be made of evaluations and research reports on similar subjects? Will there be a survey on the addressees or not? Will there be a comparative analysis or not? Should the evaluation team look for judgement references or will these references be provided by the steering group? The choice is generally made to maintain sufficient flexibility to allow candidate teams to differentiate themselves in terms of the relevance and clarity of their methodological proposals. Box 15 - Planning an evaluation method at the terms of reference stage First evaluative questions -— r u ai c ^ ^ — ^ - E x p l a n a t i on of questions Evaluability study, methodological proposals Priority questions Verification of compatibility with budget Evaluation design and management Schedule The evaluation schedule is established by taking into account various constraints, especially those concerning the decision-making schedule. As Box 16 shows, it is also necessary to integrate the main deadlines, generated by the procedures of calls for tenders and by the phases of primary data collection, and which cannot be reduced. Box 16 - Constraints bearing on the evaluation schedule D e c i s i o n - m a k i ng s c h e d u le P r e p a r a t i on Decision Evaluation L a u n ch Terms of reference Call for t e n d e rs Selection O u t p u ts Validation Dissemination Utilisation Indicative budget The financial resources available for a study partly determine the limits of an evaluation, especially the number of issues that will be addressed and the extent to which they will be analysed. When possible, the evaluation budget should be established by the commissioner by considering the interest and difficulty of the questions asked. To establish a suggested Preparing an evaluation and defining its content budget, a "top-down" or a "bottom-up" approach, or a combination of the two, can be adopted. to 1. 4% (Review of ongoing evaluation Function A "top-down" approach consists of determining the evaluation budget in relation to the budget of the programme to be evaluated. Some references can be used in this approach. A review of evaluations conducted in the framework of the Irish CSF for the 1994-1999 period found that the annualised cost of evaluation in relation to annual OP expenditure ranged from 0. 6% in the Community Support Framework (CSF) for Ireland, 1994-1999, published 1998). These figures are far higher than current spending devoted to the evaluation of European Structural Funds, which is globally around 0. 02% of the programmed funds. In certain contexts, the cost of the evaluation was as high as 10% or even 50% of the cost of the evaluated action. Such percentages can be justified for highly innovative interventions or for experimental operations, but it is not relevant to refer to the cost of the evaluated intervention. A better reference is the cost of the programme which will be decided on if the evaluation concludes that the experiment was successful. The main factors which can justify a small or large evaluation budget are as follows: • a major, expensive policy which is both simple and routine (reduced %); • the programme is frequently evaluated (high % for all the evaluations together, but reduced % for an individual evaluation); • a regulatory policy with significant effects and minimum costs (high %); • an innovative or experimental programme (high %). The "bottom-up" approach consists of estimating the probable cost of each evaluated question. The cost of an evaluation consists of two main categories of spending: general costs (structuring the evaluation and producing a report) and costs concerning the collection and analysis of data. General costs include expenditure related to the analysis of programme documents and monitoring data, an overview of all documentation, interviews and meetings with managers and the main stakeholders, and the establishment and presentation of intermediate and final reports and discussions on them. The second category of spending concerns case studies, focus groups, individual interviews, questionnaires, statistical analyses, etc. The cost and duration of the implementation of some evaluation techniques can be estimated as follows: Evaluation design and management Box 17 - Indications of the cost of some evaluation techniques Questionnaire surveys Surveys based on individual interviews Surveys based on focus groups Surveys based on case studies Statistical analysis Expert Panels Use of an econometric model Development of an econometric model Cost (person / month) Duration Medium Medium Low Medium Low Medium Low High XXX XX XX XXX XX xxxx XX XXXXXX Χ = approximately 1 month In reality, the cost of evaluating a programme varies considerably from one case to another. The suggested budget is often fixed by combining bottom- up and top-down approaches and taking into account significant factors such as: • the number of questions to consider in depth. In general, the cost of collecting and analysing data is proportional to the number of questions studied in depth, but it is sometimes possible to group together some of these analyses; • expectations in terms of credibility. The soundest evaluation methods tend to be the most expensive ones, although this is not necessarily the case; • the quality of the monitoring system and the existence of recordings of outputs and results; • the cost of time devoted the production of data within the administration. These costs are often hidden and should not be underestimated. to steering group meetings and to S e l e c t i ng t he e v a l u a t i on t e am The evaluation team is selected by taking into account its independence and profile and the methodological qualities of its proposal. Preparing an evaluation and defining its content Independence of the evaluation team in a context of joint decision-making. In particular, The independence of an evaluation or of the evaluation team is a strict requirement the evaluation team must not find itself in a relationship of subordination with the managers of the evaluated programme. This is especially important in intermediate evaluations or when the evaluation is supposed to judge the success of a programme. In the last chapter of this Volume it is shown that the concern for independence must not, however, prevent the evaluation team from establishing close relations with the managers. The principle of independence must also be applied with discernment when it is more cost effective to entrust part of the work to the administrations responsible for monitoring the programme. This is the case, for example, when certain management data have to be found and processed for the first time. In some circumstances, skilful mixes of internal and external contributions are probably more productive than entrusting the entire exercise to outside professionals. The evaluation team should nevertheless be in a position to control the impartiality and quality of the entire evaluation. The independence of the evaluation is not entirely guaranteed by the fact of it being entrusted to an outside team. The members of an evaluation team may be subjected to multiple pressures during the programming of the work, the collection of information and the validation of the report. The team does not always have enough weight to resist pressure, especially if it lacks familiarity with evaluation, cannot rely on sound data or does not have the time to make complete analyses. In all cases, the steering group can play a very useful complementary role in improving the independence of the evaluation, particularly if it includes one or more outside experts. In some cases Structural Fund programmes undergo a series of evaluations by the same team, for example several in-depth analyses and an is usually entrusted to a firm of intermediate evaluation. The work consultants or a research centre, but in some cases evaluation units are formed for this purpose, alongside the national administration (e. g. Ireland). In these situations, the evaluation team works for several years in contact with the programme managers. Whether the team is internal or external, it manages far more successfully to maintain its independence when there is a well-structured steering group comprising outside experts. A particularly interesting solution consists of calling on evaluators from outside the country under consideration, even if linguistic differences often limit this type of practice. Evaluation design and management Profile of evaluation teams In the field of European Structural Funds, more and more organisations are present in the evaluation market. The service providers who are selected belong to local, national or international consultancy firms. The commercial sector accounts for most of the interventions but the presence of university research centres is also significant. Each evaluation team has its own profile, depending on its basic profession. An important distinction relates to the difference between evaluation and research. The purpose of research, particularly fundamental research, is mainly to "advance" knowledge; its aim is therefore cognitive. Theoretically, it is not concerned with changing behaviour or, at least, its approach is not aimed at producing immediately useful results. Although evaluation uses the same methodological rules as research (in sociology, economics, etc. ), it is primarily intended to directly serve actors and to make judgements. Its aim is therefore normative and instrumental. Its main vocation is to be useful and to be used. This basic distinction between the "research" and "evaluation" approaches explains why, even with precise terms of reference, opting for a consultancy firm or a university laboratory can have implications in terms of the approach and therefore of the results of the evaluation. Academics have the advantage of being perceived as independent and highly credible owing to their academic requirements. On the other hand, private firms are often more readily available as far as time is concerned and are more concerned with meeting the commissioner's expectations. When the steering group puts out a call for tenders, it must specify the required competencies. While determining the main lines of the method, an evaluability study may define these competencies and specify whether the evaluation needs experts specialised in the fields of intervention of the labour market, rural programme (regional sciences, macro-economics, development, etc. ) or whether the need is rather for general evaluation know-how or specific methodological skills (survey, audit, information systems, causal analysis, etc. ). In general, each consultant or academic has his or her strengths in certain technical domains of evaluation and his or her weaknesses in others. Cooperation within a mixed team is often a solution, provided that the coordination is effective. If the candidate teams are versatile enough, it is possible to maintain a competitive situation and to avoid using the same experts repeatedly. This situation is the best for ensuring independence. Preparing an evaluation and defining its content Methodological qualities of the proposal The selection of a team is never independent of the criterion of price. In this case, the evaluation method must be determined in great detail and the candidate teams pre-selected according to their technical skills. This method forces the commissioner and the steering group to carry out a considerable amount of methodological work upstream. When possible from an administrative point of view, it is preferable to work the other way round, that is to say, to determine a budget or volume of work, to describe only the main lines of the method in the terms of reference, and then to select the team which proposes the most interesting method. The selection of the team will then require the ability to judge the methodological quality of a tender. This quality is not necessarily related to the use of obscure terminology and complicated models. The proposed method must, above all, be suited to answering the questions asked in the terms of reference. The steering group can lead this work by checking the points presented in Box 18 for each of the tenders. Box 18 - Examining the methodological quality of a proposal For each evaluative question Question 1 Question 2 The proposition of the candidate team: Does it include the collection of information sufficiently relevant Is it based on rigorous analytical techniques? Is it able to clarify the evaluation criteria in an impartial manner? 1st it likely to produce credible findings? ++ - + + + + +/- + Was the respective of the questions well understood? importance ++ Ill Performing an evaluation Analysing the logic and content of the programme As a first step, the evaluation team reconstitutes the logic of the programme, i. e. the needs to be fulfilled, the strategy, the objectives, the expected effects, the inputs, and the measures taken to implement and monitor the programme. An analysis of the programme logic provides an indispensable framework for the study of results and impacts. It is therefore a key step in any evaluation. In an ex ante evaluation, this analysis constitutes the bulk of the work because it produces a judgement on the validity and coherence of the programme and its strategy. to recommendations to improve the monitoring system and management methods. In a mid-term evaluation, it leads In an ex post evaluation, this analysis is used to clarify objectives when these were not clearly expressed. This clarification is necessary to structure the collection of information and analyse impacts more effectively. In general, this step is used to verify the coherence of objectives within the same intervention, and the coherence of the same objectives in relation to those of other interventions. The programme content is examined in light of the budget allocated, the modes of intervention chosen (projects selected, participants' responsibilities), the impacts identified and the monitoring system put into place, in which the system of indicators plays an important part. The logic of a programme is defined as the connections between all its elements, i. e. the needs to be met, the strategy chosen, the objectives set (global, specific, operational), the inputs, the measures taken to implement it, the outputs and the expected effects (results and impacts). The articulation of all these elements is, by custom, presented in the form of a logical series of steps called the "programming cycle", as shown in Box 19. Evaluation design and management Box 19 ­ T he orograr nmi ng cycle r Needs issues Problems Objectives ■ *­ ' r Specific objectives ­*­ J Operational objectives ■ ♦­ V Resources. Implementation system *­ *­ »­ Λ ì Expected impacts 41 Expected results 1 1 Expected outputs ;> J ­^ Ex ante evaluation examines the logic of the proposed programme and judges whether it is clear, complete and coherent. This is its main purpose. The following pages therefore present a key step in the realisation of an ex ante evaluation. In the second stage, ex ante evaluation forecasts impacts. M ► In mid-term and ex post evaluation the first step may also be devoted to the programme logic, as the diagram in Box 19 shows. This is obviously necessary if points in the initial logic are obscure or poorly justified, that is, if the ex ante evaluation was not performed correctly or if its conclusions were not taken into account. It may happen that the objectives were amended midway. In that case it is necessary to present a new programme logic at the beginning of the evaluation report and to check whether the overall coherence of the programme has been maintained despite these changes. More often, the needs to be met or the problems to be solved have changed during the implementation of the programme. If necessary, the intermediate or ex post evaluation re-examines the programme logic in light of this new context. When the evaluation team examines the logic of a programme, it starts by analysing the official texts (the proposed programme in the case of an ex Performing an evaluation ante evaluation, otherwise the approved programme). The "official logic" has to be clarified and its coherence evaluated. If the programme managers agree, the evaluation may be carried out in the form of a series of interviews and/or meetings in which the participants explain their understanding of the strategy, objectives and expected effects. The evaluation team then has to produce, in a participative manner, a "reconstructed logic". The first option is quicker and has the advantage of adhering to official texts, which makes it possible to report on a legitimate basis. The disadvantage is that the evaluation may reveal inconsistencies or inaccuracies which are already known to the decision-makers. The second option requires more work but produces more useful conclusions for improving the programme. Assessing the validity of the p r o g r a m me The rationale of a programme is to satisfy needs, solve problems, avoid threats and grasp opportunities. The needs or problems considered are of a social, economic or environmental nature. They concern a territory as well as one or more groups of people in that territory. If public funding has to be justified, the needs concerned must be those which cannot be satisfied by the market and the problems must be those which cannot be solved by private initiative. Prior appraisal of socio-economic problems -4 The evaluation team's initial and essential task is to re-examine the social the needs or development problems programme. The logic that supports this justification has to be investigated thoroughly and systematically, especially in the case of an ex ante evaluation. the existence of justify that The evaluation details the socio-economic handicaps and development problems of the region or Member State under consideration that warrant the attribution of support within the framework of the Structural Funds. The evaluation team must also establish a position on whether a region's needs actually correspond with the general objectives of European cohesion policy. H In the case of a mid-term evaluation, the evaluation team must also examine the extent to which socio-economic needs and development problems are evolving, whether they are worsening or improving, and try to Evaluation design and management explain why. The object of this questioning is above all to know whether private initiative could satisfy these needs or solve these problems, in the place of, and as effectively as, the considered programme. the example below shows, depending on As the period under consideration, certain problems can be requalified as being, to a greater or lesser degree, a matter of public intervention. Example: A new examination of the rationale of interventions initiative; those which offset shortcomings In thè mid-term evaluation of a set of programmes on a national level, the evaluation team sortee the interventions into four main categories, depending on the way they could be justified: those which produced public goods which could not be provided by private in the market; specific subsidies; and distribution measures. Within each category, the evaluation team noted the validity of each intervention by means of the same assessment grid. On the basis of this evaluation, it appeared that certain measures could be considered as no longer having any justification. Among them were certain incentives for the tourist sector, which were no longer of use due to the fast growth of the industry. By contrast, other measures had become more relevant, for example the improvement of inter-urban roads and the fight against dropping out of school. Relevance of the strategy and objectives -4 Establisning the strategy of a programme amounts to prioritising actions on the basis of the most important needs to be fulfilled, the importance of problems to be solved, and the probability that envisaged actions will be successful. The formulation of a strategy amounts to selecting and establishing a hierarchy of objectives and fixing the extent of their ambitions. Not all territories have the same development strategy. An ex ante evaluation examines whether the strategy is suited to the current context and to its probable evolution. It is therefore advisable to verify whether the programme has been constructed on the basis of objectives that have been classified according to their degree of importance. The evolution of the context is characterised by the issues that the strategy has to deal with. The strategy could be either active or reactive, depending on the nature of the issue. There could, for example, be a threat that had to be avoided, a possible economic obstacle that had to be removed, a weakness that had to be taken into account, an opportunity that had to be grasped, a development potential that needed to be stimulated, a strength Performing an evaluation which could be relied on, etc. The strategy is composed of defensive and offensive objectives. In the first case, the programme is aimed mainly at combating a weakness or avoiding a threat. In the latter case, it is aimed at building on a strength or taking advantage of an opportunity. An evaluation considers the strategy of a programme by trying to ascertain whether it is primarily offensive or defensive, or possibly mixed. A purely defensive strategy comprises the risk of having to confront new problems in the long run by introducing the idea that the territory or group concerned cannot develop without outside support. A purely offensive strategy can cause important threats, which are potentially disastrous for development, to be neglected. A technique which can be used to evaluate the strategy of a programme is (Strengths-Weaknesses-Opportunities-Threats) analysis. This SWOT technique, with a group composed of as many stakeholders as possible, creates a formal structure which helps to clarify and specify in detail the strengths and weaknesses of a given (regional or sectoral) situation, and the existing or potential opportunities and threats. The tool is described in more detail in Volume 3 of the MEANS Collection. Example: SWOT analysis SWOT analysis was used during the formulation of the Objective 6 regional development plan in Sweden. The results produced by means of this technique helped trends to highlight major environmental, structural and democratic characteristic of the context in the region under consideration. The major factors identified during this analysis are synthesised below: Strengths: Large protected natural areas and substantial natural resources. Weaknesses: Isolated region, low population density, vulnerability to economic and budgetary fluctuations, structural weaknesses of firms. Opportunities: Tourism, quality of forest resources, fresh-water fish farming, mineral prospection. Threats: Population loss, shortage of jobs. The recommendations drawn from this analysis were used to orient choices made in the Regional Development Plan. This tool has since been used in many regions. The evaluation team also examines the ambition of the programme in relation to the needs and problems identified. The aim is to clearly establish whether the programme was designed to fully satisfy needs or definitively solve problems during its life cycle, or whether it was simply considered to be an Evaluation design and management initial step to be prolonged into the future, or a test to provide a subsequently generalisable solution. Comparable programmes can equally be used to assess the degree of ambition of the programme. It is important for the evaluation to examine the ambition of the objectives one by one. This work is inseparable from an evaluation of the achievement of objectives since it is easy to fulfil objectives which are not very ambitious. The evaluation of the programme's effectiveness will, as a consequence, be suitably adjusted. Checking the clarity and coherence of objectives Clarity of the objectives The achievement of an objective after a period of time is easier to recognise when the latter has been clearly identified. A perfectly clear objective is therefore verifiable. In the first of the following examples, the objective is relatively easy to verify, consisting of stimulating the growth of new business creation. This objective is associated with a measurable indicator, of which the significance is perfectly clear (rate of new business creation per 10,000 inhabitants) and which was fixed in the form of a quantitative value (+ 20%). The following examples show quantified objectives which are more difficult to verify, as well as non-quantified objectives, some of team has to verify which are easier systematically to judge the clarity of the objectives. than others. The evaluation Examples: Verifiable and non-verifiable objectives Easily verifiable objectives At the end of the programme, the ratio of new business creation for 10,000 inhabitants has increased by 20%. At the end of the programme, the main towns in the region will have improved their standing when opinion polls ask the question "where would you like to live?" Objectives which are difficult to verify The programme will contribute to a 1 % increase in the per capita GDP. The programme will enhance the competitiveness of regional enterprises. When an objective is difficult or even impossible to verify, the evaluation team has to perform an additional task before moving on to the following steps in the evaluation. In the case of an ex ante or mid-term evaluation, the evaluation team -4 M collects al! relevant opinions to make recommendations which will help clarify the objectives. Performing an evaluation H ► In the case of a mid-term or ex post evaluation, the evaluation team tries to ascertain whether the programme managers or operators have redefined or clarified the initial objectives during implementation. Interviews should indicate whether there is consensus on the interpretation of the programme logic. Efforts to make an objective measurable by means of an indicator often help to clarify it. Coherence of objectives within a programme The evaluation checks whether the programme objectives are coherent, that is, whether the objectives of the measures follow on logically from the specific objectives defined for the axes or sub-programmes. This level concerns the internal coherence of the programme. This is defined as an organisation of the objectives so that the objectives on the lower level constitute a logical contribution to those on the higher level. This is what is understood by the notion of a hierarchy of objectives that is represented by the following diagram. Box 20 - Hierarchy of objectives P r o g r a m me Priority axis: C o m p e t i t i v e n e ss of enterprises Technological services to enterprises Technological c e n t r es y g - :: Ε σι Catching up the level of development Enhanced competitiveness of enterprises Enhanced competitiveness of enterprises Better technological services for enterprises Better technological services for enterprises New technological networks New technological networks New technological centres New technological centres Adapted from: EUROPEAN COMMISSION (1995) intermediate evaluations - Structural Funds - Common guidelines for monitoring and Evaluation design and management To verify the internal coherence of the programme, the objectives tree technique may also be used. This technique serves to determine the contribution of each objective on a lower level to the global objective of the programme. It consists of a diagram in which the most global objective of the programme is placed on the far left, and the most detailed objectives on the far right. Each global objective can thus be broken down into several sub-objectives which, in turn, are broken down into objectives on a more specific level. This tool is also used to identify and grade objectives. In practice, neither the objectives of each measure (specific objectives) nor the relations between the different levels of a complex programme, are articulated so clearly and explicitly. The need to clarify the different levels of programme objectives is then even more necessary as the impact of each type of intervention is uncertain. The objectives tree is used to specify the objectives and to establish a clear hierarchy between them by clearly internal coherence of the programme, both vertically, between the different levels of objectives, and horizontally, between the objectives of different measures on the same level (see Volume 4 of the MEANS Collection as well as Volume 5 for numerous thematic examples). the degree of identifying Example: Coherent and incoherent objectives Two measures aimed at "offering better technological services to SMEs" and at "offering possibilities of venture capital financing to SMEs" have objectives which are consistent with the objective on a higher level: "increasing the competitiveness of enterprises". By contrast, these objectives would be incoherent if the objective of competitiveness were stated without any measure contributing clearly towards it, or if measures on the lower level were juxtaposed without having anything in common. Although the objectives tree is used to identify and grade objectives, it cannot be used to identify causal links between them. In order to do so, another tool is used: the "logical diagram of expected impacts". This tool schematises the causal links between the outputs of the programme, situated on the left, and the specific impacts up to the global impact or impacts, situated on the right. The logic behind the use of an objectives tree is that of the decomposition of a global objective. By contrast, the logic behind the diagram of impacts is one of recomposing expected impacts or objectives (See Box 21 and Box 41). Performing an evaluation Box 21 - Example of logical diagram of expected impacts Measures Intermediate impacts Global impacts Tourism Agriculture Training Increased viability of enterprises Strengthened economic fabric Improved employability Improved productive system Structurally balanced employment Coherence between the objectives and other public policies It is important to recall that the programme represents only a part of the public policies implemented in a given region, and that the resources allocated to it account for only a small part of local public spending. Some of the needs that the programme can legitimately try to cover are also targeted by other policies, whether regional, national or European. It is therefore necessary to ensure that the programmed activities neither contradict nor unnecessarily duplicate these other policies. Thus, the external coherence of the programme must be assessed by examining whether the objectives of the programme are in contradiction or not with the other policies of the European Union, for example those relating to competition or the organisation of agricultural markets. •^ Community regulations also stipulate that information on coherence, with regard to economic, social and, where relevant, regional policies of the Member States, must be provided in each programming document. H The main risk for external coherence is the appearance, midway, of other interventions which duplicate or contradict the evaluated programme. If necessary, mid-term evaluation reviews similar policies and assesses whether they extend, complement, overlap or contradict the programme underway. This study consists of interviews with programme managers and operators and with anyone else who is in a position to provide reliable information. Evaluation design and management Example: External coherence within the framework of local development interventions The conditions governing the attribution of Common Agricultural Policy support, as part of the organisation of agricultural markets aimed at maintaining rural areas, are consistent with the objectives of economic development programmes for rural areas financed by the Structural Funds. To avoid any duplication, it is important for 5b programmes to be entirely devoted to the maintenance of a viable agricultural community in regions in difficulty. This includes the maintenance of population levels in disadvantaged rural areas and the maintenance and development of agricultural activity in these areas. Box 22 - Highlighting e of a cross-imp Kternal c o h e r e n ce by m e a ns tacts matrix Programmed activities Support for diversification Support for productive investments Support network Support for services to enterprise Strengthening of regional R&D Promotion of a new image of the region Other policies National promotion of exports European promotion of SMEs Regional promotion of foreign investments + ? - = + + + + + + = = ? + + + = = = : no interaction ? : risk of duplication + : complementarity - : risk of conflict One of the techniques which can be used to clarify relations with other policies is the establishment of a matrix of cross-impacts, (with lines consisting of programme components and columns consisting of other policies). This matrix will reveal complementarity, duplication and contradictions which are likely to arise between the impacts of the programme under consideration and other existing policies (see Volume 4 of the MEANS Collection). Use of this tool is shown in Box 22. In this example it can be seen Performing an evaluation that numerous cases of synergy exist between the programmed activities and the other implemented policies, for example at the national level. For instance, "support for diversification" and "promotion of foreign investments" reinforce each other. On the other hand, the latter policy, which relies on exogenous inputs, is in conflict with a programme promoting endogenous development through the creation of a support network for SMEs. Analysing the modalities of p r o g r a m me m a n a g e m e nt The implementation process involves a series of organisational and budgetary measures, i. e. : the establishment of the budget and the selection of projects, financial channels, modalities of management and coordination, and systems of monitoring and reporting. This stage of the evaluation involves the examination of budgetary measures, and operational instructions or guides disseminated to ensure that the programme is implemented with the greatest chance of attaining its objectives. The actual analysis of the appropriateness of the conditions of implementation are made at a later stage in the evaluation process, particularly when the reasons for the achievement or not of a given impact are sought. •^ In ex ante evaluation, the analysis will be limited to the first measures described in the programme proposal, that is, primarily the budget, the tools chosen and the main rules established. ►< ► In mid-term or ex post evaluations, it is possible to go much further and examine the details of administrative measures, the distribution of roles between managers and operators, the implementation documents given to operators, the selection criteria for projects, etc. Distribution of budgetary inputs An evaluation focuses primarily on public funds allocated by the authorities that "co-decided" the programme. Additional private investments must not be put on the same level; nor must public funds from other funding authorities, which are expected but have not been formally attributed. The programme distributes funds in the form of budgets allocated to its different components. Even before examining how the inputs are attributed to agencies and local authorities responsible for their implementation, an evaluation must examine the coherence between the allocation of inputs and the programme Evaluation design and management objectives. It must also examine the potential leverage or impulsion effect that these public inputs will have on the mobilisation of other funds, particularly from private sources. On this point see Volume 5 of the Collection. inputs and the strategic priorities of -4 The analysis focuses in more detail on the correspondence between the the distribution of budgetary programme. The evaluation checks whether the relative importance of budgets allocated to the different objectives effectively reflects the stated priorities and goals. At the programme level, it applies the same principle that is applies to all European cohesion policy, i. e. concentration. In terms of this principle, the majority of all resources must be directed towards the achievement of objectives considered to be priorities. Concentration is evaluated by answering questions of the following kind: to what extent is the programme concentrated on a few strategic measures, on limited areas which have particularly difficult problems, or on a small number of public priorities? If an objectives tree was created during the preceding stage, it can be used at this stage of the evaluation. The total amounts allocated the corresponding objectives will simply be marked on each branch. to Modes of intervention There are often several ways of meeting an objective. Each way corresponds to one of the different instruments that can be used, for example direct subsidies, consultancy services, training, the delivery of free services, construction of infrastructure, etc. In the domain of human resources, substantial funds are allocated to programmes or systems which organise or support the training and occupational reintegration of individuals. The addressees are reached by way of training organisations or through their employers. These types of programmes provide support to public or private organisations which recruit people who have been jobless for an extended period of time. Other programmes may, for example, support the retirement of older farmers so that young farmers can set up. In the productive environment assistance may be direct (e. g. a subsidised investment or low-interest loan) or indirect (e. g. an innovation centre). It may be in the form of an open grant or a more selective programme. Assistance may be reserved for physical investments or immaterial ones Performing an evaluation (marketing, prospecting of international markets, networks, innovation, quality, design, management, etc. ). It often involves an improvement in the supply of services to businesses. R&D programmes are the most indirect form of support for productive structures. -^ In the domain of basic infrastructure, assistance is decided within the framework of programmes or major projects. In some cases a fee is charged for access to a project. Some projects are implemented by public authorities and others are delegated to private operators. For each objective, an evaluation may examine alternative instruments which could be used to achieve that objective, and the reasons justifying the choice of one instrument rather than another. It may also judge the relevance of the choice of instruments in relation to experience gained elsewhere. Selection criteria for projects A programme generally consists of measures grouping together a number of projects which have to be selected in relation to certain criteria. These selection criteria must be established before the programme starts. H As a result, it will be necessary to examine, in the evaluation of a programme, whether its management is clearly oriented towards the attainment of the objectives. In order to do so, it will also be necessary to examine whether the projects have been chosen judiciously (see Example, page 57). This search for coherence between the selection criteria of projects and the programme objectives will be of major importance in the case of mid-term evaluation, when the programme can still be improved. In that case, the evaluation will contribute to the choice of criteria that will be a key factor in the programme's success, through the type of project selected or given priority. Two types of question must be asked: • Which selection criteria are used for projects? • Are the selection criteria clearly related to the programme objectives? Distribution of operational responsibilities Inputs which are at the basis of a public intervention are not only financial; they are also human and organisational. These two components are more difficult to grasp precisely, and therefore to estimate. Evaluation design and management -4 H However, in the same way as it recapitulates financial resources, an the evaluation must also examine who implementation of level (programme, the programme are, at each measures, projects), and how the articulation between the different levels and actors is organised. the actors responsible for An evaluation must examine, at each level, whether each objective is clearly under the responsibility of at least one manager. When an objective is under the responsibility of several managers, it must examine whether a coordination mechanism has been set up. in the region (external coherence). Because the European Structural Funds are implemented jointly, evaluation must also examine the consequences of this partnership on coherence within the programme (internal coherence) and in relation to other policies In such circumstances, applied evaluation theoretically consists of questioning and examining the extent to which the programme's effects were better and more significant owing to the partnership. With this approach it is possible to observe whether the programme has had an effect of synergy or whether its effects contradict those of policies implemented in the same region by the other partners. Still with regard to the different types of input, the evaluation must try to ascertain whether major problems of effectiveness exist as a result of delays occurring in financial channels, control procedures, decision-making systems, etc. E x a m i n i ng t he p r o g r a m m e 's i n f o r m a t i on s y s t em The "information system" of the programme consists of all data grouped together for the planning of the programme and then collected on a implementation stage permanent basis by the operators during (applications for assistance, assessment of projects for selection, payment of grants, project progress reports, final reports, etc. ). Some of these data are aggregated, interpreted and transmitted in the form of tables or synthesis reports. Others are retained for subsequent control and may also be used by an evaluation. the Examination of expected results The breakdown of the programme logic serves to distinguish several levels of objectives linked to one another. At each level (operational, specific and Performing an evaluation global) the objectives are related to expected outputs, results and impacts (see Box 19). These notions are presented in more detail in Volume 2 of the MEANS Collection. Outputs are the product of the operators' activity. They represent that which is obtained in exchange for the public spending. To identify an output, the following question can be asked: "is it the condition for the payment of public funds?" Outputs are under the full responsibility of the operators who constantly have to report their actions through the monitoring system. Examples of outputs include: roads built, urban wasteland rehabilitated, purification plants constructed, and training paid for by the programme. Results represent the immediate advantages of the programme (or, exceptionally, the immediate disadvantages) for direct addressees. An advantage is immediate if addressees benefit from it while in direct interaction with the programme outputs. The expected results of a specific project are usually recorded during the monitoring process because they generally feature in the application for assistance submitted by the operator or direct addressee. Results correspond to changes experienced by direct addressees, e. g. : time saved for users of a road; reduced rates for telephone calls; qualifications obtained by trainees; diversification of a farmer's activity; new productive capacity installed by an industrialist; satisfaction of firms which received advice. Identification of expected impacts As shown in Box 23, impacts represent the consequences of the direct and immediate interactions of the programme with addressees. Thus, a first category of impacts consists of the consequences which appear and last in the medium term among direct addressees of the programme (specific impacts). Examples include: traffic on a road one year after its opening; the placement of trainees after twelve months; sustainable jobs created in a factory built with programme support; and the survival of firms created with programme support. Some impacts may be unexpected (side effects) but are not part of the programme logic. A second category of impacts consists of all the consequences which affect, in the short or medium term, people or organisations that are not direct addressees. These impacts may occur in close proximity (e. g. improvement of the quality of life for people living near a rehabilitated industrial wasteland Evaluation design and management or improvement of the quality of beaches near a new purification plant). They may also stretch far afield to distant people or organisations as in the case of macro­economic impacts. Box 23 ­ Public organisation: feedback through evaluation PROGRAMME SOCIO­ECONOMIC ENVIRONMENT Impacts The mechanisms of impact propagation are often divided into two categories: market effects (e. g. impacts on the suppliers or subcontractors of the assisted businesses) and non­market effects (e. g. the positive impact of the improved ¡mage of the region or the negative impact of a deteriorated environment). Non­market effects, or externalities, are not taken into account in private decisions, even if they have economic consequences for private actors. It is therefore particularly useful to identify them in the context of a public programme. Because of the time lag or their indirect nature, impacts cannot easily be recognised by operators in the context of their daily management. They are therefore analysed mainly during evaluation work. An illustration in terms of resources, proposed in Box 24. outputs, results and impacts is ■4 M An evaluation examines the system of indicators and, in particular, checks whether it adequately reflects objectives (the quality criteria of a system of indicators). If necessary, and in an ex ante or intermediate context, the evaluation makes recommendations for improving indicators. Performing an evaluation Box 24 - Inputs, outputs, results and impacts of a road In the example of the widening of a road to boost the development of an isolated valley, the different notions can be illustrated in the following way: • Inputs: • Outputs: • Results: • Impacts : Budgets allocated, laws used compulsorily purchase land, enterprises mobilised, information instruments used, etc. Kilometres of new roads, enlarged intersections, improved road surfaces, etc. time Number of cars using the road, reduced between two destinations, reduced number of accidents, but also more pollution for people living near the road, etc. travelling More rapid economic development of the towns served, the improved access to the leisure activities offered by regional capital (positive impacts), decreased value of neighbouring land (negative impact), or a reduced number of accidents (positive impact not directly aimed for by the programme), etc. Context-related information implementation of indicators concern all aspects of the Programme programme and also serve to monitor its effects (results and impacts), from the most direct to the most distant. They will, for example, measure the extent to which a priority group was reached or the degree to which a sustainable advantage was obtained for the end-users of the programme. Within the framework of monitoring and evaluation, a programme indicator can show that a particular intervention is a success or failure. Context indicators, in contrast, apply to the entire population without any distinction as to whether it has been affected by the programme. They will, for example, show the extent of the population's needs, or the extent of disparities that must be reduced. They can serve to assess whether the objectives are sufficiently ambitious compared to the problems faced by the population. In an evaluation, a context indicator is able show that an intervention has lost its rationale or must be modified because the problem that needed to be solved has disappeared or changed. The meeting point between impact indicators and context indicators seems obvious. It appears that by following these two indicators the estimation the Evaluation design and management programme's effects should be feasible. However, this theoretical facility is hindered by the existence of effects arising from confounding factors, which have to be taken into account. The expected final state can thus be broken down into three parts: initial state, expected effect and effect of confounding factors. For each objective, the evaluation examines whether it is linked to an indicator and whether initial state, the expected final state and/or the expected effect are quantified. the evaluation makes proposals for objectives which have no indicators: it either proposes relevant indicators or else a qualitative treatment (see the following section and Volume 2 of this Collection). If necessary, Quantification of outputs and results -4 An information system must be mobilised ex ante during the negotiation process of the programme, to specify the success criteria and quantify the desired effects. Thereafter, the apparatus set up for monitoring must allow for an examination of outputs and results, as they are obtained in the field. Each operator provides information to the programme managers in the form of progress reports or periodic reviews. The authorities responsible for the programme receive this information, discuss it in monitoring committee meetings and react accordingly. H ► At the time of mid-term and ex post evaluations, data from the system must be used as effectively as possible, provided the quality of information provided by this system has been controlled. The evaluation has to check whether adequate measures have been taken so that, throughout the life of the programme, it is possible to ensure that objectives are being achieved, that these objectives remain relevant and that decisions will be taken if this is not the case. The evaluation must also ensure that adequate measures (monitoring- evaluation) have been taken to report on the achievement of objectives. In fact, this monitoring-evaluation work has to be organised according to a plan designed before the start of the programme. It has to clearly specify who is in charge of the monitoring and decision-making, and according to what schedule it is to be carried out. Provision must have been made for adequate financial and human resources. Performing an evaluation It is by means of this framework that the pace of spending and the progress of the programme can be monitored (monitoring system), and that the different evaluations likely to be performed during the programme or at the end of it can be planned. It is also this framework that must specify how the evaluations must be discussed by the authorities managing them and how the public is to be informed. The evaluation must examine the steps that were taken to quantify the indicators. The collection and synthesis of information can be facilitated by the creation of a computerised system, for new information technologies facilitate the collection and circulation of a large mass of information. However, the construction of overabundant systems of indicators must be avoided. Evaluation may include in-depth reflection to select the most useful indicators. The evaluation team must examine the probability of the indicators being correctly informed. It makes proposals so that, where relevant, input and indicators are systematically quantified by the operators and output managers, as part of their responsibilities. A number of criteria can be used to check the quality of a system of indicators presented in detail in the second Volume of the Collection. To briefly summarise the content, there is a group of quality criteria applicable to each of the indicators selected, and a group that concerns the entire system. An indicator is of a high quality when it is quantified at regular intervals and with a frequency that is suited to the work considered and published. Moreover, an indicator must be usable shortly after the collection of the data on which its construction was based. It is also necessary to check the sensitivity and reliability of indicators in relation to the phenomena to be measured. Finally, an indicator must allow comparisons between the measures of the programme, and with other programmes or groups of addressees. There are four main quality criteria for a system of indicators: good coverage of the entire programme; good balance between outputs, results and impacts; pragmatism as regards numbers; and, finally, relevance, i. e. a selection of the strategic and operational importance of the information. indicators which takes into account Evaluation design and management Qualitative information Box 25 - Qualitative a s s e s s m e nt grid of t he environmental i m p a ct of p r o g r a m m es Priority 1 / Measure 1 : Quality sites and premises for industrial and commercial investors Objective: Guaranteeing an adequate supply of quality industrial and commercial sites for stimulating local growth and increasing job opportunities. Project : Reference : Yes Not applicable No 1. If the project is implemented on a rehabilitated industrial site, will the wasteland be reduced and will certain buildings be converted into factories? 2. Does the project comply with instructions concerning the natural environment or urban Heritage? 3. Does the project include sound energy saving practices, the recycling of water or the reduction of waste, or innovative environmental measures? 4. Does the site have an environmental management plan or are there measures in place for environmental management? 5. Have measures been taken to ensure that the site is easily accessible by public transport? 6. Has the development been designed to improve the local urban environment, for example by minimising the effects of delinquency or the lack of a public spirit? (this criterion is considered less important than the others, the score is therefore less than 1) 7. Does the development project include proposals for environmental improvements such as landscape development? (see above comment) Total score 1 1 1 1 1 0. 5 0. 5 0. 5 0. 5 0. 5 0. 75 0. 38 0 0 0 0 0 0 0. 75 0. 38 0 Performing an evaluation A good evaluation cannot exclude an objective from its field of analysis simply because it is thought to be difficult to measure quantitatively. It may in fact be one of the most important objectives, which cannot be neglected simply because of difficulties in defining a quantitative indicator. information to an Well-structured qualitative absence of information. It is therefore advisable to structure information by means of an assessment grid, structured to a greater or lesser degree and, if possible, graduated according the level of success of the intervention under consideration. is obviously preferable The construction and use of these grids are presented in Volume 4 of the MEANS Collection devoted to technical solutions adapted to complex socio-economic programmes. Box 25 gives an example of an assessment grid established for the purpose of identifying the environmental impacts of a particular measure of an Objective 1 programme in the U. K. The association of economic objectives with other objectives is interesting in this example. In all three cases (semi-structured, structured or graduated grids), qualitative information can be transformed into scores for the purpose of comparison within the programme. -4 H F °r objectives which cannot be associated with an indicator in any relevant way, the evaluation can make proposals for qualitative treatment in the case of ex ante and mid-term evaluations. Evaluating the impacts of an entire programme Socio-economic development programmes usually concern numerous domains. Although their impacts converge towards a global development objective, they materialise through very different mechanisms. When an evaluation examines an entire programme, it is expected to systematically study all impacts. Given their diversity and the number of possible causes, the applicable methods are necessarily global. These methods are based on the use of secondary data, on qualitative surveys of operators and a limited number of addressees, and on extrapolations of the results of other evaluations. Programmes financed by European cohesion policy aim at accelerating socio-economic development in one of its dimensions (income, employment, quality of life, etc. ). The main impact sought is a structural and therefore lasting impact on the macro-economic or macro-social level. To obtain this impact, most programmes adopt multi-sectoral strategies, consisting of measures of diverse kinds (infrastructure, consultancy services, subsidies, training, communication, etc. ) and in diverse domains (transport, research, tourism, etc. ). Each measure first produces results on its direct addressees, after which its impacts are materialised in successive stages on increasingly distant groups, through multiple and ramified mechanisms. The fact that the programme exists leads to the assumption that all these mechanisms converge towards the global impact of development. A programme evaluation can take one of many different forms, depending on the way its impacts are to be analysed. In this Volume, three situations will be presented as follows: 1. Global evaluation, consisting of an examination of the impacts of all the measures. This is the first situation described below (starting on page 117). The entire process is described, from the preparation of the work to the writing of the report, through the collection and analysis of data. 2. More in-depth analysis of macro-economic impacts. This is the second situation described below. It is limited to the specific characteristics of this type of analysis (see page 135). 3. More in-depth analysis of a particular impact at the micro-economic or micro-social level. This third situation is also described in the form of a supplement to an evaluation, rather than in the form of an entire evaluation in its own right (see page 147). Evaluation design and management It is important to note that the first two situations (overall evaluation and macro-economic analysis) can be encountered at all the stages of the programme cycle. By contrast, the in-depth analysis of a particular impact is possible only in ex post evaluation, long enough afterwards for an impact to have had time to materialise. This analysis involves a large amount of work and cannot be applied to an entire multi-sectoral programme, as explained in the following sections. Choosing a mode of impact analysis Imputing change to a programme Usually, when an impact is to be analysed, all that is known is what has been observed, that is, a change (reduced unemployment rate, number of businesses created, increase in the traffic on a road, etc. ). The impact accounts for only the share of this change that is imputable to the programme and only that share. An impact is also called a net effect or additional effect. All methods used to quantitatively estimate an impact consist, in one way or another, of describing what would have happened without the programme, of comparing that with what actually happened, and of estimating the difference. In other words, the impact of a programme is considered to be the difference between the real changes observed after the intervention (i. e. the policy-on situation) and the fictional changes estimated in a situation without the intervention (policy-off situation). The fictional situation is also called the "counterfactual situation". This reasoning is illustrated in the Box 26, by means of the example of an impact on job creation. An imputation technique consists of materially creating both situations by recruiting candidates to participate in the programme and by randomly selecting a first group to participate in the programme and a second, control group, which will not participate. A survey is then carried out on each of these two groups, before and after. The difference between the results of the two surveys gives a reliable estimation of the impact, provided that the two groups are sufficiently representative (each one has between 300 and 1,000 individuals, depending on the acceptable rate of error and the heterogeneity of the population surveyed). This technique is time- consuming, expensive, technically difficult and sometimes contestable from an ethical point of view. This explains why it is rarely used, despite its Performing an evaluation very strong capacity to produce robust conclusions. Instead, techniques are used which artificially reconstruct the policy­off situation, such as the comparison group technique or that of econometric simulation, developed below (see page 155). Box 26 ­ Policy­on and policy­off situations ω en c 03 «<S& *>*°ío«s' Baseline Evaluation Impact analysis is simplified to a large extent if it is carried out directly by the addressees. That is what happens, for instance, when the following questions are put to entrepreneurs who have received assistance: What will the value added generated by the assistance be in your firm? How many jobs have been created thanks to this support? These questions can be asked when the projects are selected, when payments are made, or during an evaluation survey. The answers provide a direct estimation of the effects and make the construction of a policy­off situation unnecessary. In fact, it is the surveyed people themselves who conceive the policy­off situation when they answer the questions. When the impact is estimated by means of this method an additional analysis must, however, be made. Addressees tend to over­estimate programme effects for various reasons, especially when they want to show their gratitude Evaluation design and management to the administration from which they received the assistance. In this instance the evaluation team has to perform an analysis of the real impacts of the intervention under evaluation, that is, the "net effects" as opposed to these apparent effects or the effects declared by addressees (i. e. the "gross effects"). It is necessary to determine the extent to which the assistance really did make it possible to implement a project, and how much the absence of public intervention would have affected the implementation of the project. When public assistance is granted to projects that would have been implemented even without assistance, the term "deadweight" is employed for those who have benefited from the support. The net effect is obtained by subtracting the deadweight effect from the gross effect. Example: Deadweight related to support for employment The example most frequently cited is that of a firm which decides to recruit and then tries to find out whether it qualifies for some form of public support. Many evaluations in the employment domain estimate the deadweight for employment support to be very high (between 40 and 80% of all gross effects). The public authorities of a country or region may, however, decide that the social problem of unemployment is such a high priority that the deadweight effect is the price to be paid for stimulating the recruitment of the unemployed. In some cases of evaluation, it is also necessary to analyse displacement or substitution effects. Displacement effects appear, for example, when a job is created in an assisted geographical area to the detriment of another job which is lost elsewhere (a recent case of this is the outcry that occurred when a car manufacturer closed a plant in Belgium and simultaneously created jobs in Spain). Substitution effects appear, for example, when a person finds a job thanks to a programme but this deprives another person, who finds him or herself at a disadvantage, of the job. Substitution and displacement effects are sometimes intentional, for instance when firms are encouraged to move into areas with a high unemployment rate. In such cases the impact of the programme obviously includes these effects. By contrast, substitution or displacement effects must be subtracted from gross effects to estimate the net effect when they cannot be justified, e. g. assistance which enables a jobless person to take the job of someone else who, in turn, becomes jobless. In short, as shown in Box 27, estimation of the net effect can be based on the gross effect only if deadweight and, in certain cases, the displacement and/or substitution effects are subtracted, when these were not intended by the programme. Performing an evaluation Box 27 - Deadweight, displacement and substitution effects Net jobs created Unacceptable displacement effect (a job is created and another job is destroyed in the area) Gross jobs created (statements made by participating enterprises) Unacceptable substitution effect (a jobless person is recruited and another one loses her/his job) Deadweight (a job is created which would have been created in any case, even without the aid) _ Most displacement and substitution effects can be analysed credibly by means of interviews with addressees. It is more difficult, however, to analyse deadweight in the same way because few people will admit, even anonymously, to having received assistance without anything concrete in exchange. A good analysis of deadweight requires the use of comparison group techniques, as the following example suggests. Example: The difficulty of highlighting deadweight in a rural development programme Within the framework of a rural development programme in Denmark, farmers were to receive a subsidy for the purpose of diversifying their activities. In a survey carried out as part of the evaluation, the farmers were asked whether they agreed with the following sentence: "the support received conditioned the implementation of my diversification project". About 75% of the assisted farmers gave a positive answer. An initial estimation of the deadweight would therefore be around 25%. However, a complementary survey was carried out on farmers who had requested assistance but had been refused for various reasons. The results revealed that all the farmers had realised their projects even without the support. Deadweight can therefore be considered to be 100%. This conclusion is nevertheless hasty, in so far as some of these farmers were not selected for assistance precisely because the project selection committee judged them capable of realising their projects without public support. In this case, a more in-depth analysis, using a genuine comparison group, would be necessary to understand the real deadweight effect. Evaluation design and management The above example shows that impacts must be analysed with extreme care if they are to be valid. Whether the estimation of impacts is based on observed changes or on the opinions of programme participants, the evaluation team must never rely on appearances. On the contrary, it must always make a thorough analysis and, whenever necessary, verify the results in the field. A particularly credible form of verification is that which is based on a survey of non-participants. Validity of the analysis The main authors of evaluation literature consider that the validity of an impact analysis has three forms, described in Box 28: construct validity, internal validity and external validity. Box 28 - Forms of validity of evaluations Construct validity Internal validity External validity Construct validity means that the indicators used for the analysis accurately and reliably measure the observed changes. It is particularly important for the impact to be expressed in sufficiently precise terms, and for observations in the field to be strictly coherent with what has to be measured. For example, if the impact of support for innovation is to be measured by means of a Performing an evaluation survey of regional SMEs, the notion of innovation must be defined precisely and the questions asked must correspond precisely to these definitions (e. g. number of new products or production procedures). is obtained when the evaluation Internal validity team reaches an indisputable conclusion on the fact that the programme has had a given effect. For example, programme support has doubled the innovation rate in directly or indirectly assisted SMEs (e. g. number of innovations per year for 1,000 jobs). This type of conclusion is more credible if the collection and analysis of data has carefully taken into account all the characteristics of the programme and its context (e. g. categories of businesses assisted; form of innovation peculiar to the region). Thus, to improve internal validity, the evaluation has to be adapted to all the peculiarities of the programme and its context. This runs counter to the third form of validity, external validity, consisting of producing conclusions that can be generalised to other programmes. To revert to the above example, the evaluation would have good external validity if it could conclude that similar support, attributed to SMEs in other regions, would probably have had the effect of doubling their innovation rate. External validity is clearly valuable in a context in which many programmes apply similar methods. It makes it possible to apply lessons learned in one region, to numerous other regions. • Levels of in-depth analysis The preceding sections have shown that an evaluation team cannot make a valid impact analysis without verification in the field, surveys on non- participants, and a detailed analysis of the the programme and its context. As a result, a valid analysis of an impact requires time and an adequate budget. This can, in turn, lead to the impossibility of analysing all impacts of a multi-sectoral and multi-objectives programme, or at least the impossibility of analysing them all in detail. implementation of the The choice of a level of detail is therefore particularly important. An evaluation may be performed on a purely superficial level (level 1 in Box 29) or may mobilise huge resources for making quantitative estimations (level 4). In this case, it is generally necessary to focus the analysis on a few impacts. The most superficial treatment of an evaluative question (level 1) consists of a simple description of observable changes in the concerned groups or Evaluation design and management eligible area. It is not rare for evaluations to be limited to a purely descriptive treatment of this kind, even though it well known that such analyses can give an erroneous idea of programme effects. The analysis can be improved by correcting the observed data, so as to get closer to the real effects of the programme. For example, deadweight can be taken into account by applying a correction coefficient taken from the evaluation of another programme, provided that the external validity of that evaluation has been verified. Similarly, impacts on employment can be estimated on the basis of investments made by supported enterprises through the use of an extrapolation coefficient. Box 29 - Choosing a level of impact analysis Level 1. Descriptive study of c h a n g es r Level 2. C o m p a r a t i ve s t u dy of c h a n g es •Level 3. Inductive analysis of causal relations •Level 4. Quantitative estimation of i m p a c ts Comparative analysis (level 2) must at least be based on the observation of groups or territories not affected by the programme. Analyses at this level are always possible, even with limited means, but are unfortunately performed far too seldom. The cost and time required to perform field studies is though admittedly high. Without actually demonstrating the existence of causal relations between the programme and the socio­ economic changes which took place, comparative analysis highlights a range of qualitative and quantitative indices on the nature and importance of the programme effects. Inductive analysis of causal relations (level 3) must be considered if an understanding of the mechanisms of impact production is required, and Performing an evaluation when numerous interdependencies exist with the socio-economic context (confounding causes and externalities). This level of analysis is always possible, notably through case studies, but is demanding in terms of field surveys, evaluation techniques and time. The most detailed level of analysis (level 4) is that which allows the quantitative importance of programme impacts, based on assumptions about all the competing relations of cause and effect, to be shown. This analysis, which is cumbersome and technically difficult, generally requires several months of work. Furthermore, it is not always sure to produce a conclusion, for the reasons explained below (see page 156). To conclude, the overall evaluation of a programme systematically focuses on all the impacts of all the components of the programme. Yet, given that socio-economic programmes are characterised by the multiplicity of their impacts, it follows that their overall evaluation cannot hope to go further than the first level of analysis. The validity of an overall evaluation therefore depends strongly on the external validity of the other evaluations on which the estimation of impacts is based, and, when necessary, the performing of more in-depth analyses. With more in-depth analyses (levels 2 to 4), it is possible to attain a high level of validity, but the cost involved limits them to the estimation of a limited number of impacts. It is then all the more important for these evaluations to be performed with particular attention paid to external validity. This will allow their findings to be used in the evaluation of other programmes, whether in the same region or in regions with similar problems. Exploiting existing information sources The estimation of an impact is always difficult. To produce the most robust estimation, it is advisable to use as much existing information as possible, such as administrative data, research and analyses of the domain under consideration, documents related to the life and management of the programme, published statistics, information on similar programmes, and so on. This information can be obtained rapidly and at a low cost. It can help to provide the first answers to some of the questions asked. If the terms of reference have been prepared correctly, this document will already contain a list of immediately available information. This is called secondary data, as opposed to primary data which are collected specially for the evaluation. These two categories of data are presented in Box 30. Evaluation design and management Box 30 - Sources of information for an evaluation Evaluation Secondary data Primary data Monitoring system Series of statistics Literature Experts Management documents Statistics institutes Surveys, Case studies Research Former evaluations _L ^^^^^^cc^^^^^^^^^oc^^^cc^^^^cc^^^^ Programme management documents Programme management documents contain information on planned and actual spending. This information can be used in a synthetic form, for example in the form of financial balance sheets broken down into axes, measures and actions. Elementary data for each project are systematically stored and kept accessible. It might be necessary to refer back to them, for example in the following cases: (1) to obtain the percentage of programme spending devoted to solving environmental problems; (2) to obtain the percentage of programme spending which benefited the most disadvantaged rural or urban areas; (3) to obtain the percentage of programme spending which benefited the smallest enterprises; (4) to obtain the difference between the cost per job created in a rural area and an urban area, etc. Stored data on spending often includes the name, address and telephone number of the people and organisations which received assistance. These data are valuable in cases where it is necessary to contact addressees during an investigation. Programme management documents also contain information on outputs, that is, what has been obtained in exchange for public spending. This Performing an evaluation information can be used in a synthetic form, for example through indicators or progress reports. Elementary data are often stored, project by project, but are generally not readily accessible. It could nevertheless be necessary to refer back to them, for example to obtain details on certain projects for performing case studies. The evaluation team can have access to management documents only with the help and cooperation of those who possess them. This could involve time-consuming work for managers and operators. The cooperation of those who possess information is obtained far more easily when the issue has been discussed beforehand with the parties concerned. The commissioning authority is responsible for ensuring that the necessary doors are opened, for example by involving in the steering group those who have most of the required information, or by promising to supply them with a synthesis of the first evaluation conclusions. To facilitate access to information, it is also important for the evaluation team to undertake to maintain the confidentiality of all personal data. Statistical sources Statistical sources concern an entire region, a State, or even the European Union. They are used to examine changes affecting an entire population, economy, environment of a given territory or particular population within a territory. The most useful statistics for evaluation are those which apply exactly to the territory concerned by the programme or to the eligible population. The latter sources of statistics provide context indicators which can be used to reveal apparent impacts (e. g. the number of new businesses created has doubled) to assess needs (e. g. the rate of new business creation is far lower than the European average) or to examine whether the objectives remain relevant (e. g. the rate of business creation has now caught up with the European average). Unlike management data which concentrate on operators and on the direct addressees of the programme, statistics also encompass all the people or businesses in an area that have had no contact with the programme. As a result, the comparison of a before-after statistic cannot provide an estimated impact. At best, it gives information that can be used within the framework of an impact analysis, to impute observed changes to several causes: the programme and exogenous factors (or confounding factors). Thus, for example, statistics can show an increase in unemployment due to a sharp natural rise in the working population, even though the programme has created many new jobs. Evaluation design and management In impact analysis, statistics provide useful indications on the evolution of exogenous factors, by measuring various characteristics of the territory or group concerned. They can be used to interpret or qualify observed gross effects or apparent impacts. They also supply extrapolation coefficients which are often used in evaluation (see page 126). For example, if a statistical study can be used to establish that the average size of businesses created in the past two years is 4. 5 jobs, this coefficient can be used in an estimation of impacts, for measuring support for business creation. Statistical data are directly accessible from the organisations which produce and publish them (European, national and regional public statistics institutes, private institutes, etc. ). Often these data have already been gathered by programme managers or by research organisations (e. g. regional statistical teams). For example, many regionalised data can be found at Eurostat in the REGIO database (a data base with an access fee). These data are presented in the form of about 70 standard tables per domain: demography; economic accounts; unemployment; workforce; energy; agriculture; transport; and R&D. Eurostat recently enriched its regionalised data by exploiting national annual surveys of industrial structures. The information processed is available in the ELIZE data base (a data base which is not part of Eurostat), consisting of: the number of industrial enterprises; employment; salaries and wages; turnover; and investments. The data are broken down into about thirty economic sectors and into two categories of size (+/- 20 employees). In the medium term, the REGIO data base will be completed by the use of surveys of agricultural structures. Use of statistics by the evaluation team can involve several kinds of technical problem. First, for reasons relating to secrecy or collection methods, it is not always possible to regionalise statistics in detail. Thus, surveys constructed on limited samples (10,000 to 40,000 questionnaires throughout Europe) can give meaningful results for large regions only (e. g. workforce surveys, innovation surveys, Eurobarometer). Even when statistics are regionalised in detail, their divisions do not necessarily coincide with those of the programmes. Finally, in the present state of practice, some data are calculated and published with a delay of two or three years. For suggestions on the way to use these different techniques, see Volume 2 of this Collection. Performing an evaluation Past evaluations and research Information found in past evaluations and research can play a major role in all stages of evaluation. For example, reference to specialised literature could help to suggest a relevant indicator, a tested observation grid, an explanatory model of impacts, an extrapolation coefficient, a reference for comparison, etc. It can prove very useful to use conclusions from the evaluations of similar programmes in the same region or other regions. Since the same causes can have the same effects, referring to similar programmes often helps to rapidly provide answers, at a low cost, to an evaluative question. If the evaluation team relies heavily on this type of secondary data, it must verify the external validity of the evaluations consulted, that is, the possibility of generalising their conclusions to other contexts. It is not advisable to invest too much time in listing and collecting exhaustive bibliographical data. The most effective way of gaining access to this information is by consulting experts with in-depth knowledge of the domain and/or the region concerned by the evaluation. Collecting n ew information Existing or secondary data, rarely answer all the questions that may be asked during an evaluation. In certain cases, a high quality monitoring system is enough to evaluate results, and the abundance of available research work makes it possible to extrapolate impacts. In other cases, specific investigations must be considered and primary data collected by field surveys. Interviews with managers and operators Programme managers are people or organisations that contribute towards the implementation of the programme. They allocate and manage funds. Operators, situated close to the ground, implement the programme. Broadly speaking, the addressees consist of all the people or organisations that directly or indirectly benefit from the programme. Since these different types of actor, who sometimes merge and overlap (see Box 31), have information at different levels of the programme, they will be approached during the collection of information. Evaluation design and management Box 31 - Difference between operators and addressees According to the definitions in this book, the project promoter can be either an operator or an addressee. In the example in which the project promoter is a group of businesses which receive support for developing collective actions to promote exports, two situations could exist. If the support is isolated and limited to a single project, the group is considered a direct addressee and the firms comprising it, indirect addressees. If the group repeats its actions every year, receives variable assistance depending on its activities, and reports annually on its actions and on the identity of the participating firms, then the group is considered an operator and the participant firms, direct addressees (e. g. a Chamber of Commerce and Industry which organises training courses for SMEs). During a survey for collecting primary data, the evaluation team will not always approach the group of firms in the same way. The questions asked will differ according to whether the group is acting as an operator or an addressee. Management data are secondary data which can be accessed through managers. What is under consideration here is different. It essentially concerns the opinions of the managers and operators, on issues of interest to evaluation, collected in the form of interviews. Among the multiple kinds of opinions collected for an evaluation, special mention will be made of those relating to the success and failures of the programme. The questions asked by the team during these interviews are of the following nature: which measures, actions and projects can be considered to be success stories (or failures from which lessons must be learned)? How can these successes and failures be explained? How can a particularly successful project be recognised? What are the criteria spontaneously used for this judgement (effectiveness, efficiency, relevance)? these information collected during The qualitative interviews usefully complements management data, particularly for identifying examples of best practice and for orienting impact analyses towards situations from which the most can be learned. At this stage of its work, the evaluation team must be very careful to focus on an examination of programme effects. Managers and operators often tend to judge the success or failure of a project in relation to the degree of difficulty encountered in the application of the rules or in the functioning of the administrative machine. Evaluation, however, judges the success of a project in relation to its beneficial and sustainable results for the addressees. These two forms of success are often related but a project may be problematical from the administrative point of Performing an evaluation view (e. g. barely acceptable interpretation of the texts; conflict between managers) but nevertheless an exemplary success from the point of view of its effectiveness (although the opposite is more often true!). The opinions of managers and operators are most often obtained through individual interviews. The evaluation team tries to hold enough interviews to meet at least one person fully concerned by each objective at each level of the programme. It may be interesting, from this point of view, to mark all the interviews held on the objectives tree. The number of interviews may vary from a few to several dozen, depending on the complexity of the programme. With the commissioner's agreement, the evaluation team can complement individual interviews by a series of meetings, each attended by about ten managers or operators. These meetings may be organised by town, sub- region, economic sector, domain of intervention, etc. It may also be interesting to organise meetings separately, at the different levels of the programme (e. g. operators in the field, managers of the measures, managers of the entire programme). An example of the latter approach is given in Volume 4 of this Collection, devoted to innovative techniques for the evaluation of complex programmes. Volume 3, devoted to evaluation tools, features several techniques for running meetings and for mediation, designed to make these meetings more productive and impartial. Interviews with addressees threatened by unemployment, By definition, socio-economic programmes simultaneously cover several intervention (i. e. several sectors) and numerous, often domains of heterogeneous, groups (e. g. SMEs, entrepreneurs, farmers, independent workers, wage earners jobless, researchers, etc. ). Consequently, the overall evaluation of a programme systematically encompasses all domains and concerned groups. This means that a way has to be found to collect genuinely new information in these highly heterogeneous groups, without incurring exorbitant costs. If a questionnaire survey is chosen, for example, the cost of constructing a representative sample for each of the concerned groups will be extremely high. In practice it can be done only for the in-depth analysis of an evaluative question focused on a particular public (see below). the Pragmatic techniques of primary data collection among very different addressees include focus groups and case studies. These are presented in the following paragraphs. Evaluation design and management The focus group technique has been standardised by market research professionals. A focus group consists of six to eight participants and a leader, and interviewer lasts for one and a half to two hours. The encourages a high level of interaction between the participants on essential subjects of the study. By playing on the participants' creativity and on the dynamics of the group, the technique substantially increases the capacity for exploration and helps to reveal convergent ideas. If the group is led skilfully, the product of these discussions is often far richer than that of a series of individual interviews. The presence of several participants helps to avoid extreme opinions since each argument is contested. However, clientelism among participants, the presence of "leader" personalities or a conflictual situation between certain people can spoil the quality of a meeting. It is for this reason that focus group involve managers and operators. Problems of this kind are less likely to be encountered in surveys involving addressees, and can be avoided by organising a series of three or four meetings. More information on this technique is available in Volume 3 of this Collection. interviews are more difficult when they Analysing all impacts on addressees It is important to remember that the framework being examined here is the overall evaluation of a programme which comprises heterogeneous outputs and produces multiple impacts on varied publics. This situation precludes the use of most classical impact analysis techniques, presented below (see page 153). It is furthermore important to distinguish impacts in terms of their level, that is, those which can be closely monitored among the programme beneficiaries, and those which only materialise on a regional scale. The first level of impact concerns micro-economic or micro-social mechanisms, while the second level concerns macro-economic or macro- social mechanisms. These impacts are analysed below in a section devoted to the subject (see page 135). Combining of primary and secondary data The evaluation team starts its impact analysis by observing changes that have taken place during their concomitance with a policy action. It is important to be aware at this stage of the evaluation that it is difficult to find proof of causal relations. The team the programme, and by questioning Performing an evaluation will therefore try to establish presumed causality and to explain the observed phenomena. It is interesting, for example, to combine statistical data with management data in a before-after perspective, as in the following case. Example: A combination of data which reveals a plausible impact In an evaluability study of several measures of the Objective 1 ERDF Operational Programme (1994-99) in the region of Valencia (Spain), a number of evaluation criteria were determined by type of action. For each of these criteria, between three and eleven indicators accompanied by their sources of information were also proposed. The total number of proposed indicators amounted to 123. About 40 of them were eliminated at the start of the intermediate evaluation because of their unavailability. The evaluators then selected those of the remaining indicators which provided contrasting information (Valencia region and Spain), and those based on information published periodically (for the study of trends for 1990-93) and on unpublished but available information for the most recent period. The indicators thus defined, featured in the evaluation report, and were analysed in depth by means of graphs. With these graphs it was possible to examine trends (favourable or unfavourable) for the period preceding the implementation of the programme, and to identify whether there was a variation in this tendency in the following years. Changes were classified positive when they concerned an improvement in the situation in the region, or negative when they concerned a deterioration in the situation. The intensity of the variation was qualified as insignificant, moderate or intense. For example, for the criterion "diversification and efficiency of energy resources", one of the indicators used was the number of firms using natural gas (% Valencia region compared to Spain as a whole). For the period 1990-93 this indicator revealed an unfavourable trend. From 1994, at the start of the programme implementation, a positive upward variation in the trend was observed. M ► Particularly interesting combinations can be achieved by means of a tool that is increasingly widely used: the geographical information system (GIS). This technique combines a computerised data base capable of memorising programme management data and context statistical data. A GIS serves to produce maps on which overlaps between data are visible. For example, a map may superimpose unemployment trends in the different urban or rural areas covered by the programme, and information likely to explain these trends, including the importance of the funds allocated, the number of jobless people trained and the number of firms assisted. For further information on this technique, see Volume 3 of the MEANS Collection. Evaluation design and management Exploring impact mechanisms Case study surveys examine a limited number of actions, projects or situations (cases) and produce a series of monographs describing each of these cases in a few pages. For each of the projects studied, the researcher gathers all available management data, questions the relevant managers and operators, and interviews several direct and indirect addressees. Each case study involves considerable expense but the number of case studies is sufficiently small to keep the total cost of the evaluation within reasonable limits. A case study survey provides explanations of the addressees' attitudes, real needs, reasons for participating in the programme and ways of judging the quality of the programme, as well as on the mechanisms leading to the appearance of impacts, to unexpected effects, and so on. Because the case study technique is essentially qualitative, its main function is to explain the functioning of the programme. It also helps to interpret quantitative information obtained elsewhere. Within certain limits, case studies can be used to produce quantitative information. In fact, each monograph can produce a quantitative estimation of impacts, obviously limited to the project under consideration. The question is then that of the generalisation of lessons learned from the cases. For example, if each case is selected as "typical" of a category of projects, its impact will be considered to be an image or an indication of the extent of the impact of all the projects in this category. A more interesting approach would be to study two cases per project category, one case of success and one of failure, so as to obtain a range of estimated programme effects. For a more complete presentation of the case study technique, the reader is 'eferred to Volume 3 of the MEANS Collection, and for an example of the quantitative use of case studies, to Volume 4. Quantitative estimation by extrapolation The quantitative estimation of an impact, or at least the demonstration of its existence, constitutes a central theme of evaluation. The basic techniques used for this type of analysis are comparison groups and regression analysis, as well as the modelling of macro-economic impacts. These techniques are presented further on in this volume. This section focuses on to the overall the only approach which evaluation of a complex programme: the use of extrapolation coefficients for estimating impacts. is pragmatically applicable Performing an evaluation Extrapolation coefficients are used, for example, to estimate the number of temporary jobs created per million euros of investment in SMEs, or the proportion of genuinely new jobs created, reported by firms. If the evaluation team uses existing extrapolation coefficients, it has to explain and justify its choices. A coefficient may be deduced from an in-depth study carried out during the same evaluation or a former evaluation. A coefficient may be borrowed from an evaluation or study of similar interventions, provided that the conclusions can be generalised. A coefficient may also be established for the needs of the evaluation by an expert panel or a network of experts, by using the Delphi panel technique (see Volume 3 of the MEANS Collection for further information on this technique for consulting experts). The different solutions may also be combined. •4 Extrapolation consists of applying an impact coefficient to an initial datum which may be different, depending on the case. In an ex ante evaluation, the extrapolation can be based only on basic data on the funds that interventions. Coefficients such as "number of jobs created per million euros" (or the inverse ratio of cost per job created) is then used. the programme has earmarked for the different H ► In an intermediate or ex post evaluation, output data can be used as a basis for extrapolation. Thus, for example, the number of beds in a recently built tourist facility will be used as a basis to estimate the number of jobs created in terms of a ratio of the number of jobs per bed. In certain cases, addressees are asked to make a statement on impacts during their contact with the operators. They must, for example, indicate the number of jobs created by the time the last payment was made, or the number of jobs to be created when they applied for assistance. In this case it is recommended that a correction coefficient be applied, to take deadweight into account. This coefficient may vary from 0. 8 at best, to less than 0. 3 for certain interventions where deadweight is particularly high. Analysing the effects of synergy A programme is composed of a set of projects and measures which are all oriented towards the same objective. These projects are not independent and should not be evaluated by adding or even synthesising their separate effects. On the contrary, the components of a programme are supposed to reinforce one another through synergy effects. The following box describes a number of mechanisms which produce these effects. Evaluation design and management Box 32 - Examples of mechanisms producing synergy effects in a programme A study by Cambridge Policy Consultants, entitled Program Induced Dynamic Effects, proposes a list of mechanisms likely to produce effects of synergy: • critical mass effect: an isolated project would have no impact without an adequate critical mass; • size effect: reduced costs due to the presence of several projects in the same location; • demonstration effect : multiplication of the action through example; • adjustment effect: better coordination between different public actions; • capacity effect: an isolated public action would have no impact without adequate critical mass; • productivity effect: the economies of scale of the public action itself improve its productivity; • release effect: the implementation of a project eases a constraint on another project; • Impulsion effect: the implementation of a project is an incentive to implement another project. Although synergy is an important mechanism conditioning the achievement of programme objectives, it is rarely analysed in its own right, especially because this would involve additional costs, even if the question can be treated only within the framework of an overall evaluation. To overcome this problem, a methodological exercise was carried out within the framework of the MEANS programme. The proposed approach is presented in detail in Volume 4 of the Collection. It starts with the construction of a matrix of cross-impacts, where all appear symmetrically, in lines and columns. By means of interviews with the managers and operators, the evaluation team assesses the degree of importance of potential synergy between the components (on a scale of scores from -3 to +3). These estimates are used to fill in the cells of the matrix and to calculate a synergy coefficient for each component. Certain cases of potential synergy can then be verified in the field, particularly if they have significant in terms of an understanding of programme effects or decisions to be taken. the programme components implications F o r m u l a t i ng j u d g e m e n ts a nd c o n c l u d i ng t he e v a l u a t i on Judgement references The proximity of the word evaluation to the word value, implies that an evaluation is, by definition, a value judgement. The credibility of the Performing an evaluation evaluation is based entirely on the clarification of the references of this judgement - an essential step in any evaluation and one which is too often neglected. A value judgement relates to one or more criteria, each of which may also be associated with a measurement and a standard to be met, as is symbolised by the thermometer in Box 33. Box 33 - Reference, norm and criterion degrees Celsius or % of trainees employed after 12 months^ Unit of measurement Judgement reference criterion temperature or Rate of sustainable integration into employment and society Each evaluation and each answer to an evaluative question can be based on a specific evaluation criterion chosen during the design phase. Certain evaluation criteria predominate, however, because they are recurrent and essential. They are found in almost all evaluations. These main criteria are the relevance of the programme in relation to needs, the effectiveness of the programme in achieving its objectives, and the efficiency of the programme, that is, its capacity to achieve the objectives at the lowest cost ( see page 70). Once the criteria have been chosen and their respective standards set, the measurement tools that are selected are usually indicators and, in certain cases, scores. If the programme could be evaluated in relation to a single criterion, for example efficiency in terms of cost per job created, the formulation of the judgement could be based on a single technique, in this case cost- effectiveness analysis. The programme could then be judged in relation to this criterion. Evaluation design and management However, in the evaluation of a programme in which the impacts are not all in-depth known very precisely, this type of technique, more suited to evaluations of simple actions, is not able to give very reliable results. It is usually recommended in such cases to take cost as a criterion of effectiveness but to use this criterion for establishing comparisons. The average cost per job created can then be calculated and this ratio compared to other ratios obtained elsewhere, in the context of other programmes with similar characteristics. The "¡deal" ratio, generally the best one, must be defined and will become the "standard" or the norm. This principle of benchmarking, which requires the identification and validation of references, is also a technique which can be used to establish a value judgement in relation to a model known for its quality. The multiplication of objectives requires the identification of multiple cases of best practice, each identified for its quality in a particular domain. Grading criteria The evaluation team is often confronted with the need to take into account different criteria and to choose between them when drawing conclusions. For example, what can be concluded about a measure which is expensive and creates few jobs, but affects a highly disadvantaged public? This type of trade-off must be explicit and justified, or it could compromise the impartiality of the evaluation. In case of difficulty, it is better for the evaluation team to ask to be guided by the steering group, in defining priority criteria. If the steering group also has problems making a choice on subjects considered too sensitive, it can request the assistance of a mediator. In this respect, it is possible to use certain evaluation tools such as mediation techniques. Tools like METAPLAN® or SWOT analysis (strengths-weaknesses-opportunities-threats) have the characteristic of eliciting all opinions and of identifying points of consensus, without causing conflict (see Volume 3 in this Collection). It is also possible to use multicriteria analysis which allows several criteria to be simultaneously taken into account by explaining the relations between them. Use of this technique is frequent in project evaluation but is still considered innovative in the overall evaluation of multi-decision-maker programmes. An example of the application of this technique is described in Volume 4 of this Collection. In programme evaluation, multicriteria analysis implies the active participation of a steering group, if possible extended to include about ten members. The analysis is used to help the Performing an evaluation participants collectively formulate a comparative judgement on the different measures of the programme. Apart from the capacity to produce consensual conclusions; the technique is also useful for clarifying and explaining judgement criteria. Cost-benefit analysis is another approach that can be used to grade criteria. Each impact is weighted by the evaluation team, by means of a monetary value representing what society or persons concerned are prepared to pay to obtain positive impacts or avoid negative impacts. Use of this technique is often problematical, mainly because of the difficulty of putting certain incontestable financial terms (especially environmental impacts). Moreover, the technique is extremely difficult to apply to a large programme and is practically limited to the evaluation of infrastructure projects. impacts into When it seems too difficult or complicated, in an evaluation, to embark on explanations or the grading of criteria, a good alternative is to ask an expert panel to formulate conclusions. In fields such as research or education policies, the expert panel technique is used extensively. The experts are chosen to represent all points of view, in a balanced and impartial way. These experts are independent specialists, recognised in the domain of the evaluated programme. They are asked to examine all the data and all the analyses made during the evaluation, and then to highlight consensus on the conclusions that the evaluation must draw, and particularly on the answers to give to evaluative questions. The panel does not fully explain its judgement references nor its trade-off between criteria, but the credibility of the evaluation is guaranteed by the fact that the conclusions result from consensus between people who are renowned specialists and represent the different "schools of expertise". Writing up the conclusions and draft report The independence of the evaluation team gives it a wide degree of latitude for formulating the conclusions that it considers can be drawn from the collected and analysed data. This latitude is nevertheless framed by the constraints of professionalism which cannot be ignored. Initially, the conclusions are constructed and presented in the form of answers to evaluative questions. These answers are expressed in the same terms as the questions asked in the terms of reference, and are as conclusive as possible. If one of the questions asked is not answered, the Evaluation design and management reasons are given explicitly. In addition to answers to the questions asked, the evaluation includes all the conclusions it considers to be valid and relevant for programme managers. This is the case, in particular, when the evaluation has made it possible to identify unexpected negative effects, or if it has shown that an objective is no longer relevant. In the case of overall evaluation of a programme, conclusions are formulated at the programme level, that is to say, they are transversal to the measures or actions comprising it. These transversal conclusions may be formulated in the following terms: the best results in employment or the environment were obtained in a particular part of the programme; certain measures are particularly efficient; certain measures have become less relevant after changes of context, etc. It is not acceptable for the overall evaluation of the programme to result in the juxtaposition of conclusions peculiar to each measure. Volumes 4 and 5 of this Collection provide technical proposals for helping evaluation teams formulate transversal conclusions. One of the rules of professionalism consists of linking each conclusion, as explicitly as possible, to the data and analyses that can be used to support it: statistical or bibliographic sources, facts and opinions collected during surveys (respecting anonymity when necessary), results of analyses, precise references of comparisons, etc. Methodological limits are also discussed, to show whether the link between the data and conclusions is robust, and to what extent the conclusions can be generalised. There is a difference in the nature of conclusions and recommendations. Conclusions follow from the observations and analyses of the evaluation team. Their value judgements are based on clear and explicit references, and their future lies in the hands of the political officials of the programme. The independence of an evaluation requires conclusions to be the ultimate responsibility of the evaluation team, even if they can usefully be discussed by the steering group in their draft form. Recommendations, in contrast, are more subjective and political. It is not unusual for recommendations to be made by both the evaluation team and the steering group. This is a suggestion developed in the last part of this Volume (see pages184 and 185). Performing an evaluation Box 34 - Contents of the evaluation report The contents of the report must include all the elements which allow for a clear understanding of the results of the evaluation and the reasoning followed. The volume and variety of information in the report makes it necessary to precisely distinguish contents according to their nature. Six main parts can be distinguished: • Elements of information on the evaluated programme: the history of the programme and the measures comprising it; the analysis of documents; the displayed objectives; etc. • A description of the method, which recalls the different stages in the evaluation process, the choices made by the steering group, reports on the studies carried out and their respective contribution to the evaluative questions, and the limits of the method used. • The presentation of information constituting the basis of the evaluation: primary or secondary data collected. • The arguments, that is, the analyses and interpretations which make it possible to formulate conclusions from the information collected. • A clear and precise description of the conclusions which must answer the questions asked in the terms of reference. • And, when the terms of reference specify it, recommendations reflecting the opinions of the evaluation team. These recommendations should be clearly graded according to their importance (in terms of usefulness). Furthermore, any evaluation report should contain an executive summary, that is, a two to five page summary of the six parts. Box 34 presents the typical contents of an evaluation report. Validation of t he evaluation and final report The commissioner or, more often, the steering group, expresses an opinion on the validity of the evaluation team's work at the draft final report stage. One of the following decisions is then made, based on this assessment: • The evaluation team has not fulfilled its contract and must perform additional work without an additional contract. • The evaluation team must c o n d u ct new field studies to empirically support some of its conclusions or recommendations which appear interesting but concern subjects about which little is k n o w n. This work necessitates an additional contract and budget. • The report is technically and qualitatively acceptable. Evaluation design and management In the first two cases, the steering group's opinion must be confirmed by the commissioner of the evaluation. It is only after validation of the conclusions by the competent authorities that the report can be definitively adopted and the work finalised. This step is tricky because it consists of clearly distinguishing both the technical quality of the evaluation team's work, and the conclusions themselves. Although it is normal for the commissioner to control the technical quality of the evaluation, he or she must avoid interfering with the conclusions if the evaluation is to remain independent. But the quality of the work which enabled the conclusions to be drawn is often related to the content of these conclusions. To facilitate this final validation stage, it is preferable to announce in advance the criteria on which the technical quality of the work will be judged, for example by referring to the grid proposed on page 180. Once the report has been validated, the evaluation team and steering group can classify the main conclusions in a table with the following notes: "valid, credible and soundly supported conclusion"; "conclusion not satisfactorily supported and may require more in-depth analyses"; "conclusion leading to interesting recommendations"; "conclusion contributes new knowledge"; etc. To conclude these first two chapters, it should be underlined that the performing of an overall evaluation of a programme is indispensable insofar as the principle of programming is an essential part of Structural Fund regulations. Such an approach, at the overall programme level, is facilitated today by the experience acquired over recent years. The rest of the process consists of disseminating the evaluation report and organising its use. These steps are considered further on in this volume (see page 159). Before continuing, it is important to mention the ways in which more in-depth impact analyses can be performed. The following pages will be devoted to this subject. Analysing macro-economic impacts In addition to the overall evaluation of a programme, it may be necessary to analyse macro-economic impacts The impacts of Structural Fund interventions on a regional or national scale depend on the micro-economic in all domains of intervention, but also on macro-economic mechanisms which appear only on a larger scale. impacts obtained These mechanisms are difficult to evaluate credibly by means of simple techniques. It is possible to understand them more fully and to provide a schematic representation of them. Their quantitative estimation involves modelling techniques which, in recent years, were developed specially for the evaluation of European cohesion policy. Can macro-economic impacts on a regional or national scale be considered to be the sum of all micro-economic impacts? The debate is not closed, but has nevertheless led to the recognition of the importance of macro-economic mechanisms which condition the achievement of global development objectives. This section is devoted entirely to these mechanisms. The examples in Box 35 serve to distinguish these two types of mechanism. Box 35 - Examples of micro- and macro-economic mechanisms - Thanks to assistance received, a firm develops (micro level) and buys more from its regional suppliers which, in turn, buy more from their regional suppliers, etc. (supplier effect, macro level). - Thanks to assistance received, a firm expands (micro level) and pays out additional wages which are partly spent in the region. This triggers more development, followed by further income growth, etc. (multiplier effect, macro level). - A public investment fills the order book of a regional firm, which reduces competition when subsequent investments are made (micro level). This, in turn, leads to an increase in price levels, etc. (macro level). - To finance their contribution to assisted projects (micro level), regional authorities increase their taxes, reducing available income, etc. (macro level). - Thanks to assistance received, a firm becomes more competitive (micro level), which eliminates rival regional firms (macro level). Evaluation design and management From the point of view of an evaluation, micro-economic impacts can be evaluated by questioning a limited number of people, i. e. direct addressees (where direct effects have been observed) and people who have been in contact with direct addressees (where indirect effects have been observed). Significant effects are undeniably produced beyond these limited circles of addressees, as Box 35 shows. These more distant effects involve macro- economic mechanisms which potentially affect all markets or economic actors in the territory concerned. A v o i d i ng o v e r - s i m p l i f i ed a p p r o a c h es indicators such as the per capita GDP or Macro-economic changes in a region can be observed relatively easily by means of statistical the unemployment rate, the two most frequently used indicators of this kind. This availability naturally leads evaluation teams to make before-after comparisons of the regional economic situation. When a programme accounts for only a few percent or a fraction of a percent of the total investments, the observation of global developments has no regional significance in terms of impact - no more than have comparisons with the global economic development of neighbouring regions, even if they are not eligible for public support. Only in cases of a massive transferral of public funds towards a region that is lagging behind in its development, can evaluation be founded on the observation of major economic trends. This is what was done in the 1960s for the evaluation of regional policy, by means of a technique called shift- share analysis. This technique and all those presented in this chapter are described in information sheets in Volume 3 of this Collection. Shift-share analysis is a top-down evaluative approach which compares observable in the assisted employment territories. It breaks down regional employment into large sectors, and applies to them the development rates observed at a national level for the same sectors. In this way a counterfactual situation is constructed, showing how the region would have developed if its development had corresponded to the national average. The comparison with real development reveals a difference (the residue) which is entirely imputed to the effects of outside support. The extremely approximate nature of this analysis can be justified only if external support is very large. trends on a national and regional scale A number of macro-economic effects can be estimated by means of extrapolation coefficients drawn from former research or analyses, according Performing an evaluation to the same logic as that described above (see page 126) for the estimation of micro-economic impacts. The coefficients of macro-economic effects apply to those micro-economic effects which are supposed to have been estimated beforehand. The formulae are of the following type: Macro effect = Micro effect X Coefficient. The supplier effect coefficient varies between 1. 1 and 1. 5, depending on the size and degree of openness of the regional economy. The multiplier effect coefficient varies within a comparable margin. The evaluation team must take very serious precautions and, if necessary, call on experts for their advice, before using macro effect coefficients. One of the reasons for this need for caution is that the value of the coefficient varies widely from one region to the next. The other reason is that these macro-economic mechanisms are numerous and the unfavourable ones are often overlooked. Thus, for example, eviction or price increases require the use of significant negative coefficients. the limits inherent the simplest approaches, two in Because of complementary directions are proposed for the evaluation of macro- economic impacts. The first consists of developing a logical model of relations of cause and effect between the programme and its macro- economic objectives. This qualitative approach emphasises those effects which are the most important in terms of development, i. e. structuring effects. The second approach, of a quantitative nature, consists of using econometric modelling to simulate regional economic development with and without the programme, to deduce an estimated impact. These two approaches are considered in the following sections. Linking effects at t he micro a nd m a c ro levels A development programme aims at producing sustainable structural change throughout a region or country. The intended effects have two dimensions, defined as follows: effects which must last even after the public support is discontinued (notion of structuring effect) and effects which occur at the macro-economic and macro-social levels. By contrast, programme outputs are always micro-economic or micro- social and are situated at a particular point in time. They are not expected to last indefinitely, for this would implicitly amount to the failure of the programme. An evaluation must therefore try to identify and connect the Evaluation design and management sequences of causes and effects which make it possible to establish links between outputs at a given date and permanent effects, and between the micro and macro levels. This approach is similar to that of the logical diagram of impacts which can be applied both to the overall evaluation of a programme (see page 95) and to in-depth analyses (see page 150). Global impact on social and economic cohesion The diagram in Box 36 provides an overview of the mechanisms linking Structural Fund programme outputs to the global objective of the policy under which they fall, i. e. economic and social cohesion. It is a logical representation of the effects of interventions on development, through the mechanisms of structuring effects. The following sections study this process, step by step. The diagram was developed by the authors of this volume, following their work on the MEANS programme. Other authors would probably draw different diagrams or use different vocabulary. If these diagrams were used within the framework of a particular study, the evaluation team and steering group would need to adapt them to suit the programme strategy and regional context. Box 36 - Micro-macro articulation: an overview Micro level Macro level Outputs the domains of. in. human resources. the productive environment. basic ¡nfrast'ucture Immédiat impact on development \ Sustainable impact on development Reduced unemploy­ ment Increased income Better living conditions Economic and social cohesion Performing an evaluation The objective of economic and social cohesion, as envisaged in the context of European policy and supported by all the different Structural Funds, finds its origins in Article 130a of the European Union Treaty. There it is defined in terms of "harmonious development" with a specific geographical dimension: "reduce the gap between the levels of development of the various regions, and the backwardness of the more disadvantaged regions, including rural areas". Agenda 2000 recalls this objective and adds a dimension related to the extension of the Union to the countries of Eastern the struggle against following objectives: Europe, and unemployment; the promotion of equal opportunities between men and women; and the stimulation of competitive development and of sustainable development from an environmental point of view. the to • Impact of investments in basic infrastructure The impact of infrastructure on cohesion indirect and collectively benefits businesses and people. Relations of cause and effect through which impacts are produced are multiple. is essentially Examples of infrastructure projects which lead to sustainable impacts at a macro-economic level: (1) A new harbour, road, motorway or electricity supply network cuts the production costs of regional firms and enhances their competitiveness. (2) A sewerage or water supply system helps to remove impediments to the development of local enterprises; an airport promotes the development of tourism. (3) Improvements to the telecommunications network allow for the establishment of new relations between local enterprises. These relations improve the functioning of regional innovation networks, which in turn improves the competitiveness of regional SMEs. Infrastructure programmes can benefit existing economic activities but also increase the relative attractiveness of a territory for outside investors. Infrastructure may also generate direct benefits in terms of access to public facilities (e. g. shorter travelling time to a hospital). Evaluation design and management iri tui rig e f f e c ts in t he d o m a in •e B ox 37 - Ex; än 3E i p le of s t r u c ti sic infrastruc of Motorway -►· Used by local entreprises ^w Macro level r 1 Reduced costs / \ / Increased market share Easier travelling N, /\ ι». Living conditions Attractive for outside investers N Development induced for other entreprises Economic and social cohesion • Impact of interventions on the productive environment For a long time regional policies were based on direct support for so-called "capacity" investments. This logic implied that the supported investments would increase the capacity to produce the commodities for which an unsatisfied demand existed. Today, when support is granted directly to businesses, the assumptions are often very different. The intention is to encourage or stimulate spending likely to directly increase the competitiveness of the assisted firms. The idea is that European support helps firms to establish competitive advantages. These advantages are said to be "visible" when they result in better "price- competitiveness", owing to productivity gains or reduced production costs. The "invisible" or immaterial advantages are increasingly decisive in world competition and are therefore encouraged by public support. Examples include: support for innovation, marketing, quality, etc. Finally, more and more public support is intended to create or strengthen competitive advantages which collectively belong to all the firms in a region Performing an evaluation or sector. The productive environment comprises all the factors that the enterprise can find outside its own internal organisation, to optimise its performance. In the example Box 38, the programme subsidised a conference centre intended to enhance the competitiveness of regional businesses. Box 38 - Example of structuring effects in the productive environment Micro level Conference centre Utilisation by local firms Macro level Economic and social cohesion Impact of interventions in the human resources domain The initial effect of support for human resources is that it enhances and increases the addressees' qualifications and capacities. The concept of human capital accounts for the accumulation of educational and training actions "invested" in individuals living in a region. This concept builds a bridge between the social nature of the results of a training action and its macro-economic effects. Human capital encompasses measurable effects on labour productivity and less palpable effects on adaptability, relevance of specialisation or the quality of relational networks. Structural interventions in favour of human resources have a direct impact on the concerned groups by increasing their employability, that is, their Evaluation design and management ability to find a job or keep one in the context of a given region or specific economic circumstances. Support in favour of human resources may have an impact on businesses (e. g. a specific training programme accompanies the installation of a new factory; a training programme accompanies the restructuring of a firm). However, businesses benefit mainly in an indirect manner, in the form of improved adaptability and the manpower productivity of the labour market concerning them (e. g. advanced technical training can favour innovation and enhance the advantages of a region already specialised in a dynamic sector). Finally, interventions in the domain of human resources often have essential non-economic impacts, such as the reconstruction of social links destroyed by unemployment, the return to social integration, and a new-found balance in people's lives. These impacts are often related to a return to employment but are worth identifying and, if possible, evaluating separately. Box 39 - Example of structuring effects in the domain of human resources Micro level Training courses Qualifications obtained Macro level Macro-economic and strategic impacts The objective of cohesion implies a reduction in disparities between regions or between social categories. To progress towards this objective, the Performing an evaluation assisted regions or categories must have a faster pace of development than the average pace of European economic development. For this to be possible, the programmes must not only stimulate socio-economic development, but also boost the driving force behind this development. The different types of strategy which have been adopted to this end give rise to very different expectations regarding impacts. try Socio-economic development can be sought by external forces such as investment, immigration of qualified professionals or technology transfer. factors of development. Many programmes Attractiveness is the keyword of this type of development, called exogenous development. Better attractiveness is a goal of many kinds of programme, the results of which are expressed, for example, in the form of infrastructure, manpower lower property prices, changes qualifications, quality of urban life or protected countryside. to attract these the in The relocation of a firm towards an assisted territory is not, in itself, proof of success in terms of development because it does not guarantee a lasting effect. Certain investments can be attracted because they are mobile. This mobility is likely to last to the point where these firms leave the territory after a few years and move to areas offering lower production costs. Consequently, the chain of cause and effect which concerns exogenous development must be completed by a final link: it must be assumed that recently established businesses will establish adequate links with the manpower and productive environment in the territory so as to remain attached to that territory for a long time. Socio-economic development may also be maintained by internal causes. Ideally, endogenous development is generated spontaneously from a sound economic base, that is, a set of firms and workers who are strongly rooted in their territory, who manage to impose themselves on their respective markets and whose markets are buoyant. The strength of a territory's economic base can rely on its specialisation in a growth sector. A programme aimed at reinforcing this base could stimulate sectors of strategic and dynamic activities through indirect support: training; collective services; networking; access to information; innovation; research; etc. Support for endogenous development presumes that a growth factor already exists in the form of specialisation in certain dynamic activities or in the form of a regional competitive advantage. The programme supports the activities concerned and enhances their competitiveness and specialisation, thus reinforcing the dynamics of these activities. Evaluation design and management Endogenous development of a territory depends to a large extent on the existence of local non-commercial networks. Efforts to create network effects is at the basis of assistance to "technopoles" and "innovation systems". These effects lead to the creation of an economic, social and political environment capable of constructing and reproducing its own success factors. In these systems, essential competitive advantages do not belong to individual firms, nor do they rely on factors of production such as manpower or capital. The key factors of success of a territory belong, collectively and informally, to a network of firms and local actors. This network constitutes the main source of competitive advantages (technology, innovations, know-how, etc. ). Firms take advantage of the network in formal and informal ways: professional associations; support to local and regional authorities; informal networks; inter-firm mobility of staff; parallel information channels, etc. These means of interaction are generally independent from market transactions, which means that they correspond to the definition of the word externality (economic effects which do not depend on a customer-supplier relationship). in terms of purely economic A programme aimed at encouraging endogenous development must not be impacts. The creation of an evaluated economic, social and political environment to construct and reproduce its own competitive advantages implies all sorts of dimensions. The term "local potential" represents this set of factors which governs the renewal of the growth process: innovative capacity; institutional dynamism; acceptability of development; etc. likely P e r f o r m i ng a q u a n t i t a t i ve e s t i m a t i on Earlier in this volume, simplified approaches for quantitative estimation, such as shift-share analysis and the use of coefficients, were examined (see page 136). These approaches were intrinsically limited in their analytical capacities. A theoretically ¡deal approach would consist of performing quantitative modelling of all the macro-economic mechanisms, as outlined is probably not qualitatively completely realisable, but considerable efforts have been made in that direction. in the preceding pages. This approach Modelling techniques are used to represent the simplified functioning of an entire economy and of the different markets composing it (e. g. product market; labour market; capital market; etc. ). A model schematises the Performing an evaluation functioning of the economy at equilibrium, but is also capable of simulating changes in this economy as a result of an exogenous shock (e. g. fluctuations in the oil price; variation in exchange rates; taxation rates), the adjustment of modelled variables, and the achievement of a new market equilibrium. It is thus possible to use a model for evaluation purposes by considering that the programme provokes an exogenous shock (a micro- economic first round impact) and by simulating the repercussions of this shock throughout the economy. The new equilibrium takes into account all the macro-economic mechanisms, particularly structuring effects, if these mechanisms have been modelled. In practice, one scenario is simulated with public intervention and one without. The macro-economic impact is estimated by calculating the difference between the two. The models are entirely quantified and thus provide an estimation of the extent of the impacts on all the macro- economic variables which have been modelled. -4 ► The estimation of effects can be ex ante. In that case, two scenarios (with and without the programme) are compared. It is also possible to use a model in an ex post evaluation situation, although that is a far more innovative approach. In that case, the comparison is between the model's approximate calculation of past economic developments, and a scenario without public intervention, reconstructed ex post by the model. Available macro-economic models are generally limited to the national level. The models constructed on a regional or local scale are rare, especially because economic flows are very quickly diluted outside the territory concerned. Modelling nevertheless remains meaningful for small economies, provided insular economies are often equipped with macro-economic models. Moreover, the use of a model is all the more interesting when the intervention of the programme reaches a sufficient mass compared to the modelled economy. It is considered that to reach this critical mass, the economic importance of the programme must be 1 % of the GDP of the territory's economy. they are relatively closed. That is why Within the framework of European structural interventions, the most advanced macro-economic evaluation experiment was performed with the HERMIN model. This work is presented in more detail in Volume 3 of the Collection, in parallel with two other experiments. The HERMIN model was used for evaluations in Spain, Ireland and Portugal. These evaluations introduced an interesting innovation by explicitly simulating supply-side Evaluation design and management effects, a notion that is fairly similar to structuring effects mentioned above (see page 137). In the case of HERMIN models, the effect of programmes on the productivity of work and/or other factors of production, as well as on the attraction of foreign investments and the increasing competitiveness of exports, was simulated. The above experiment provides an interesting example of current R&D on the modelling of macro-economic impacts in terms of development. This work aims at surpassing the limits of earlier experiments in modelling, which simulated only demand-side effects. Demand-side effects result from the fact that public money spent by the programme continues to circulate in the regional economy in the form of consumption by businesses and households. They do not take into account gains in productivity and competitiveness. Clearly, an approach reduced to demand-side effects would be too limited because it would implicitly assume that a programme had no more effects when public spending ceased. This obviously contradicts for socio-economic development, which is why it will be particularly interesting to follow developments in the modelling of supply-side effects in future years. the basic hypothesis of support Carrying out more in-depth analyses In addition to the overall evaluation of a programme, which has to study a large number of impacts with its own particular methods of analysis, it is possible and useful to carry out specific analyses of a small number of particular impacts. These analyses use the entire range of evaluation techniques, collect primary data and result in credible quantitative estimations of the achievement of a programme's objectives. A more in-depth analysis implies undertaking an investigation to observe changes experienced by programme addressees, followed by an analysis of causality, aimed at determining the part of change that can be attributed to the programme. H ► Unlike the overall evaluation of a programme which is intended to be exhaustive, the in-depth study of a question concerns a category of outputs, a public and a mechanism for making impacts apparent. The overall evaluation of a programme may be accompanied by the in-depth study of one or more questions. It may also be decided to launch an evaluation devoted entirely to the in-depth study of a single question. While the notion of thematic evaluation relates to the study of one question, it is not the same as an in-depth study. A thematic evaluation examines how a single impact (e. g. competitiveness of SMEs) is obtained by several kinds of intervention, or how interventions financed in a specific domain (e. g. R&D) produce several kinds of impact. Thematic evaluations are described in Volume 5 of this Collection. They are too complex to fit into the simple logic of this section, i. e. the evaluation of a specific intervention from which a main impact is expected (e. g. impact of R&D on the competitiveness of SMEs). The question treated, thus simplified, makes it possible to use all the classical evaluation techniques, especially for observing addressees' behaviour and for attributing to the programme its share of the changes observed. Since in-depth study mainly concerns the observation of effects in the field, these analyses concern mid-term evaluation and ex post evaluation. Detailed analysis of the internal logic of t he p r o g r a m me The overall evaluation of a programme requires the clarification of its internal logic and, in particular, an examination of the articulation of its objectives at Evaluation design and management different levels (see page 95). The logic of a programme can be represented schematically in the form of an objectives tree, the global objective being the trunk and the multiple outputs being the branches. The analysis of a category of outputs and an impact amounts to isolating a branch of the tree. It is then possible to examine this branch in far more detail. This is shown in the following sections. Logical framework impact. The The logical framework technique, originally designed for ex ante evaluation of projects in developing countries, is a tool which can be used to simply and rapidly describe the logical relations between an output and an expected technique consists of producing a matrix presentation stating the expected outputs (cf. the bottom left hand side of Box 40) and the main logical steps separating the global impact outputs (left column). The logical framework matrix thus describes a branch of the objectives tree by placing the "branches" at the bottom (outputs) and the trunk at the top (global impact). At each logical step, the evaluation team states the means that have been or must be provided to verify whether the objectives have been achieved (verifiable success indexes). At the same time, what must be considered a success or failure (sufficient / insufficient outputs, results and impacts) are defined. By making it necessary to fill this second column of the matrix with indicators or precise qualitative descriptions, the technique often helps to provide useful qualification which makes it possible to revert to a more precise description in the first column. The third column, called "conditions and risks", lists exogenous factors which can facilitate or hinder the passage from one step to the next. This point will be examined in more detail below (see the "explanatory model"). More details can be found on the logical framework in Volume 3 of this Collection devoted to evaluation techniques. It is recommended that the logical framework technique be implemented in a participative way, involving the managers and operators responsible for the evaluated outputs. The column on the left of the logical framework matrix describes the sequence of effects of the programme in four steps (outputs, results, specific impact and global impact). The column on the right, by recalling all the external factors which may condition the passage from one step to the next, shows that the achievement of impacts is based on hypotheses. It is Performing an evaluation for this reason that evaluation specialists call the left column the "theory of action". The theory of action consists of all the hypotheses used to explain that a given output is going to produce a given impact. The main purpose of the evaluation is to compare this theory with reality. Box 40 - Logical f r a m e w o rk Statement of expectations Verifiable Sucess indexes Conditions and risks Global impact Economic and social development Specific impact Savings for regional enterprises Result Less travelling time on typical journeys Output Improved roads Logical diagram of impacts Within the framework of in-depth analysis, the four lines of the logical framework matrix are generally not enough to provide a sufficiently explicit and detailed theory of action. In particular, there are often grey zones between results and impacts. This is usually illustrated by the image of the black box of impact hypotheses. This image recalls the frequent situation in which designers and operators know how to explain what they do and what the anticipated global impact is, but do not know how to convincingly explain the process that is supposed to lead to this impact. The example in Box 41 proposes the breakdown of impact hypotheses into four steps. Between each of these steps, it may be necessary to identify intermediate impacts and to refine the logical diagram of impacts. In particular, the diagram is worth detailing each time a relationship of cause intense and effect seems obscure or ill-justified. This necessitates Evaluation design and management interaction with the managers and operators, during which the evaluation team reveals its implicit hypotheses and produces a complete theory by trying to progressively eliminate any logical weaknesses in the reasoning. Box 41 - More detailed logical diagram of impacts Advice to SMEs on exports Reduced unemployment Advice to SMEs on exports International contacts New and sustainable export growth Survival of SMEs Reduced unemployment Experience has shown that when the steering group itself constructs the logical diagram of impacts, it considers this step as significant value added for the evaluation. The 'example in Box 41 shows that by opening the black box of implicit hypotheses, the evaluation work can be directed towards certain impact hypotheses rather than other, less strategic ones. This example shows that the evaluation has little chance of reaching a conclusion if the observation is limited to the number of days' consultancy services provided to SMEs and to unemployment statistics. By contrast, the analysis of impacts will be more interesting if the managers of the assisted SMEs are asked how many the international contacts have been generated or programme, how these contacts have developed, how many of them are still active, what turnover they generated, what percentage of the firm's business they account for, etc. An in-depth analysis of the programme logic is used to focus the field observation on the right level of impact. facilitated by Performing an evaluation ■ Carrying out observations in t he field The empirical and pragmatic dimension of an evaluation is based on its capacity to observe programme effects in the field. It is therefore through surveys on programme addressees that the evaluation contributes its full value added. The in-depth answer to an evaluative question often consists of a collection of primary data from persons or organisations which have participated in the programme, and from non-participants when the evaluation uses comparative techniques (see pages 110 and 155). The following sections describe the main survey techniques used in evaluation. Collecting quantitative data The questionnaire survey is the most widely used way of collecting quantifiable information. It may be carried out by mail, telephone or interview. The formulation of the questionnaire is a decisive step. The questionnaire consists of a limited number of clearly stated questions, formulated so as to be of interest to the interviewees. It proposes closed questions which the interviewees can answer by choosing among suggested answers. A few open questions may also be used, but the answers must then be coded (classified by category) after the data collection. This involves a large amount of work. A pre-test of the questionnaire on a limited number of persons is indispensable for validating the form and content of the questionnaire and confirming its capacity to collect information. The organisation of a survey may require several months and its cost will vary, depending on the size and accessibility of the sample, the desired rate of response, etc. The different surveys undertaken in the region must also be carefully coordinated so as not to provoke rejection by the addressees (the refusal to participate in a survey is increasingly frequent). Questionnaire surveys are used extensively for a wide variety of purposes, but primarily for collecting quantitative information from addressees. Because the questions asked are closed, the data provided by a questionnaire can be used for the construction of indicators. A basis for questionnaire surveys is the use of a sample, so that the conclusions can be generalised to the entire population. A sample of 200 to 300 addressees is often enough to obtain the correct quantification of an indicator. The sample must, however, be larger if the quantitative nature of the information provided by a survey is to be exploited to the full, particularly when the results of two surveys are to be compared, as in the following Evaluation design and management three examples: a survey of participants on a training course, given that some of them are in employment after 12 months, and that a comparison is to be made with a group of non-participants; a survey of assisted firms, given that a small number of them develop hi-tech projects and that a comparison is to be made with the results of a national survey; a survey of farmers' incomes, given that part of this income is from tourism and that a comparison is to be made with the results of a survey carried out five years earlier on a partially identical sample. In each of these examples, the evaluation team wants to compare the results of surveys on different or partially different samples. In this case, the rules of statistical validity preclude comparisons if the samples are not large enough. The minimum size ranges from 500 to 2,000 individuals, depending on criteria which are independent of the size of the surveyed population. A professional may be employed to choose the right size of the sample in a concrete situation. Otherwise, the size of statistically comparable samples is usually between 900 and 1,100 individuals. A particular form of survey is that which is used to monitor individuals or groups of individuals through time, and thus to measure changes of attitude or situation. These panel surveys are generally used in the case of interventions in the employment domain (the participants are questioned before a rd after training, for example). They are spread over several years and are therefore difficult to launch during an evaluation which lasts for a few months only. However, when panel surveys do exist, evaluation uses their results as secondary data. Qualitative survey techniques An in-depth analysis generally combines the collection of quantitative and qualitative information. Several qualitative survey techniques are presented in Volume 3 of the Collection, devoted to evaluation tools. The semi- structured individual interview and case study (see page 125) are the most common tools. The evaluation team must not restrict its technical choices on the basis of habit. It is for this reason that two techniques which may prove highly decisive are discussed below. The first is the focus group. Provided it is conducted by a qualified person, the focus group is used to create a dynamic of interaction between the participants, to take advantage of their creativity, and to gather far more information than that which would result from individual interviews with Performing an evaluation each of the participants. The focus group is highly recommended when the the addressees have a very vague opinion about programme has produced for them (case of a SME which participated in a technological network, for example). Their answers to a questionnaire or during an individual interview would probably be polite answers. In these cases the focus group is used to obtain the most reliable answers. the effects that The second example is less typical. This is ethnographic observation, requiring the researcher to merge into the group of programme participants (as a participant or assistant on a training course, for example). This technique has been successfully used for investigations of difficult populations which lack motivation and are sometimes even reluctant to participate. Analysing i m p a c ts In-depth analyses allow impacts to be studied most thoroughly (see Box 29). This is why particular attention is paid to impact analysis in the following sections. Depending on the situation, impact analysis combines two types of approach in varying proportions: inductive analysis and deductive analysis. Inductive analysis suggests causal links based on the observation of facts. It is therefore essentially exploratory and aims at producing a theory explaining causes and effects, based on observations—usually qualitative—in the field. Deductive analysis considers, on the contrary, that this explanatory theory is a starting point that needs to be tested by comparing it to reality. In cases where the initial hypotheses are validated, it may lead to a qualitative estimation of impacts. The following two sections explain how to construct an explanatory theory, how to estimate impacts, when to use the theory and what the limits of impact analysis are. The order of sections is merely a convention because an evaluation can be performed the other way round if this can be justified by the circumstances. Explanatory theory of impacts This volume has, on several occasions, mentioned relations of cause and effect between the programme and its impacts, and the way in which these relations can be represented schematically in the form of a logical diagram of impacts. The explanatory theory is only an additional step in the Evaluation design and management representation of the process of impact production, but is nevertheless a difficult and decisive step in the evaluation. Box 42 shows that the step from the logical diagram to the explanatory theory adds all factors which may facilitate or hinder the production of impacts at each level. This is what the logical framework technique calls "conditions and risks" and what the evaluation literature calls "exogenous factors" or "confounding factors". In any evaluation situation, there is a potentially infinite number of exogenous factors. The role of inductive analysis is to determine which of them appear most influential in relation to what was observed during field surveys on addressees. Box 42 - Theory of action and explanatory theory New and sustainable export growth Survival of SMEs When carrying out inductive analysis, the evaluation team will also try to identify all the unexpected effects observable among the addressees. For the example, several evaluations have shown a negative effect inhabitants of assisted areas, resulting from the fact that these areas are declared "disadvantaged" and consequently become less attractive (stigmatisation effect). Fortunately there are also positive effects. for The case study technique has an excellent cost-effectiveness ratio for identifying exogenous factors and unexpected effects. It is probably the Performing an evaluation basic tool of inductive analysis. A lot can also be learned from a series of individual interviews with people who have participated in the programme and with others who have not. Inductive analysis should not, however, be considered to be qualitative in all cases. Techniques such as variance analysis or factor analysis are also used regularly to reveal the most influential exogenous factors. Without using such sophisticated tools, others such as tables, maps and graphs, when they are well chosen, can often reveal overlaps and similarities which clearly indicate the importance of certain exogenous factors. Estimation of impacts Impact analysis consists of attributing part of the observed changes to the programme and another part to exogenous factors, or at least to the main exogenous factors. The latter become confounded with the programme to explain "before-after" changes. This is why exogenous factors are also called "confounding factors". To estimate impacts it is necessary first to estimate exogenous factors, then to subtract this effect from observed changes, and finally to attribute the difference to the programme. The "counterfactual situation" is the result of the influence of exogenous factors only. To estimate a counterfactual situation, the most frequently used technique consists of constituting a comparison group with people who have not participated in the programme, for example applicants whose project was rejected, applicants who dropped out of the programme midway or non-applicants with a similar profile to the participants (matching groups as in the following example). A fictive comparison group can also be created from individuals who participated (statistical comparison by adjustment). Once the comparison group has been formed, different methods of comparison can be applied: "after only" comparison, "before- after comparison" or the use of time series which look at the situation before, during and after. For a detailed description of the technique, the reader is referred to Volume 3 of the Collection. in a national survey The evaluation team must create a comparison group that is sufficiently similar to the group of direct addressees of the programme. Both groups must have the same essential characteristics, which amounts to saying that they must be influenced by the same exogenous factors. Differences between these factors would probably explain part of the differences Evaluation design and management o b s e r v ed b e t w e en the t wo g r o u p s, w h i ch w o u ld distort t he estimation of i m p a c t s. The c o n s t r u c t i on or a d j u s t m e nt of t he c o m p a r i s on g r o up therefore implies a s o u nd k n o w l e d ge of e x o g e n o us f a c t o r s, that is, an explanatory theory of i m p a c t s. Example: Impact analysis using matching groups A comparative analysis of matching groups was carried out in the U. K. , during the evaluation of a programme offering SMEs consultancy services in management, training a rd information. The impacts of this programme could not be expressed in terms of jobs created since the aim was to increase productivity, which would in the short term lead to a reduction in jobs. It was therefore necessary to evaluate the programme in terms of other criteria such as productivity, profitability or investments induced. A sample of about 300 recipient firms was constituted. Another sample of about 200 non-recipient firms was also constituted by means of the matching pairs technique. The following criteria were used to match the pairs: type of activity of the firm, size of the firm measured in terms of number of employees, age, and geographical situation. A questionnaire was addressed to the recipient firms and to those of the matching group. A statistical comparison between two groups provided quantitative indications on the effects of the programme in terms of turnover, the productivity and questionnaire included numerous and partially redundant questions (triangulated questionnaire). The survey also included a search for deadweight. "biased" statements, induced. To avoid investments in the firms • U se of e x p l a n a t o ry t h e o ry Explanatory t h e o ry is even m o re necessary w h en the estimation of i m p a c ts uses regression analysis. U se of this t o ol starts w i th the d r a w i ng up of a small m o d el to s c h e m a t i c a l ly represent c a u s es (the p r o g r a m me a nd its main e x o g e n o us factors), effects (observed changes) a nd relations of cause and effect. This m o d el is t h en a d j u s t ed to represent as accurately as possible the available d a ta ( m a n a g e m e nt d a ta a n d / or survey data). The result of the analysis is a quantitative e s t i m a t i on of t he i m p a c t. The validity of the e s t i m a t i on strongly d e p e n ds on the quality of t he initial m o d e l, and particularly on the f a ct of this m o d el t a k i ng e x o g e n o us factors into a c c o u nt correctly. Regression analysis is a t y p i c al e x a m p le of d e d u c t i ve analysis. The t e c h n i q ue necessitates the formalisation, f r om the outset, of a set of Performing an evaluation hypotheses on causes and effects (which is why it is also referred to as hypothetico-deductive). For this purpose it uses an explanatory theory either borrowed from the scientific literature or taken from an expert's knowledge, or else constructed by inductive analysis in the context of an evaluation. In a second stage, the hypotheses are compared to reality and the evaluation team draws conclusions on the existence of the impact, on its quantitative importance or on its non-existence. Limits of impact analysis The evaluation of an impact is an exercise which is similar to forecasting, even when the approach is retrospective. What would have happened if public money had not been spent? There is no ex post evaluation without an answer to this question. Yet, the description of what would have happened without the counterfactual situation, is nothing other than a reverse forecast. Thus, evaluation has all the same limits as forecasting, and it is well known that there are many such limits. the construction of the public action, i. e. Certain contexts lend themselves readily to forecasting and evaluation, whereas others are so complex that neither of the two exercises produces credible results. In an uncertain context, the evaluation team cannot establish the counterfactual situation convincingly. Even the participants in the programme are unable to estimate the impact that was produced within the framework of their own project. Favourable situations are those in which the explanatory theory is relatively simple and linear. For example, public investments improve an airport, which leads directly to the multiplication of air links. This, in turn, leads directly to an increase in tourism. In this example the cause is clearly known and mechanically leads to a relatively predictable effect. It is possible to estimate impacts in such a situation, provided the price is paid in terms of surveys and analyses. Unfavourable situations are those in which the explanatory theory is complex, particularly when it contains relations of circular causality. This is the case of the impact of training programmes, the effects of which follow entangled paths. On the one hand, they produce new qualifications which must fit with economic needs in order to improve the employability of trainees and result in high placement rates. On the other hand, they provide firms with more productive manpower, which enhances their productivity Evaluation design and management but also reduces employment. Productivity gains nevertheless have a favourable influence on competitiveness, which indirectly helps to create or safeguard jobs. Impact analysis in this context becomes very difficult because the causal relations can be explained in various ways: the placement rate may be either the cause or the consequence of gains in competitiveness. The difficulty is even more obvious when the phenomenon of circularity also applies to the policy itself. It is to be hoped, for example, that the unemployed are oriented towards training in areas where placement rates are high, rather than being oriented randomly. Does this not mean that the policy itself is influenced by the impacts it is supposed to produce? With this type of circular causality, the evaluated programme can be seen alternately as a cause or an effect. In this situation, the concepts of effects, impacts and results lose their meaning. Thus, certain impact analyses are impossible to perform due to the very nature of the problem posed, no matter how professional the evaluation team may be. The terms of reference must therefore be drawn up with care because certain questions relating to programme impacts defy evaluation, even when it includes in-depth analyses. IV Using an evaluation Optimising the use of an evaluation If an evaluation is to add real value in the institutional and decision-making spheres, its conclusions must be disseminated correctly and must be viewed favourably by potential users. The evaluation report is not necessarily the only nor the most effective medium, particularly when users are elected representatives or citizens who will receive the results through the media. These considerations represent a strong argument in favour of the early establishment of a plan for communicating the results of an evaluation. After the dissemination stage, the last and most important stage of the evaluation is its use. The entire evaluation process must be geared towards obtaining the most effective utilisation possible. It is advisable, however, not to be over-optimistic about certain forms of use. Although a good evaluation report always produces new knowledge (cognitive use), it does not necessarily lead readers to revise their judgements in line with its conclusions (normative dimension), nor to adopt all the recommendations in their decisions concerning the programme (instrumental value added). Indeed, the public decision-making process often involves several actors and a large number of factors. Disseminating t he conclusions of an evaluation In the following sections, the term "user" is employed to refer to any person who reads or hears the conclusions and recommendations of an evaluation, and for whom this information leads to a change in their knowledge, opinions or attitudes. Potential users of evaluations range from co- decision-makers, evaluation operators and managers, to readers of a newspaper or journal in which an evaluation is mentioned. The dissemination of an evaluation is an essential phase which follows the production of the evaluation report, but which warrants being planned from the early stages when the terms of reference are drawn up. It is preferable for the terms of reference to announce to whom the evaluation results will be addressed and in what form. Decisions in this respect are confirmed by the commissioner after the submittal of the report, but by carefully planning the dissemination strategy the use of the evaluation can be optimised. Evaluation design and management Communication channels The dissemination strategy consists of a compromise between the objective of maximising the visibility of the evaluation and that of establishing a climate of trust in which interactions between all the evaluation stakeholders can take place. As far as surveys conducted by the evaluation team on addressees and programme operators are concerned, trust and visibility can be guaranteed by treating all survey results as confidential by protecting the anonymity of the interviewees throughout the evaluation process. The question of interaction between the evaluation team, the steering group and the commissioner may, however, be difficult. Interaction focuses on sensitive points such as the clarification of criteria and norms, the identification of examples of good and bad practice, the importance given to a particular question or criterion rather than another, etc. Sensitive questions cannot be treated properly with total openness. On the contrary, the evaluation team must be sufficiently discreet in its work and its report to enable members of the steering group to express themselves sincerely in their own name and, if necessary, to venture away from the official discourse of their institutions. In certain cases, the evaluation team writes its report in such a way as to maintain a high degree of discretion as regards its interaction with the programme managers, and adds a confidential note to attract their attention to more sensitive points. Box 43 - Communication channels Communication channels ra 0) ■Ό E S o õ υ Ξ * Ι E il E E £ Β co Q. Q SS IV * u c CD ta 3 Commissioner Steering group Managers Operators European, regional and national, institutions Citizens Journalists * * *. 9 ­c 11 (fi 'tn ω. c >% to ύ cu υ E «Γ £ o û­O ω. C OJ lì CÛ ra C g CO C Φ co £ * * * * * * iV Using an evaluation All too often the problem of the use of an evaluation is reduced to the question: "To publish or not to publish the final report?" when in fact a larger set of questions ought to be taken into consideration. Apart from the question of publication, the way in which information is disseminated should be carefully considered and selected. Various forms of communication and numerous possible media are presented in Box 43. The people involved in the programme (evaluation steering group, managers and operators) tend to use evaluation as soon as the first results are presented by the evaluation team to the steering group (e. g. results of surveys and analyses, intermediate reports, draft final report). From their point of view, the use of evaluation starts before the communication of the final report. For the other users it is essential to plan a suitable communication strategy that combines a medium and a deadline. Box 43 shows that different communication channels are possible and need to be chosen in terms of the availability of the different categories of user ("receivers", as communication specialists would say). Communication plan The commissioner and the highest administrative and political authorities do not have the time to read evaluation reports in detail. The most effective mode of communication at this level is a short synthesis in a few pages, referring to certain sections of the final report for further details. This can usefully be complemented by an oral presentation by the steering group or its President, during a seminar or meeting after the conclusion of the work. For communication to European, national and regional institutions, a short synthesis can similarly be used. This is facilitated if the synthesis is presented in the form of a brochure containing concrete examples of programme outputs, and showing how these outputs have been judged by the evaluation. The best way of informing social partners and programme addressees is by publishing special documents (newsletters, synthesis sheets, etc. ). Other documents are prepared to provide information for the "public at large" (brochures, articles, etc. ) and the media. In this respect, a press conference, attended by the steering group, may be arranged. Because the evaluation, a particularly strong impact on the journalists can be obtained. the members of the group enhance the legitimacy of In most cases, the rules of administrative transparency require the publication of the final report. This is nevertheless only one of many ways —and not always the most effective way—of making the conclusions of the evaluation Evaluation design and management available to its users. A real effort to produce a complete publication and to disseminate it widely will, however, have the advantage of exposing the evaluation to critical examination by experts, which is a guarantee of quality. Moreover, if the commissioner announces that the report will be published and widely disseminated, the evaluation team will be more motivated. The evaluation will probably be more credible because its organisation is more transparent, and the methodological quality of the report is likely to be better. On the other hand, there is a risk of self-censorship since conclusions and recommendations will be more cautious. To avoid this drawback, which could prove to be a real handicap, the evaluation team could be asked to produce a confidential note in its report. Good communication to users of an evaluation also requires a "staggered" dissemination process. If the evaluation includes field surveys and an in- depth analysis of certain questions, part of the results obtained should be presented, without waiting for the conclusion of the final report. The communication plan must, above all, be carefully synchronised to correspond to the main phases of the programme: dates of the main management meetings; dates of the main decisions; release of public reports on the policies concerned by the programme at the regional, national and European levels. In reality, it is not only the communication plan but also the entire evaluation process which ought to be organised in relation to the users' schedule. This concern is present from the outset (see page 67) and is decisive as far as the usefulness of the evaluation is concerned. C o n s i d e r i ng all f o r ms of u se to be effective: Evaluation is frequently judged in terms of its own criteria. It is naturally is value added of an evaluation sufficient expected compared to the financial and human inputs devoted to it? To answer this type of question, a large number of meta-evaluations have been performed and published. Most studies conclude that there are several types of use, and that direct use of evaluation recommendations by decision-makers is only one form among others. Three main forms of use are highlighted. For raising the level of knowledge -4 M ► Any evaluation will, first and foremost, afford its users an learn and, if opportunity to discover—sometimes to their surprise—to Using an evaluation possible, to understand. The design stage of an evaluation has an important part to play in this respect because it helps to clarify objectives which were badly expressed or only implicit in the programming documents, to explain impact mechanisms, to identify changes of behaviour, to check whether expected effects were obtained and to highlight unexpected effects. The evaluation exercise transforms the user's representation of the programme and its effects. This form of use may be conceptual when new knowledge focuses on the explanatory theory of impacts. Whatever the declared aim, a successful evaluation must have a visible and recognised cognitive use. For deciding on improvements that they can subsequently appropriate M The evaluation team and steering group make recommendations to the decision-makers. The active participation of the steering group in this stage of an evaluation has the advantage of lending a more realistic dimension to the recommendations and of taking into account the concerns of the these stakeholders so recommendations more easily. Irrespective of the choice made, it is important for recommendations to be clearly separated from conclusions, and for their status to be clearly explained by answering the following questions: Does the implementation of this recommendation have a financial implication? Can it be implemented in the short term? Is there consensus on the recommendation? Even if there is never an obligation to follow the recommendations of an evaluation, the commissioner should find it worth while monitoring the future of these recommendations for a year or two. Example: Instrumental use of evaluation The intermediate evaluation report of the Irish 1994-99 CSF provided a sound basis for discussions during the mid-term review of that CSF. Following negotiations between the European Commission and the national authorities, the Monitoring Committee agreed on a series of amendments concerning almost 3% of the total budget. Part of the main changes in allocations, as well as certain other financial decisions, clearly resulted from recommendations made by the evaluators of the CSF. Thus, out of the 18 measures or groups of measures for which allocations were called into question, eight had their support reduced. On the other hand, five of the eight domains identified as requiring additional support, were granted such assistance. Non-financial decisions relating to changes of priorities were also taken, i. e. training and the strengthening of the capacity of small businesses for expansion, at the expense of support to the private sector in general. Evaluation design and management At best, traces of recommendations can be found in decisions taken subsequent to an evaluation. In such cases, evaluation constitutes a decision-making tool and can serve to improve a programme midway or at the time of in the example. This use of evaluation its renewal, as conclusions is qualified as "instrumental". Forjudging success and failure ► Evaluation provides users with arguments for forming or modifying their opinions. It helps to clarify criteria and define norms for comparison. It is also used to draw conclusions on the success or failure of a particular part of a programme, and consequently to identify best practice and exemplary projects. This form of use is essential when the evaluation serves for accounting to the political authorities and the public at large. It is a use which is outside the programme and which feeds into debate on the justification of this programme. In such cases an evaluation is similar to reporting and is intended to enable people outside the context of the programme (public authorities, elected representatives, the general public) to form a global opinion on the intrinsic value of the action, independently of the opinions of the stakeholders. In the same context, users compare evaluation conclusions with their own opinions. They consequently maintain, support or change their personal judgement on the programme. This form of use is called normative because it comprises a large degree of judgement. Not all evaluations are used in this way because normative use is the most difficult. Users who have to revise their judgement are likely to resist, especially if this judgement has already been expressed publicly. The specialised literature usually distinguishes two evaluation practices, depending on whether it is primarily intended for actors directly concerned by the programme or exogenous addressees. An evaluation is called "formative" when its use, within the evaluated action or programme, is likely to give their behaviour, improve the action and thus transform its object. information enabling the stakeholders to change them Evaluation is said to be "summative" when it serves both to judge a programme and to help the supervisory authorities decide on its launch, maintenance or reconstruction. In this case users are outside funding Using an evaluation authorities or other social actors who may decide on the appropriateness of adopting the best practices identified during an evaluation. Irrespective of the nature of the decision taken subsequent to an evaluation, the process of use involves the three successive forms of utilisation. the Decision-makers start by programme. They then formulate, strengthen or modify their judgement and, finally, express themselves convincingly at the right moment, so as to persuade their co-decision-makers to follow suit (since public decision­ making is almost always taken jointly). their understanding of transforming Thus, numerous logical steps are involved in the passage from evaluation to decision-making, and the last of these steps brings several actors into play. This is why the instrumental use of an evaluation is difficult to demonstrate. This is confirmed by meta-evaluations which often conclude on the absence of a direct short-term link between recommendations and decisions. -4 It is in this sense that ex ante evaluation is distinguished from a feasibility study of a programme. A feasibility study is always based on the objectives of a political authority, for directly preparing a decision. In no way does it challenge these objectives. Its only aim is to aid decision-making. An evaluation, by contrast, always has a dimension of judgement about the appropriateness of the programme, taking its political objectives and the point of view of addressees. Even in the case of an ex ante evaluation of a programme proposal, instrumental use is not the only mode of use; cognitive and normative dimensions are inevitably present. into account Evaluation also differs from basic research from the point of view of its utility because it responds to an order by the programme managers, at a time when they require information and answers to questions. The exercise is the scope of always highly constrained by theories of action and investigation chosen by the decision-makers. By contrast, the results of evaluations are often accepted more readily by public authorities and, as a result, are used more. Assessing the quality of an evaluation Although the quality of an evaluation is often judged on the sole basis of the final report, it largely depends on the way in which the entire process was managed, from the commissioning of the evaluation to the formulation of recommendations and the communication of results. A fair judgement of the quality of an evaluation requires the clarification of the criteria used. This assessment may be based on eight criteria constituting important professional standards: meeting the needs of the commissioners; relevance of scope of an evaluation; appropriateness of the methodology; reliability of data; soundness of the analysis; credibility of results; impartiality of conclusions; and clarity of the report. It is necessary to assess the quality of an evaluation to verify whether the team satisfied the commissioners' requirements, but also to distinguish between those conclusions which are valid and soundly backed-up, and those which are likely to be contested or which necessitate more in-depth empirical work. Assessing the quality of an evaluation is, or should be, a preliminary step preceding the use of its conclusions and recommendations. This is a logical preliminary because only an evaluation of a high quality can stand up to the criticisms that inevitably arise when judgements on success or failure are made. It is also a chronological preliminary because quality is often formally assessed before the dissemination of the evaluation report and before conclusions and recommendations are taken into account. We have chosen to place this chapter at the end of the three parts of the Volume devoted to the design and its realisation of an evaluation project, because the quality assessment grid provides a checklist of all the proposals and recommendations made in preceding chapters. It was developed within the framework of the MEANS programme, with the help of experts from several European countries. It can be used by readers wishing to check whether their own practice meets prevailing professional standards. M a n a g i ng a nd improving t he quality of an evaluation process With the development of evaluation and, consequently, the general increase in budgets devoted to it, attention is being focused more and more on the Evaluation design and management quality of work submitted and on the quality of the entire process, from the commissioning of an evaluation to its utilisation. Quality control checks whether the evaluation team has fulfilled its contract. Its very existence helps to raise the evaluation team's level of awareness, which helps to improve the final report. More indirectly, the assessment can improve professional practice by spreading standards of best practice. It can also serve a broader democratic objective by enhancing the credibility of the information disseminated by the public administrations. The quality criteria presented below were used by the Commission for assessing a number of evaluation reports during the 1994-99 programming period. They have also been applied by certain national authorities. Thus, for example, the quality of several dozen evaluations performed in the Republic of Ireland during the period 1994-98 was assessed by referring to these criteria. This work resembles exercises in meta-evaluation, i. e. the evaluation of one or more evaluations. Quality is a requirement that is sufficiently crucial for it to also be the concern of every commissioner of evaluations. Quality assessment involves co-decision-makers of every programme, in terms of assessing the quality of their calls for tender, and quality control of the work carried out by the evaluation team. Co-decision-makers are not, however, the only actors to have an interest in the quality of reports (see Box 44). The steering group, in particular, decides is sufficient or must be whether the collection and analysis of data improved, whether the conclusions are acceptable in view of the proof gathered, and whether the report answers all the questions in the terms of reference or whether further work needs to be undertaken. In general, regional, national and European authorities analyse the quality of evaluations themselves, in relation to their own constraints and criteria. In the case of highly technical evaluations or the publication of sensitive conclusions, to comment on the quality of the process and to assess the robustness of the conclusions. The quality criteria presented below are largely drawn from professional standards, developed within the framework of assessments of this kind. the commissioner may call on an outside specialist Using an evaluation Box 44 - Who assesses the quality of evaluations? Quality assessment can be done by. with a view to. • the steering group with the help, when necessary, of a specialist • validating the final report and verifying its robustness • the co-decision-makers of • gauging the reliability of conclusions the programme and recommendations • national and European • gauging the reliability of conclusions authorities and recommendations • improving the evaluation system currently in use • independent experts • guaranteeing the genuineness of the conclusions disseminated to the public • developing professional standards (meta-evaluation) The assessment can be carried out in a spirit of quality assurance of the entire evaluation process, for the final report is merely the result of a system which mobilises a number of people within a vast institutional ensemble, and of an apparatus specially set up for the evaluation itself. In this case, it is a question of monitoring and promoting the quality of the main stages of the evaluation. Taking the course of evaluation, from its inception to the production of the report and the dissemination of the results, it is possible to improve its quality by acting on the following: • The quality of information and the monitoring system. • The terms of reference defining what has to be evaluated and the funds and time allocated. • The choice of an evaluation team (qualifications, references, experience, etc. ). • The contract signed with the evaluation team, which outlines the work to be performed. • The frequency and nature of interactions between the evaluation team and the steering group. • The rapid and complete provision of available high quality information. • The mode of communicating conclusions, including the report itself and the other channels of dissemination. • Use of the evaluation for reporting to the public authorities and citizens. Evaluation design and management A system of quality assurance concerns the entire evaluation process and therefore covers all these points. It may be represented as follows. Box 45 ­ System for controlling overall quality Terms of reference (contract) \ Interaction with commissioners Competence of the evaluation team \ / \ f ÎI Performing the evaluation Π Quality of available information 1 Responsibility towards the stake­ holders / public 1 Reports |l ►■ A n a l y s i ng t he q u a l i ty of an e v a l u a t i on r e p o rt In general texts devoted to evaluation, the notion of quality is a subject of much debate. There is no system of professional certification, anywhere in the world, which institutionalises rules and quality criteria. Somewhat disparate grids of criteria are proposed, based on the evaluation models elaborated by various authors, but no consensus exists in this domain. Moreover, the nature of the criteria mentioned does not always appear clearly when an attempt is made to put them into practice. Within the framework of MEANS, eight quality assessment criteria for evaluation reports were selected. The global assessment of an evaluation is established synthetically after the final report has been analysed in terms of these eight criteria. The criteria are presented below, with the reference situation taken to be mid-term evaluations, although the grid has a more general scope. It can be applied to all evaluation situations, bearing in mind that the formulation of certain questions and the relative weight of the criteria must take into account the specific nature of the exercise under consideration. Throughout the rest of the text such adjustments for ex ante and ex post evaluation are mentioned, in relation to intermediate evaluation, when they exist. Using an evaluation Eight criteria were defined for assessing the quality of evaluation reports. The global assessment of an evaluation is established synthetically after the final report has been analysed in terms of these eight criteria. Meeting needs Within the framework of European Structural Fund programmes, the main stakeholders are the members of the steering group. Ideally, their diverse requirements are grouped together in the terms of reference. The evaluative questions formulated in this document must be complemented during preliminary meetings and interviews, in order to go beyond what is already obvious for the commissioners. It is therefore necessary to check whether the evaluation has answered the questions in the terms of reference satisfactorily and whether the report covers additional requests for information, as well as new questions which might be essential for the commissioners. Thus, to ascertain whether "evaluation deals adequately with requests for information formulated by the commissioners, and corresponds to the terms of reference" (see the synthetic grid in Box 46), the following questions must receive particular attention: • Has the evolution of global and specific objectives of the programme been analysed? • Does the report cover the entire programme? If not, is the selection justified as regards the priorities stated by the commissioners? • Does the evaluation provide useful feedback for the managers? • Does it include lessons on successes and failures which may be of interest to other regions or countries? Unlike mid-term evaluations, for ex post evaluations it is important to check whether the information needs of the new generation of programmes have been taken into consideration. It is therefore necessary to check whether the evaluation has managed to reach a reasonable compromise between the following two contradictory requirements: rapidly obtaining information for feeding into the new programme cycle and not drawing hasty conclusions before all the impacts have been observed. Relevant scope The scope of an evaluation must cover questions which are relevant from the point of view of Structural Fund programmes. The brief description of essential programme characteristics and the analysis of the strategy of a Evaluation design and management programme understanding the evaluated action and its rationale. for the region concerned are essential prerequisites for In order to check the relevance of the scope of an evaluation, it is necessary first to check whether the essential characteristics of the programme have been well described and whether the problems and successes in the implementation of the programme have been properly clarified. Secondly, because the results and impacts of the programme have to be analysed in order to judge the extent to which programme objectives have been achieved, it is necessary to check whether they have been included in the evaluation. It is also necessary to check whether the evaluation has not overlooked other potential or future results or impacts, as well as any unexpected yet significant effects and results which may exist. Finally, the scope of an evaluation depends on the programme target which can be defined in terms of eligible geographical areas or non-localised target groups (e. g. the long-term unemployed). It is therefore necessary to check whether: • the limits of the scope, in terms of areas or social groups, are defined according to the logic of the intervention; • the scope includes peripheral areas or non-eligible groups which are nevertheless likely to be affected by the evaluated interventions; • lastly, if the evaluation considers the evaluated programme in isolation or includes its interactions with other European or national programmes. Within the framework of ex ante evaluations, the report has to add all new information which may be useful for the improvement of programme documents: strategy, implementation of choices. It is necessary to check whether the report evaluates in sufficient detail the target groups' expectations and the corresponding relevance of the programme objectives and strategy. internal/external coherence, • Defensible design This essential criterion relates to the technical qualities of the evaluation. Methodological choices must be derived from the evaluative questions and demonstrations required. The evaluation must, moreover, make the best possible use of existing research and analyses. For this criterion as well as the following two, the commissioners could, if necessary, request an audit by programme evaluation specialists. Using an evaluation In any case, three types of question have to be asked: • Have the relevant knowledge and existing literature been collected and used wisely? • Are the construction of the method and the choice of tools really justified for answering the evaluative questions properly? • Were the reference situations chosen (counterfactual or similar) appropriate for making valid comparisons? Any evaluation report must include a description of the method used and clearly define the sources of data. Similarly, the limits of the method and the tools used must be clearly described. It is necessary to check whether: • the method is described in enough detail for the quality to be judged; • the validity of data collected and tools used is clearly indicated; • finally, the available data correspond to the tools used. Because a causal analysis of effects is the most important question in ex post evaluations, the method used to analyse these causal relations is the priority in this type of evaluation. It is necessary to check whether the evaluation adequately analyses relations of cause and effect for the most essential questions. • Reliable data Evaluators use existing data (secondary data) from the monitoring system and from other sources of information, or else primary data which they have collected for the evaluation. In the latter case, the method used to collect and process the data (choice and application of the tools used for this purpose) are very important factors in the reliability and validity of the results. In order to assess the reliability of the data used, it is necessary to examine: • whether available sources of information have been identified and whether the reliability of this data has been checked; • whether sources of information taken from the monitoring system and previous studies have been used optimally; • and, lastly, the extents the monitoring system and its stock of useful data and indicators of use in ex-post evaluation. to wich mid-term evaluation reviews • whether the techniques used to collect the chosen data were complete and suitable for answering the evaluative questions; Evaluation design and management Whether the collection of data used quantitative or qualitative techniques or a combination of both, each of the following steps must be analysed: definition of "mother populations"; selection of samples; construction of tools (the questionnaire; the indicators chosen). It is therefore necessary to inquire whether: • the mixture of qualitative and quantitative data is appropriate for a valid analysis of the phenomenon; • the "populations" used for data collection have been correctly defined; • the survey samples or cases studied have been selected in relation to established criteria; • the main data collection implemented with appropriate tools and in such a way as to guarantee an adequate degree of reliability and validity of the results. techniques have been For ex ante evaluation, analyses are limited to existing data because ottime constraints. This implies that the sources of information selected are sufficiently reliable so that a credible judgement can be made of the relevance and rationale of the programme. Sound analysis Quantitative analysis consists of the analysis of data in the form of tables or any other form of statistical analysis. Qualitative analysis consists of the systematic comparison and interpretation of information sources in the form of cross-referencing. In both cases it is necessary to assess whether the methods of analysis used are relevant as regards the type of data collected, and whether the analysis has been carried out according to instructions in the relevant technical manuals. In the case of the Structural Funds, relations of cause and effect are complex and therefore constitute a particular challenge for evaluation. It is necessary to check: • whether the relations of cause and effect underlying the programme are sufficiently explicit and relevant so that the object of analysis can be focused, • and to what extent the analysis uses suitable techniques. Within the framework of in-depth evaluation, a before-after comparison or, when similar groups do not exist, a comparison between beneficiaries and Using an evaluation a control group, is recommended. In the former case this before-after comparison must be carried out appropriately, otherwise it must be established whether that would have been possible. In the latter case, the comparative analysis must be able to rely on the collection of data within similar or control groups. Furthermore, the exploitation of evaluation work carried out at the programme level should enable the Commission to prepare syntheses at the European level. This implies coherent definitions (e. g. calculation of the net effect), homogeneous figures, etc. , so that the results can be used to compare the performance of the programme with that of other programmes. Credible results if there The credibility of results is defined here as the fact that they follow logically, and are justified by, the analysis of data and interpretations based on carefully presented explanatory hypotheses. The validity of the results must be satisfactory. Even the fundamental contradiction between internal validity (absence of technical bias in the collection and processing of data) and external validity (representativeness of results), the balance between two must nevertheless be justifiable, given the type of evaluation considered (in- depth or overall). It is also necessary to check whether the results of the analysis were produced in a balanced and reliable way. is no single solution for solving the The need to perform in-depth analyses of a part of the programme poses the problem of extrapolation, from case studies, for the programme (or Member State) as a whole. In this context, it is necessary to check: • whether the interpretative hypotheses and extrapolations are justifiable, and whether the limits of validity have been defined; • whether the selection of cases and samples makes it possible to generalise the findings. With the stakeholders' participation in the discussion of results, the relevance of the evaluation and, consequently, the utility of the programme can be enhanced. The following questions need to be asked: • Are the results credible in the sense of reflecting an acceptable compromise between the reality of the programme, as perceived by the actors, and the reality described by the observed or estimated figures and facts? Evaluation design and management • To what extent were the stakeholders involved in the process of interpreting empirical facts? • Were the results presented impartially to the stakeholders? • Impartial conclusions Conclusions include suggestions and sometimes even recommendations which are more than results. Whereas results are "technical" and can be analysed without too much risk of impartiality, conclusions and, a fortiori, recommendations are issued on the basis of value judgements. The quality of the judgement is thus decisive. To answer the question "Are the conclusions fair, free of personal or implemented partisan considerations, and detailed enough concretely?", it is necessary to check: to be • whether the elements on which the conclusions are based are clear; • whether the conclusions are operational and sufficiently explicit to be implemented; • whether controversial questions are presented in a fair and balanced way. • Clear report Evaluation results can be disseminated and communicated the stakeholders in writing or verbally. The final report is only one means of diffusion among others, and continual communication of results is desirable. The legibility of the report will depend on the quality of the presentation of results and the limits of the work performed. It is necessary to check: • whether the report was written clearly for its addressees and whether it is to set out logically; • whether specialised concepts are used only when absolutely necessary and whether they are clearly defined; • the extent to which the presentation, tables and graphs enhance the legibility and intelligibility of the report; • and whether the limits of the evaluation, in terms of scope, methods and conclusions, are clearly shown. In many cases only the summary of a report is read. It is therefore essential for this summary to be clear and concise. It must present the main conclusions and recommendations in a balanced and impartial manner. It Using an evaluation must be easy to read without the need to refer to the rest of the report. It is also important for conclusions to be clear as regards broader European issues, such as environmental impacts, equal opportunities, and so on. impacts, employment M a k i ng a fair global a s s e s s m e nt Synthetic assessments recapitulate all the quality criteria. It is not necessary to recommend any particular weighting for the different criteria because their importance varies from one situation to the next. The synthetic assessment must take into account the constraints weighing on the evaluation and the team which performed it. The initial conclusion reached by strictly applying quality criteria must therefore be qualified. For example, a report may be judged to be scarcely adequate, not because of shortcomings related to the actual evaluation work, but because the terms of reference were unrealistic. Very often, the time and financial resources allocated to evaluation teams limit the scope and soundness of conclusions far too much. Contextual constraints must into consideration for judging the report in fine. therefore be taken inputs, the time allocated, and is concerned. Constraints relating to Two types of constraint can be distinguished, depending on whether the commissioning of an evaluation or the management of the evaluated programme the commissioning procedure are: clarity and precision in the terms of reference, the volume of budgetary the commissioners during the implementation of the evaluation. Constraints relating to the programme and its management may result from a lack of clarity on the objectives, a high level of heterogeneity among the measures and the monitoring system, as well as crises or ruptures which may occur during the implementation and which impact on the course of the evaluation. inadequate quality of the data generated by involvement of interventions, the It is up to the commissioners to limit, if not totally eliminate, the effects of these constraints, and up to the evaluation teams to anticipate technical and operational problems before embarking on the evaluation process. Evaluation design and management Box 46 - Grid for a synthetic assessment of the quality of evaluation work With regard to this criterion, the evaluation report is: excellent good acceptable inacceptable 1. Meeting needs: Does the evaluation adequately address the requests for information formulated by the commissioners and does it correspond to the terms of reference? 2. Relevant scope: Have the rationale of the programme, its outputs, results, impacts, interactions with other policies and unexpected effects been carefully studied? 3. Defensible design: Is the design of the evaluation appropriate and adequate for obtaining the results (with their limits of validity) needed to answer the main evaluative questions? 4. Reliable data: Are the primary and secondary data collected or selected suitable? Are they sufficiently reliable compared to the expected use? 5. Sound analysis: Are quantitative and qualitative data analysed in accordance with established rules, and are they complete and appropriate for answering the evaluative questions correctly? 6. Credible results: Are the results logical and justified by the analysis of data and by interpretations based on carefully presented explanatory hypotheses? 7. Impartial conclusions: Are the conclusions just and non-biased by personal or partisan considerations, and are they detailed enough to be implemented concretely? 8. Clear report: Does the report describe the context and goal, as well as the organisation and results of the evaluated programme in such a way that the information provided is easily understood? D D D D D D D D Π D D D Π D D D Π D Π Π Π Π Π Π Π D Π Π Π D D Π In view of the contextual constraints bearing on the evaluation, the evaluation report is considered to be Π Π Π Π ν Guaranteeing the success of an evaluation Better preparation of an evaluation The programmes financed by the Structural Funds are subject to a particularly intense evaluation effort. Since 1989, several hundred programmes have been evaluated in their entirety, before, during and after their implementation. This is without counting transnational thematic evaluation and more in-depth analyses carried out in certain regions. Evaluation practice has rarely spread as rapidly and on such a massive scale. Progress still needs to be made, however, from the point of view of quality/cost ratios and that of utilisation. A number of the recommendations likely to increase the use of evaluation concern its preparation. Pluri-annual planning of the different evaluation exercises must be continued, the commissioning authorities need to make their decisions more visible when launching an evaluation, and more prior investments also need to be put into the drawing up of the terms of reference, if necessary in the form of an evaluability study when complex issues need to be addressed. ■ Pluri-annual p r o g r a m m i ng of evaluations As described in the second part of this book (see Box 7) the sequence of programmes, period after period, is unavoidable. The programmes follow one another uninterruptedly and the regulations require a series of evaluations for each period (ex ante / mid-term / ex post). In practice, these different constraints have created and will continue to create problems of articulation between the different evaluation exercises. In particular, it is often difficult to learn from the implementation of a programme, at a time and in a form that is really adapted to issues involving decisions. Experience shows that it is necessary to think about a pluri-annual evaluation programme with the goal of improving the opportunities for feedback (a process which leads from the observation of results and impacts, to decisions to reorient the programme). To deal with this problem, many managers and evaluation teams wish to develop on-going evaluation. Experiments in this domain have been run in Spain (for regional operational programmes in Andalusia and the Independent Community of Valencia), Ireland (for about half of the operational programmes) and France (programmes in the Burgundy region), for example. These attempts warrant careful study. Evaluation design and management Many experts are, however, sceptical about evaluation conceived as an on­ going process. The system creates a risk of confusion between evaluation, monitoring and management, which could well result in the watering down of the value added by evaluation. On the other hand, the establishment of a pluri-annual evaluation programme can be recommended for linking up a sequence of different types of work at different stages: prior appraisal, in- depth analyses, mid-term overall evaluation, and ex post evaluation. This recommendation is inspired by Australian and Canadian experience. It was adopted by internal evaluation system (SEM 2000). the Commission its own in This type of programme should enable different exercises to be linked together and to benefit at any time from lessons from past experience, irrespective of the programming period from which they are drawn. it possible to coordinate evaluation work Moreover, commissioned at different levels in the same territory, and to avoid the same people being questioned repeatedly in surveys. it should make Several examples exist where commissioners considered it preferable to request a single team to carry out a series of evaluations in the same region or domain of the advantage of guaranteeing automatic coordination of evaluations since they are all it cannot be extended carried out by the same supplier. However, indefinitely without compromising the independence of the evaluations. intervention. This approach has Involving decision-makers through an evaluation mandate Even if an evaluation is not mandatory, the decision to evaluate (see page 65) remains a political act resulting from a decision taken by the main commissioner. Experience has shown that it is preferable for this decision to be formalised in a brief, clear and visible document, often called the is particularly valuable when "evaluation mandate". This document personally signed by the highest authority concerned by the evaluation. from The mandate specifies what is going to be evaluated (the scope of the evaluation), the motives for the launching of the evaluation and the use expected it. The main commissioner specifies whether or not recommendations are expected to be made at the end of the evaluation. He or she states a few main questions for which the evaluation must find answers, which generally gives an initial idea of the evaluation criteria, and sets a deadline for submitting the report. He or she may also determine a budget. Guaranteeing the success of an evaluation By proceeding in this way, the commissioner sets the limits which seem reasonable and will apply to the entire evaluation process. The mandate must nevertheless avoid restricting the steering group's and evaluation team's freedom. It must not become an instrument of preventive censure. Its main advantage is, on the contrary, to concentrate the evaluation on strategic subjects of interest at the highest levels. The mandate also ensures that evaluation does not move onto politically risky ground where it could be totally rejected when the results are released. The mandate aims at preventing a number of excesses, as listed below. • Strategic issues have given way to the short-term concerns of the managers. • The authorities do not feel committed by the conclusions, and those high up in the hierarchy do not allow their subordinates to participate in the evaluation. • The evaluation team acts outside its ambit. • The evaluation report arrives too late and is therefore impossible to use. • The evaluation tends to cover too much and cannot reach conclusions. Another advantage of the mandate is that it establishes clear and realistic rules of the game. In this perspective, the signatory authority officially hands over responsibility for the evaluation process to a person, preferably assisted by a steering group. At the same time the composition of the steering group and the limits of its missions are determined. Investing in t he preparation of the t e r ms of r e f e r e n ce to be bureaucratic, scarcely orientated Experience has shown that the terms of reference are often drawn up hastily, based on standard models or documents. This could well cause the local evaluation particularities, badly-suited to the programme issues and hardly motivating for the managers. For as long as the writing up of the terms of reference is considered an office job, it will be difficult to make the necessary choices for it to be properly adapted to the needs of its future users. towards The situation is different if, from the outset, the commissioner officially mandates a steering group for the entire duration of the evaluation. This group can then be given an active role in the preparation of the terms of reference (see page 75). This implies the organisation of one or two or even three meetings which will, above all, make it possible to unambiguously Evaluation design and management formulate evaluative questions, check that the essential (or embarrassing) questions have not been forgotten, reject questions which are more a matter of auditing or control, select priority questions, and specify the judgement criteria and norms. Experience has shown that, after this type of preliminary study, several different evaluation strategies are envisaged, choices have to be made, and the quality of the evaluation is finally improved. An extreme but salutary choice may be the postponement or cancellation of the evaluation, when the steering group foresees that its results will not be used. The preliminary study is not an additional task but rather an alternative way of preparing the evaluation terms of reference. The two meetings devoted to it by the steering group provide an opportunity to do what is usually done by the evaluation team in the definition of its tasks, as Box 47 shows. Box 47 - Alternative process for evaluation Current practice Terms of reference Commissioner Steering group Recommendation Mandate Preliminary study, evaluability study terms of reference Outputs conclusions recommendations Evaluation team Outputs conclusions Dissemination utilisation Commissioner Dissemination utilisation Steering group Recommendations This process consists of going back to the programme rationale and strategy, recapitulating its objectives, and highlighting the order of priority and the articulation of these objectives. At this stage the steering group checks whether there is any misunderstanding about the programme logic and, where necessary, clarifies it. During the preparatory meetings, the steering group carefully formulates the evaluative questions and selects the most important ones. It expresses an opinion on the judgement criteria and benchmarks which are to be used by the evaluation team. Guaranteeing the success of an evaluation In the preparatory study, the programme managers perform certain tasks which only they are competent and authorised to perform. They must strive to avoid the frequent situation where the evaluation team reproduces, with a loss of quality, information already known to the administration. The team must, on the contrary, focus its work on the production and analysis of new data. The evaluability study is a similar exercise which may be combined with preparatory meetings. It consists of technical control recommended by several North American authors, and concerns evaluation in general and in particular. The study reviews the level of available each question knowledge and concludes on whether the technical and institutional conditions will be met so that reliable and credible answers can be found to the questions asked. This diagnosis is performed rapidly by an evaluation methodologist (a member of an evaluation unit or an outside expert). It results in the elimination of certain questions, considered too difficult or for which answers already exist. The evaluability study defines the main lines of the evaluation method so that the call for tenders can be directed towards those teams with the most appropriate competencies. It also allows for the adjustment of the number of questions asked and their level of difficulty, in relation to the financial and human resources available for the evaluation. This, in itself, is a good reason to invest in the preparation of the terms of reference. The unrealistic ambitions of certain terms of reference are frequently the cause of dissatisfaction for commissioners, and problems for evaluation teams. It is not rare to find terms of reference which require a quantitative analysis of multiple impacts yet which allocate only a few months and a limited budget. A study on about 30 impact analyses in Ireland showed that the average cost of in-depth evaluation was between 50 and 100 Keuro, and that it took from six to 12 months. Opening evaluation to more partners The concrete use of evaluation could also develop through a better balance between the independence of the evaluation and its interaction with the interaction programme managers. Experience has shown that strong between the evaluation team and the managers enhances the quality and utility of the exercise. It is furthermore preferable, in many cases, to involve more partners in the steering group, in order to increase the legitimacy and relevance of the conclusions. It is, however, advisable to strictly control the number and choice of people invited to participate in the evaluation, as well as the efficiency of the evaluation process. The organisation of real collective reflection can be organised within a larger steering group by using appropriate working models. Involving the m a n a g e rs to facilitate appropriation Use of external evaluation teams is justified by the need for competencies which the administration does not always have, whether they concern evaluation techniques or a particular impact mechanism. An external evaluation also has the advantage of allowing for a new, more objective and impartial view of the evaluated programme, and of enhancing the credibility of the results. It is the latter advantage which justifies the demand for independence, mentioned texts concerning evaluation. in all European Union The concern for independence must not, however, cause the evaluation team to work in isolation. In the domain of European Structural Funds, experience converges to show that interaction between the evaluation team and the managers strengthens the managers' motivation, avoids problems of access to information, and leads to a better use of the results. By contrast, the credibility of the evaluation team may be weakened if its independence is accompanied by a lack of familiarity with administrative systems, or if it is mistrusted and consequently induces defensive attitudes. Because a lack of contact with the managers is always prejudicial, they must be required to be sufficiently available, as must the people appointed to perform certain tasks. Sufficient time must also be set aside for interaction with the evaluation team at each intermediate step. Evaluation design and management To avoid blockages, a common sense rule consists of setting up systems of feedback to the managers during the course of the evaluation. The steering group can circulate intermediate reports and the first results of surveys or analyses. This solution has the advantage of making it possible to correct factual errors which are easily detected by the managers. In at least two cases (the Brandebourg Land and the Aquitaine region), this exchange of information was computerised. For the evaluation exercise to be considered impartial, it is also possible to include several programme managers and/or operators in the steering group. These participants are asked to speak on their own behalf, to respect the confidentiality of the points of view expressed in the meetings, and not to act as "official representatives" of their institution. This practice has positive effects in terms of the quality and credibility of the work, in so far as the people invited to the steering group can voice the expectations and fears expressed by the managers. They may also attest to the impartiality of evaluation work. Another recommendation is to view evaluation not as the systematic search for problems and failures, but as a balanced examination which attaches sufficient importance to the identification of successes and best practice and to their explanation and development. In this way the image of evaluation can be enhanced. ■ Opening up the steering group to external partners • Pluralistic evaluation committee The implementation of European Structural Funds is characterised by joint funding and decision-making between several administrative levels. Even if the final responsibility for commissioning an evaluation is almost always at a single level, evaluation cannot do without information from other levels nor disregard expectations at other levels. In this context, it would be dangerous and futile to consider evaluation as a unilateral exercise controlled by single decision-makers for their exclusive use. For this reason, evaluation practice often involves the constitution of a steering group which allows the expression of diverse points of view (see page 67). Whenever a steering group exists and plays an important part in the evaluation, its expansion can be considered. As shown above, it is useful to Guaranteeing the success of an evaluation include a few managers and/or operators in the group to make evaluation more interesting and credible for that category of partners. The same approach can be adopted to include other partners in the group. Several North American authors recommend including in the evaluation the terms used are "stakeholders", or main partners concerned. The protagonists. Any person who is potentially a user of the recommendations, is interested in the information produced or is likely to win or lose in the process, can potentially be associated with the evaluation. Thus, there are clearly a large number of stakeholders concerned in evaluation. More simply, it is possible to divide them into large categories, usually consisting of funding authorities, managers, operators and concerned groups. The inclusion of a category of stakeholders in the steering group has the same advantages as those mentioned above, in respect of the participation of managers. The people invited into the group voice the hopes and fears expressed at their level, enhance the relevance of the questions asked and, by their presence, render the evaluation more credible. Wider diversity within the steering group brings together more points of view, facilitates their comparison, and helps to raise the level of consensus. Each participant adds her/his questions, value system, judgement criteria and norms. Provided it is able to take advantage of this plurality, the evaluation produces richer, more credible and more useful conclusions. When the expectations of several partners are taken into account, the results are likely to be accepted more willingly and with greater interest. These advantages are synthesised in Box 48. Box 48 - The advantages of a larger evaluation committee The constitution of an evaluation steering committee consisting of the different stakeholders in the programme makes it possible to guarantee: • better acceptance of the evaluation by those evaluated, by creating relations of trust; • easier access to information and a better understanding of the facts and events which took place while the programme was underway; • • interpretations and recommendations which take into account all the important points of view; the dissemination of conclusions and taking into account of recommendations more rapidly and informally. Evaluation design and management The people to include in the evaluation may be academics, company directors, leaders of professional syndicates, heads of non-governmental organisations, etc. The participants are always invited in a personal capacity and must undertake to respect the confidentiality of points of view expressed during meetings. In some cases of evaluations intended for accounting to the general public, the decision was made to involve citizens, selected randomly, or journalists. If the steering group is extended to about ten people with differing points of view, its role in the evaluation process becomes more visible and decisive. In such cases it is given a different name, reflecting its size, such as evaluation committee or evaluation authority. When an evaluation committee is formed, it is important to list all the stakeholders likely to have an interest in the evaluation and its results. For this purpose, the diagram in the following box, called a "sociogram", may be used. This diagram divides the stakeholders into three categories: political authorities (elected representatives, decision-makers or funding authorities), actors in the implementation (managers, operators, agencies) and the concerned groups (direct or indirect addressees). It is important to check whether the participants are available (and motivated enough) to participate regularly and actively in the different meetings of the group and to monitor its work. The composition of the committee must be as representative as possible of the stakeholders, without being too large (a maximum of 15 people). In the case of European programmes, representatives of the Directorates General as well as national and local management and decision-making authorities are present. Representatives of the addressees of the programme are also asked to participate alongside these institutional representatives. The question of whether or not to include the managers and representatives of the operators in this body is often posed in terms of the neutrality and independence of the evaluation. It seems that the advantages of including them outweigh the drawbacks, despite the substantial risk of these people assuming too much power because they are inevitably more motivated (it is their raison d'être) and more technically competent than the other members. A numeric balance between the three categories of actor provides guarantees in this respect. It is furthermore possible to take the precaution of appointing as chairperson of the committee a political authority which has no direct supervisory responsibility, or a renowned personality (the latter solution is the most Guaranteeing the success of an evaluation frequent one in situations of multiple partnership because it avoids all suspicion of bias). Box 49 - Sociogram of people concerned by an evaluation Political authorities. ,—"Funding authorities Actors in the implementation Direct addressees Indirect addressees Concerned V <w'™ers and losers) groups Enlarged steering group, ¡ Evaluation committee ! Example: Classification of the stakeholders of a rural development programme representatives, Some actors may belong to more than one of the three categories defined above. for example, are also decision-makers, Regional elected spokespersons for the concerned groups, and themselves beneficiaries of the funding. There are also actors which are both operators and representatives of the beneficiary group, such as Chambers of Agriculture. These organisations are interlocutors farmers' associations. Local administrations are similarly part of the three categories: decision-makers, operators and concerned groups. the public authorities and have strong links with for In relation to the diagram in Box 49, it is interesting to pause on the notion of partnership, a term commonly used in European vocabulary for the past decade. In the context of the administration of Structural Funds, the idea of partners, very close to that of stakeholders, has a fairly specific meaning. It denotes structures (partners) which can intervene in the definition and application of programmes financed by the Structural Funds. The distinction made above, between vertical and horizontal partners, is important in so far as only the former can legitimately define the evaluation mandate. However, they are all "stakeholders" in the programme and are likely to be involved in the evaluation procedure, according to the modalities defined. legitimately Evaluation design and management 9 Organisation of collective reflection When the evaluation setting consists of an extended body, its work is the same as that of a steering group, i. e. to check whether essential questions have been forgotten, select priority questions, finalise the writing up of questions, specify the judgement criteria and norms, validate the terms of reference, react to intermediate reports by the evaluation team, comment on conclusions, validate the recommendations of the evaluation team, and issue its own recommendations. Thus, the difference between a steering group and an evaluation committee is not so much in the nature of the work done as in the way it is done. Whereas a steering group is composed of a few administrative officials who are used to working together, an evaluation committee is more open, has more members and has to hold highly organised meetings to produce effective work. The following setting is generally chosen: the commissioner appoints the chairman of the committee, within the framework of her/his mandate, as well as a mediator. Under the responsibility of the chairman who chairs the debates, the mediator impartially supervises the preparation, organisation and minutes of the meetings. Both chairman and mediator must be sufficiently neutral to be accepted by all the stakeholders. The mediator must, moreover, have sound technical knowledge of evaluation. He or she may belong to an internal evaluation unit or be recruited from outside experts on the basis of a few days' work. The mediator prepares work sessions, if necessary by holding individual interviews with all or some of the members of the committee. He or she organises the discussion in such a way that it is productive and conclusive, and notes elements of consensus in the minutes. The work of a pluralistic evaluation committee includes phases of exploration during which the expression of differences of opinion is encouraged (as regards evaluative questions, judgement criteria, recommendations, etc. ). These exploratory phases have a maieutic dimension, for the mediator tries to elicit the expression of all points of view, without censuring minority opinions. The phases of creativity are followed by phases of searching for consensus which, in a spirit of impartiality, lead to the production of collective work (finalisation of the terms of reference, validation of the conclusions, production of recommendations). Volume 3 of this Collection, devoted to techniques tools, provides two examples of group work evaluation Guaranteeing the success of an evaluation (METAPLAN and the coloured vote) used to conduct these two phases of work rapidly, impartially and consensually. Must an evaluation be totally consensual? This question becomes important when the process is extended to numerous partners whose points of view differ. Most evaluations try to produce a list of conclusions accepted by all the partners, but is this not simply overlooking the fact that the plurality of points of view and opinions is a basic democratic principle? In several recent experiments the evaluation process was designed to accept and even explain the differences in the partners' opinions. One of these experiments, run as part of the MEANS programme, is described in Volume 4 of this Collection, which presents several technical innovations. This experiment was based on the principle of multicriteria-multijudge analysis. As its name suggests, the technique recognises that the members of an evaluation committee may not agree on the same priorities for the same criteria, and allows several evaluation criteria to be taken into consideration. For example, certain partners prefer a measure which has a favourable short-term employment impact, whereas others will opt for long- term growth of competitiveness. The two criteria are, moreover, not necessarily contradictory. By means of confidential individual interviews with several partners, the evaluation team clarified its choices of evaluation criteria. Despite significant disagreement on the weight to give to the different criteria, the partners classified the actions of the programme in very much the same way. By taking the risk of accepting divergent conclusions, the evaluation made it possible to reveal a partial but real area of consensus on certain effective or ineffective actions. This outcome seems preferable to that of an evaluation which tries to reach a totally consensual conclusion, but one which may well appear artificial. Consolidating evaluation techniques Today, the range of classical evaluation tools remains ill-suited to the evaluation of socio-economic programmes. The reasons for these technical difficulties are rooted in the specific characteristics of the programmes: entangled objectives, multiplicity of interventions, macro-economic dimension of the main expected impacts, and expectations of several co- decision-makers vis-à-vis the evaluation. This collection recapitulates and presents a series of short- and medium- term technical advances which should make it possible to adjust evaluation methods more satisfactorily to suit the context of European cohesion policy. The main progress made involves the collection of more reliable data, a higher level of impact analysis and a greater capacity in evaluation teams to reach conclusions. M a n a g i ng c a s es of a m b i g u o us or e n t a n g l ed objectives Socio-economic programmes generally have multiple objectives. At the ex ante evaluation stage, but sometimes at the following stages as well, the evaluation team notes ambiguities or inconsistencies in these multiple objectives. Should this pretext be taken to declare the programme unevaluable and to stop the work at this stage with a critical conclusion? Opting for the simple way out is too often the solution adopted by evaluation teams, with the commissioners' permission. If the evaluation team is to surpass this limit, it has to be able to judge successes and failures by means of evaluation criteria which go further than the objectives, when these are not defined clearly enough. A good solution consists of putting the following question to the managers and operators: "what impacts (positive / negative) do you think your programme will produce?" The answers to this question form a list of "expected impacts which can be used to reconstruct the logic" of the programme with the best chance of it being évaluable. This approach has often been used with success outside the domain of Structural Funds. In pilot studies carried out within the MEANS programme, a regional development programme was tested in an evaluation context. This experiment is presented in Volume 4 of the Collection. It uses the impact mapping technique which makes it possible to effectively associate several dozen managers and operators in efforts to clarify the programme logic. Evaluation design and management The experiment concerned the economic development programme co- funded and co-decided by the European Commission, the Spanish government and the Valencia region. The evaluation started with a series of meetings during which the partners identified expected impacts and sorted them into groups (e. g. "diversification and efficiency of energy resources"). Each group of impacts could thus be defined precisely enough to construct effectiveness criteria and choose impact indicators (e. g. "number of firms which use natural gas"). The multi-sectoral and multi-objectives character of the programmes poses another evaluation problem - that of synergy. The measures or actions of a programme are generally evaluated as if they were independent from one another, without taking into account the fact that the impacts can reinforce one another (effect of synergy) or contradict one another (effect of negative synergy or "anti-synergy"). Synergy is rarely evaluated at the level of results and impacts for addressees. At best, factors of synergy are evaluated as the administrative coordination, the combination or the superimposition of interventions. A technical proposal was made within the framework of the MEANS programme. It consisted of the construction of a matrix of cross impacts, the search for the strongest potential synergy, and the verification of these hypotheses of synergy by means of a survey among addressees. This approach is presented in Volume 4 of this Collection. It is designed to be conducted without substantially increasing the volume of evaluation-related work. ■ A n a l y s i ng i m p a c ts b e t t er • Enhancing the reliability and relevance of data Many evaluation teams have complained about the insufficiency of monitoring data, in respect of both quantity and quality. Progress can certainly be made in terms of consolidation of monitoring systems and particularly in terms of the choice and quantification of indicators. In an ex ante or mid-term exercise, the evaluation team always plays an important part by assessing the quality of the programme indicators and, if necessary, recommending ways of improving it. Volume 2 of the Collection considers problems of quantification in detail and proposes a method for assessing the quality of indicators. Guaranteeing the success of an evaluation Considerable efforts have been made on a European scale, in the sense of regionalising statistics with a view to facilitating the quantification of context indicators. Given the size of the task and the investments required, it would probably warrant the continuation of these efforts in the next few years. Particularly promising perspectives are opening with the generalisation of geographical information systems. Progress could also be made in the sense of a better understanding of the roles which could be played by monitoring and evaluation in data collection. Monitoring data are collected nominatively, systematically, exhaustively and continuously. These data are expected to be completely reliable as regards inputs mobilised and programme outputs. It is more difficult to collect such information when it concerns results and, above all, impacts. To make the process less cumbersome, obtain more reliable information and avoid giving the impression of bureaucracy, it is often preferable to collect impact data by means of selective and anonymous surveys, limited to small samples. This data collection technique is the most suitable for in-depth analyses. There is therefore a fine balance that has to be found between the improvement of monitoring systems and the development of in-depth evaluations. Improving the analysis of macro-economic impacts The analysis of macro-economic impact mechanisms can also be improved. This type of analysis is a necessity for the evaluation of large programmes of which the main objectives are expressed on a regional or national scale. It is also a major difficulty because, despite a decade of efforts, evaluation techniques in this field have not yet reached an adequate level of development. Research and experiments should be continued, to schematise and model supply-side effects (see 144) which play an essential role in the production of sustainable impacts in terms of development. These effects are still difficult to take into account and therefore evaluate, but promising work on the subject is underway and should, hopefully, in the medium term, result in techniques which can be generalised. Developing thematic evaluations Evaluation tends to favour the analysis of those impacts which correspond to the objectives of the programme axes and measures. It must not, however, neglect the expectations of the political authorities at higher levels. Evaluation design and management At the European level a number of common priorities need to be taken into account, e. g. the environment, employment and equal opportunities. It is also expected, at the European level, that the success obtained by certain programmes in a particular domain of intervention (e. g. transport infrastructure, occupational training, R&D) will be evaluated in depth, for the purpose of reproducibility. Improvements can be made in the mainstreaming of these priorities, far too often neglected in evaluation reports. Thematic evaluation may be conceived either at the programme level, in which case it synthesises all the possible impacts of the different programme measures in relation to a given theme. Otherwise the impacts of similar interventions can be analysed and a thematic evaluation performed in a region or country, or even several countries. Volume 5 of the Collection proposes technical solutions for the performance of this type of evaluation. ■ H i g h l i g h t i ng o p e r a t i o n al c o n c l u s i o ns • Comparing performance Evaluation teams currently lack benchmarks for judging whether an objective is realistic or excessively ambitious (case of an ex ante evaluation), or whether a cost per job created is reasonable or excessively high (case of ex post evaluation). The result is evaluation reports without real conclusions, or with conclusions which focus on details of implementation and overlook questions of results and impacts. Progress which can still be made in the régionalisation of statistics and the harmonisation of indicators gives reason to hope that, in the long run, the heterogeneity of data used for programme evaluation will be reduced. This will afford interesting, albeit very new, prospects for multiplying comparisons between programmes and for constructing benchmarks. The advantage of such prospects certainly warrants continued efforts at harmonising indicators, particularly in relation to the results and impacts obtained. In the shorter term, it is also possible to progress through inter-regional cooperation involving two or more European regional partners. With this cooperation the various tools used can be compared, the actions can be assessed in terms of common criteria, the respective effectiveness of the two regions can be compared, and the evaluation can be finalised with clear and acceptable conclusions. This exercise, similar to the benchmarking Guaranteeing the success of an evaluation technique, was tested in an interesting way by the Emilie-Romagne and Pays-de-Loire regions, in the domain of rural tourism. Comparisons could also be made more frequently within each programme. They would allow each evaluation to conclude more clearly on the fact that certain measures are successful and others not. Volume 4 of this Collection makes several proposals in this respect. The first proposal is that scoring techniques should be used more often. These techniques are used extensively in the U. K. for selecting projects to be financed by the European Structural Funds. Thus, scoring is not really an innovation. Moreover, in numerous evaluations, including the evaluation of programmes co-financed by the Structural Funds. Scoring is used to formalise, in a comparable manner, the judgement made of several measures within a single programme. Provided it is based on explicit criteria and is impartial, this technique helps the evaluation team to produce real conclusions. it has been successfully used The second proposal is that multicriteria analysis be used to compare measures or actions of the evaluated programme. This proposal is more innovative because multicriteria analysis is generally used to evaluate variants of a project. With this technique progress can be made in the overall evaluation of multi-objectives programmes. The approach was tested in the MEANS programme (see Volume 4 of the Collection) and its use could be further developed. Considering the use of evaluation as a priority Use of evaluations is certainly the point which could be improved most. Since the probability of an evaluation being used is linked to its appropriation by users, considerable progress could certainly be made by further involving managers in the evaluation process and by opening it up to the outside, towards potential new users, as proposed above (see page 191). It is furthermore possible to envisage the integration of evaluation into a more global perspective of programme performance management. The striving for better performance is a legitimate concern, both at the European Union level and at that of a Member State or within a programme. This point is presented and developed in Volume 2 of the Collection. Considerable progress could probably also be made, in the short term, from the point of view of the use of evaluation as a tool for accounting to Evaluation design and management democratic institutions and citizens. Like the cohesion reports published regularly by the European Commission, each programme is likely to use evaluation work for recalling which problems have been treated and for showing the programme's contribution to their resolution. It would certainly be worth extending this democratic evaluation culture, currently limited to northern Europe. In this way it would be easier to meet the expectations of citizens and the public taxpayers, whose demands with regard authorities are constantly growing. to Annexes Annex 1 Process of ex ante, mid-term and ex post evaluation The tables in this annex make it possible to organise the evaluation process from A to Ζ by scanning all the stages described in this Volume. Three evaluation situations are presented in parallel: ex ante, mid-term and ex post evaluation. The considerations presented in this annex are of a general nature and must be completed by referring to the notes and guides disseminated by the national and European authorities, when such documents exist. Evaluation design and management Deciding on, designing and launching an evaluation ex ante < Page 65 Page 66 mid-term M Page 65 Page 66 ex post ► Page 65 Page 66 Deciding on the evaluation Defining the scope Specifying the motives for the evaluation and the expected use e. g. : improving the e. g. : proposing e. g. validating relevance and coherence of reallocation of funds best practice the programme Planning the participation of Page 67 Page 67 Page 67 the main partners in a steering grouo Including managers of Including the spokes­ measures and operators persons of the concerned (Page 191 ) groups (Page 192) Formalising the decision to Page 184 Page 184 Page 184 evaluate in a mandate and/or a pluri-annual evaluation programme Drawing up the terms of reference Particularly useful for not "forgetting" ex post evaluation Asking the partners to Page 69 Page 69 Page 69 express their expectations: selecting evaluative questions Primarily : Primarily : coherence, Primarily : effectiveness and judgement criteria; rationale, relevance effectiveness and efficiency and efficiency of specifying questions to consider in more detail and coherence of the first results the results and impacts Possibly : potential Possibly : Possibly : effectiveness, particularly relevance, given the evolution of the macro-economic evolution of the context concerned groups Recalling the regulatory Page 76 Page 76 Page 76 framework and describing the programme Programme proposal Including amendments decided during the implementation Listing available knowledge Page 76 Page 76 Page 76 Including monitoring data Including monitoring data and ex ante evaluation and mid-term evaluation Checking the evaluability of Pages 72 and 77 Pages 72 and 77 Pages 72 and 77 the questions and defining the main lines of the method Setting the 'ules of conduct. Page 78 Page 78 Page 78 the schedule and the suggested budget Annexes Deciding on, designing and launching an evaluation (contd. ) ex ante m i d - t e rm M ex post Launching the evaluation Defining the required skills Page 82 Page 82 Page 82 for the evaluation team, indicating the structure of the applications to make, having the terms of reference validated by the steering group, putting out the call (or tenders Selecting the evaluation team Planning the evaluation team's work with particular concern (or the quality of the final report Page 82 Page 169 Page 82 Page 169 Page 82 Page 169 Evaluation design and management Performing an evaluation ex ante 4 mid-term M ex post ► Examining the logic of the programme Analysing the strategy and Page 89 Page 89 Page 89 assessing its relevance: analysing the objectives and The assessment of Examining whether the Reconstructing the assessing their clarity; the relevance and clarity strategy remained relevant objectives which examining the internal and of the objectives and in view of a changing were followed external coherence of the of their coherence is context; showing how the de facto objectives particularly necessary managers and operators understood the objectives; making proposals for clarification if necessary Examining the coherence Page 92 Page 92 Page 92 between the objectives and modalities of implementation: budgetary allocations; modes of intervention; selection criteria for projects; division of responsibilities. This examination is especially necessary Identifying the results and Page 100 Page 100 Page 100 expected impacts Examining the quality of the Page 100 monitoring system, as well as Drawing up a logical Drawing up a logical diagram of impacts diagram of impacts (Page 95) Page 100 (Page 95) Page 100 that of the system of This examination is limited The examination of the indicators to the known monitoring existing system of principles. The examination indicators is especially of proposed indicators is necessary especially necessary. Annexes Performing an evaluation (contd. ) ex ante < mid-term M ex post ► Examining the programme effects Selecting and using existing Page 117 Page 117 Page 117 information Carrying out additional surveys Primarily : Including monitoring data Including monitoring data expertise, literature, statistics Page 121 Page 121 Page 121 For a more in-depth For a more in-depth knowledge of certain knowledge of certain results and impact results and impacts mechanisms (Page 151) (Page 151) Estimating results and Page 124 Page 124 Page 124 impacts Primarily by extrapolation Also owing to the more Also owing to the more from the impacts of in-depth analysis of in-depth analysis of similar interventions certain results and impact certain results and impacts (Page 126) mechanisms (Pages 110 and 153). (Pages 110 and 153) Possibly : Estimation of macro- economic impacts (Page 135) Possibly : Estimation of macro- economic impacts (Page 135) Formulating and validating the conclusions Interpreting the results of Page 128 Page 128 Page 128 surveys and analyses with the help of the steering group; e. g. : Judgement on the e. g. : Judgement on the preparing an impartial ambition of the objectives contribution of the different judgement on the value of and the probability measures to the success the programme of achieving them of the programme Writing up an evaluation Pages 131 and 200 Pages 131 and 200 Pages 131 and 200 report clearly and carefully, and formulating real conclusions by clearly answering evaluative questions. Assessing the quality of the Pages 133 and 169 Pages 133 and 169 Pages 133 and 169 report and validating it Evaluation design and management Communicating and using the results ex ante < mid-term M ex post ► Communicating conclusions and recommendations Discussing the Page 164 Page 164 Page 164 recommendations formulated by the evaluation team and e. g. : adjusting objectives; e. g. : improving the e. g. : highlighting best improving them clarifying objectives; modalities of practice; generalising improving the system implementation; lessons learned of indicators. improving the monitoring system, reallocating funds Deciding on a communication Page 163 Page 163 Page 163 plan as soon as the report is validated Organising a seminar to Page 164 Page 164 Page 164 present the conclusions and recommendations e. g. : presentation to the e. g. : presentation to the partners involved in the authorities responsible design of the next for the programme programme Disseminating a publication Page 166 Page 166 Page 166 and/or a brochure; issuing a press communiqué e. g. : brochure presenting the best results Monitoring the taking in to account of conclusi ons Asking the authority managing Page 52 Page 52 the programme to make a clear statement on the Integrating the conclusions Integrating the conclusions conclusions into the programming into the programme documents, including the management, including conclusions on the conclusions on the ambitions of the objectives reallocation of funds, the system of indicators and the selection of projects Integrating the results into national or European syntheses Pages 52 and 58 Pages 52 and 58 e. g. : national report on e. g. : European the implementation of Commission report on the Structural Funds social and economic cohesion Annex 2 Guide for drawing up the terms of reference is intended to give methodological This annex the elaboration of an evaluation project. The indications are based on experience gained through evaluations performed in administrations, and on our cooperation with the Conseil Scientifique de l'Evaluation (CSE, 1996). They must be adapted to each particular situation. indications for Origin of t he evaluation The terms of reference specify the regulatory framework of the evaluation, the project initiator, and the agent responsible for the "political mandate". Delimitation and description of the s c o pe of the evaluation The terms of reference describe the scope of the evaluation and may also define its limits. For example, is the entire programme to be evaluated or only certain aspects of it? Are the effects of diverse public actions to be evaluated with regard to society as a whole or to a particular social group only? This description of the scope covers at least the following three themes: • The objectives of the programme and the way they have been defined, as well as the social problems which motivated and which justify public action. • The intended or expected effects and the target groups. • The means implemented and the procedures used. If the objectives are badly defined or obsolete, it has to be decided whether the evaluation should reformulate the implicit objectives, based on the different logics applied. If so, the evaluation will be more fruitful when a preliminary synthesis of the objectives, even a limited and partial one, is presented in the terms of reference. At this stage it is advisable to attempt to "model" the evaluated public action; in other words, to describe the official "theory of action". What are the sequences of cause and effect that those responsible for the policy in question anticipated, even implicitly, causing them to recommend the Evaluation design and management solutions finally opted for? For example, a particular financial incentive will induce a particular type of behaviour in firms, etc. The social change that the public authorities wish to promote should be described according to the grid, with three points: inputs, outputs and outcomes. The latter point should be broken down into immediate objectives of the action, on the one hand, and the aims or ultimate, ideal justification that is never directly attained, on the other. If necessary, the terms of reference specify that the evaluation team must reconstruct the programme logic as it was implemented in the field. ■ Future use of evaluation results Defining the aims of the evaluation consists of clarifying what the initiator expects from it and why he or she believes it to be appropriate or necessary. Preliminary reflection and interaction is essential because the direct manager of the evaluated intervention does not always have the required objectivity to orientate the evaluation in the right direction. The main reason for which the evaluation is requested must be explained. Is the intention to change the strategy of the programme and if so, why? Is the intention to modify the monitoring procedures or project selection? Are resources to be re-allocated? In the same spirit, it is preferable for the terms of reference to anticipate the decision likely to be taken at the end of the evaluation. This anticipation is not intended to dictate the conclusions of the evaluation nor to limit the evaluation team's capacity to question the conditions stipulated in the terms of reference. The idea is to enable the team to have a better knowledge of the commissioners' requirements, thus avoiding misunderstandings as to their expectations. Whenever possible, the schedule for decisions to be made following the evaluation is to be specified. In order to enhance the future utility of the evaluation as much as possible, it is necessary to understand the context of the programme. The evaluation project must take into account key factors such as: degree of innovation of the action, the conflict arising from it, and the administrations' familiarity with the evaluation procedure. The interests and strategies of the different stakeholders must be recorded accurately to understand the motivations behind the actions and the conditions that need to be met if the evaluation is to be credible and its conclusions accepted by the actors. The key questions are: Annexes • Is the evaluated action a continuation of past policy or a break from it? • What are the areas of uncertainty and those for which the stakeholders have common references? • Who are the "legitimate decision-makers"? Who are the operators who implement the intervention? • Which groups could gain or lose from the action? The future users of the evaluation must also be carefully identified. The nature of relations between the actors (cooperation, competition, etc. ) must be specified by distinguishing three levels: political authorities, operators and concerned groups. ■ Evaluative questions and judgement criteria The terms of reference pose a small number of clear and concise questions, to which the evaluation team must provide answers in its conclusions. Whenever possible, each evaluative question integrates a judgement criterion, such as: • Is the action justified with regard to the social needs or problems to solve (relevance)? • Are there positive, negative or other unexpected consequences? • Could more beneficial effects be produced by adopting other implementation measures? (effectiveness) • Could more beneficial effects be produced with the same resources or could the same results be obtained with fewer resources? (efficiency) • Could better complementarity be achieved (coherence) or more synergy be produced within the programme, compared to related public interventions. Available knowledge and data The terms of reference mention what the existing sources of information are and whether impact explanation systems or comparison references are already available. It specifies the form of existing knowledge (e. g. raw data, information structured in a computerised data base, data already analysed). It also specifies who is responsible for gathering information (e. g. information made available to the evaluation team by a particular actor; information to be collected by the evaluation team itself). Evaluation design and management Methods Without determining the details of the method to use, the terms of reference must provide essential information for applicant teams to make reasonable methodological proposals, for example: • the required depth of impact analysis; • the importance of the credibility of conclusions, of the possibility of generalising results, and of the appropriation of these results by the actors; • if necessary, the sites that have already been chosen to carry out case studies and the reasons for these decisions; • the most important indicators from the commissioner's point of view. ; They must also present the main interpretation hypotheses for use in causality analysis: • Why does such an intervention seem to be a success or failure? • What environmental or exogenous factors (related policies, local contexts, etc. ) need to be taken into account for interpreting observable effects? S Evaluation setting The respective roles of the commissioner, the steering group and the evaluation team must be clarified. The terms of reference also specify the composition of the steering group and the modes of functioning (composition, chairperson, leader, secretary). Where relevant, the terms of reference specify whether the evaluation team is expected to make proposals to broaden the steering group, and to lead the meetings of this extended group. ■ Schedule The terms of reference set the deadlines for the evaluation teams' various tasks (intermediate report on a particular question; seminars; final report), as well as the final deadline for the submittal of these documents. ■ Inputs a l l o c a t ed A planned budget is determined by the commissioner. Where relevant, the amount is made known to all the applicant teams who will be judged Annexes primarily on the quality and relevance of their methodological proposals. Otherwise, the terms of reference include detailed methodological proposals and selection is based primarily on costs. Required qualifications The terms of reference specify the knowledge and experience required from the evaluation team. Where relevant, it grades qualifications by order of importance. ì Structure of the proposal To facilitate the selection of the evaluation team, the terms of reference oblige the applicant teams to organise their proposals in a certain way, for example: • elements of the evaluation scope and context which the candidate considers essential • reminder of the questions asked and possible interpretation of these questions • evaluation method proposed • detailed evaluation schedule • nominative composition of the proposed team, modalities of distribution of work and coordination within the team, and individual references of the team members • prices • technical, legal and financial references of the contracting organisation. Annex 3 Bibliographie references Boyle, R. , Lemaire, D. (eds), Building Evaluation Capacity, New Brunswick Publishers, (forthcoming). Commission Européenne (1997), Les Fonds structurels en 1996. Huitième rapport annuel, Luxembourg: European Communities official publications office, as well as earlier reports. Commission Européenne la cohésion économique et sociale, Luxembourg: European Communities official publications office. (1996), Premier rapport sur Commission Européenne (1997), Evaluating EU Expenditure Programmes. A guide for Ex post and Intermediate Evaluation, Luxembourg: European Communities official publications office. Conseil Scientifique de l'Evaluation (1996), Petit guide de l'évaluation des politiques publiques, Paris: CSE. Cook, TD. , Levitón, L. C. and Shadish, W. R. (1985), 'Progam Evaluation' in : Lindzey, G. and Aronson, E. (Eds. ), Handbook of Social Psychology, Vol. I. (3rd ed. ), New York : Random House. European Commission (1995), Common Guide for Monitoring and Interim Evaluation of CSFs, SPDs and operations, European Commission Structural Funds. GAO (1991), Designing Evaluations, Program Evaluation and Methodology Division, Washington. Guba, E. G. , Lincoln, Y. S. (1989), Fourth Generation Evaluation, Newbury Park: Sage Publications. Owen, M. O. , Rogers, P. (1998), Program Evaluation: Forms and Approaches, London: Sage Publications. Monnier, E, (1992), Evaluations de l'action des pouvoirs publics, Paris: CPE-Economica. Rieper, 0. , Toulemonde, J. (Eds) (1997), Politics and Practices of Evaluation, New Brunswick: Transaction Intergovernmental Publishers. Evaluation design and management Rist, R. C. , Segsworth, R. V. , et al. (1990), Program evaluation & the management of government : patterns and prospects across eight nations, New Brunswick : Transaction publishers. Rossi, P. , Freeman, H. E. (1993, 5th ed. ), Evaluation : A systematic Apprqach, Newbury Park, CA : Sage Publications. Vedung, E. (1997), Public Policy and Program Evaluation, New Brunswick: Transaction Publishers. Index Index accountability 17, 25 additional effect 112 additionality 31 addressee 121, 123, 192 aims of evaluation 76 appraisal 56 attractiveness 143 audit 62, 70 Β benchmarking 130, 201 black box 149 bottom-up approach 56 case study 126 coefficient 116 cognitive use 165 coherence 72 communication plan 164 Community priorities 53, 58 Community value added 25 comparison group 111,113,126, 155 competitive advantage 140 concentration 98 conceptual use 165 concerned group 22 conclusions 132, 161 confounding factor 119, 154 construct validity 114 context indicator 103, 119, 200 control group 110 cost-benefit analysis 57, 131 cost-effectiveness analysis 129 counterfactual situation 110,136,155 criteria 129, 130 D data analysis 176 data collection 123 deadweight 112, 113, 127 deductive analysis 153, 156 demand-side effect 146 democratic 20, 2 1, 46 direct feedback 25 displacement effect 112 dissemination 161 double loop learning 25 economic base 143 effectiveness 70, 129 efficiency 70, 129 employability 141 endogenous development 143 equity 72 evaluability 72 evaluability study 82, 187 evaluanda 65 evaluation committee 2 1, 190, 192 evaluation criterion 70 evaluation method 174 evaluation report 131, 133 evaluation setting 67, 68, 194 evaluative question 69, 72, 73, 186 evaluator 20 ex ante evaluation 31, 36, 37, 38, 52, 69, 70, 88, 89, 90, 92, 97, 98, 99, 100, 102, 104, 107, 127, 167 ex post evaluation 31, 53, 55, 88, 93, 97, 104, 125, 127, 147,200 exogenous development 143 exogenous factor 154, 155 expected effects 197 expected impacts 101 explanatory hypotheses 73 explanatory theory 156, 165 Evaluation design and management external coherence 95, 100 external evaluation 189 external validity 61, 115, 116, 121 externality 102, 144 extrapolation coefficient 120, 127 internal coherence 93, 100 internal validity 115 joint decision-making 27 feedback 183 focus group 124, 152 formative evaluation 25, 166 learning 24 logical diagram of impacts 94, 149 logical framework 148, 154 information geographical 125, 199 gross effect 112 system M H hierarchy of objectives 93 human capital 141 human resources 141 I impact 24, 62, 70, 87, 101, 126, 173 impact mapping 9, 197 implementation 97 in-depth analysis 199 independence 81, 189 indicative budget 78 indicator 5, 7, 40, 64, 129, 198 indirect feedback 25 individual interview 152 inductive analysis 153 infrastructure projects 57 input 87, 99 instrumental contribution 25 intermediate evaluation 37, 55, 71, 81 intermediate impact 149 135 macro-economic 124, maieutic 194 mainstreaming 9, 58 major project 60, 99 managerial evaluation 20 mandate 184 matching groups 155 matrix of cross impacts 9, 96,128,198 MEANS 5, 6, 40, 42 measure 60 measurement tool 129 mediator 130, 194 meta-evaluation 170 METAPLAN 130 methodologist 187 micro-economic 124, 135 mid-term evaluation 31, 38, 52, 88, 92, 93, 95,97, 99, 100, 102, 104, 107, 125, 127, 147 monitoring 62, 70, 175 monitoring committee 38, 68 monitoring system 104 multicriteria analysis 9, 130, 201 multiplier effect 135 Index N needs 24, 70, 87, 89, 129 net effect 112 network effect 144 normative use 166 objective 87 objectiveness 20 objectives tree 94, 148 officials 6 operator 121, 192 output 101 overall evaluation 117, 123 partnership 32, 41 pluralistic evaluation 21 pluri-annual evaluation plan 54, 55 pluri-annual programming of evaluation 183 policy 31 preliminary study 186 primary data 117, 121, 124, 175 productive environment 141 programme 30, 173, 177 programme indicator 103 programme logic 87, 147 programme manager 121 programming cycle 87 project promoter 122 protagonist 191 public 61 quality 169, 172 raison d'être 24, 89 rationale 89 recommendations 66, 132 reconstructed logic 197 regression analysis 126, 156 relevance 70, 90, 129 reproducibility 72 result 24, 25, 62, 70, 87, 100, 101 schedule 78 scope 65, 184 scoring 129, 201 scoring card 9 secondary data 117, 124, 175 shift-share analysis 144 single loop learning 25 socio-economic development programmes 6 sociogram 192 specific impact 101 stage 31 stakeholder 191, 193 stakeholder evaluation 21 standard 129 statistical sources 119 steering group 68, 190 step 31 stigmatisation effect 154 strategy 87, 90 structuring effect 137, 146 subsidiarity 24 substitution effect 112 summative evaluation 53, 166 supplier effect 135 supply-side effect 146, 199 sustainability 72 SWOT analysis 91, 130 synergy 56, 61, 72, 127, 198 Evaluation design and management terms of reference 68, 75, 76, 185 thematic evaluation 9, 61, 147 theory of action 149 transparency 25, 35, 37, 53 transversal 61, 132 u unexpected effect 154 user 161 Acknowledgements The preparation of this Volume took place over several years from 1993 to 1998. The preparatory work included the following steps: • Experts, the majority of them European, were brought together to participate in several workshops on the following themes: "Evaluating structuring effects", "Organising intermediate evaluations" and "Assessing the quality of evaluations". • These three workshops gave rise to the publication of a number of handbooks, the contents of which have been partially reused in this Volume. • Nearly three hundred European Commission civil servants participated in training seminars. The contents of these training courses gradually evolved according to the wishes of the participants. This Volume is built on the contents of these training courses. • Two reports were produced in the form of a memorandum about the performing of ex ante and ex post evaluations. The main points of the memorandum are presented in Annex 1 of this Volume. The production of this Volume was coordinated by Philippe Goybet and Anastassios Bougas (D. G. XVI. G. 2 : Coordination of Evaluation) and involved several Directorate­ Generals of the European Commission, notably D. G. II, V, VI, XIV and XVI, and in particular Miguel Benito Alonso, Andrea Mairate, David Sweet and René­Laurent Ballaguy (ENA). This Volume has been written by the Centre for European Evaluation Expertise, notably with the participation of Pascale Ambrois and Céline Fontaine. It was translated into English by Liz Libbrecht. The authors wish to thank all those who contributed, either directly or indirectly, to the production of this Volume, and in particular the following experts: H. Aalbu (Nordlands Forskning), J. Bachtier (EPRC, Glasgow), A. Blalock (Admiralty Inlet Consulting, Seattle), R. Boyle (Institute of Public Administration, Dublin), J. Bradley (ESRI, Dublin), R. Brancatti (Ministerio del Bilancio, Rome), E. Calderón (Universidad Politechnica, Madrid), E. Chelimsky (American Evaluation Society), Hagens J. (TERR Amersfoot), P. Nijkamp (Vrije Universiteit, Amsterdam), Β. Perret (Conseil Scientifique de l'Evaluation, Paris),. O. Rieper (AKF, Copenhagen), L. Senn (Università Bocconi, Milan), E. Stern (Tavistock Institute, London), L. I. Ström (SIR, Östersund). European Commission Evaluating socio-economic programmes Evaluation design and management Luxembourg: Office for Official Publications of the European Communities, 1999 1999 — 225 pp. — 14. 8 χ 21 cm ISBN 92-828-6626-2 Venta · Salg · Verkauf · Πωλήσεις · Sales · Vente · Vendita · Verkoop · Venda · Myynti · Försäljning Pavia Hatza 1 HR­10000 Zagreb Tel (385­11481 94 1 Fu (385­1) 481 94 1 MAGYARORSZAG Euro Info Service Europa Haz M a r o si >get , fn '■a í H­1396 Budapest 62 Tel I30­1) 350 80 2b Fax (36­1) 350 90 32 E­mail eu ι οι nlo β mail ma la v hu URL. http //www eurointo hu/inde» htm ROY International 4 1, Mehmar Hayarden Street PO Box 13056 61130 Tel Aviv Tel (972-3) 649 94 69 F ai (972-3) 648 60 39 E­mail royil O netvision nei il URL· http/A*ww. royjnt co il Sub­agent for the Palestinian Authority Index Information Services PO Bo»19502 BELGrQUE'BELGlE J » « n D « L i n n oy Avenue du Roi 202/Korungslaan 202 B­1190 Bruielles. 'Brussel Tel 132­2) 538 43 06 Fai (32­2) 538 08 41 E­mad («ende lannoyOrnroboardbe URL http//www iean­de­lannoy be La librairie européenne/ De E u r o p e e· Boekhandel Rue de la Lo. 244/WetstraaI 244 B­1040 Bruxeees/Brussel Tel (32-2) 295 26 39 Fu (32­2) 735 M 60 E­mail rnailOkbeuropbe URL http I www hbouroç Do Moniteur belge/Belgisch Staatsblad Rue de Louvam 40­42 Leuvenseneg 40­42 B 1000 Bru'eUes. Brussel Tèi (32­2) 552 22 11 Fax (32­2) 511 01 84 DANMARK J. H. Schultz Information A/S Herstedvang 10­12 DK­2620 Albertslund Tit (45)43 63 23 00 Fu (45) 43 63 19 69 E­mail Schultz O schwitz dk URL http y/www Schultz dk DEUTSCHLAND Bundesenzeiger Verlag GmbH Vartneosabletrung Amsterdam«! Sunn« 192 D 50735 Köln Tel ( 4 9 - 2 2 1 ) 97 66 80 Fu ( 4 9 - 2 2 1 ) 97 66 82 78 E­Mail veflnebObundesanzeiger de URL http //www bundesanzetger de ΕΛΛΑΔΑ/GREECE G. C. Eletlheroudakis SA International Bookslore Paneptstimiou 17 GR­10564 Altana Tel (30­1)33141 Β0. Ί/2/3/4/5 Fu (30-1) 323 98 21 E-mail elobOoksOnetor gr ESPANA Boletín Oficial def Estado Trafalgar. 27 E­280 71 Madrid Tel (34)915 38 21 11 (Ubros) 9 1 3 84 17 15(Suscnp | Fu (34) 915 38 21 21 iLibrosi 9 1 3 84 1 7 1 4 ( S u a c n p) E­mail dientes O com boe es URL http //www boe es Mundi Prensa Libro«. SA Caste»), 37 E­28001 Madrid Tel (34)914 36 37 00 F u ( 3 4) 9 1 5 75 39 98 E ma. ! libreria Q munotpren sa es URL http //www muncfcpronsa Com FRANCE Journal officiel Senne« des pubbeabons des CE 26 rue Dosati F­75727 Pans Cede« 15 Tel (33) 140 58 77 31 Fu (33) 140 58 77 00 URL http //www journal ottioel gouv li IRELAND Government Supplies Agency Publications Sed ion 4­5 Harcourt Road Dublin 2 Tel (353­1)66131 11 Fu (353­1)475 27 60 ITALIA Licosa SpA Via Duca di Calabria 1/1 Caseia poslaiu 552 1­50125 Firenze Tel (39) 0 55 64 83 I Fu (39) 055 64 12 57 E­mail kcosaOftbu: rt URL http //www Itone i/kctsa LUXEMBOURG Messageries du livre SAP. L 5. rue ílartfeisen L­2Í11 Luxembourg Tel (352)40 10 20 Fax (352) 49 06 61 E­mail mailOmdl lu URL http. "www mdi lu NEDERLAND SOU Servicecentrum Ullaevers Cnristotlol Pianti|nstraai 2 Postbjs 20014 2500 E A Den Haag Tel (31 70)378 98 80 Fu (31­70) 378 97 83 Email sduØsdunJ URL: Http. /www sdu m Mani'sehe Vertag*­ und Unlverattatsbuchhandlung GmbH Korimarkt 16 A­1014 Wien Tel. (43­1) 53 16 11 00 Fu (43­1) 53 16 11 67 E­Mail bestellen O m a ni co at URL: http /www man* al/index htm PORTUGAL ~ Distribuidora de U v r os Bertrand L d. " Grupo Bertrand, SA Rue das Terras dos Vales. 4­A Apartado 60037 f 2700 Amadora Tel (351­1)495 90 50 Fa» (351­11 496 02 55 Imprensa Nacional­Cas* da Moeda. EP Rua Marqjf ? Sa da Bandeira. 16­A Ρ­1050 Lisboa Code« Tel ( 3 5 1 - 1 ) 3 5 3 03 99 Fa« (351­1)353 02 94 E­mail del mcmOmail telepic p; URL: http 'www mem pt SUOMLFINLANH Keskuskatu 1. Con Ir alga lan ■ PL/PB 128 FIN­0010­ ΗβΙ»-ηκιΉβΚιηαίοηι Ρ/Dn (358-9) 121 44 18 F/lax (358-9) 121 44 35 Sahkûposli akatilausOakatcemmoncom UFIL. http //www akateeminen com SVERIGE B T J AB Traktorvägen 11 S 221 82 Lund T!n (46­46) 18 00 00 Fa» (46­46) 30 79 47 E post tjttou­putjCDij se URL: http//www blj se UNITED KINGDOM The Stationery Office Llri IMernabonal Rales Agency 51 Nint E»ms Lane London SW8 50R Tel (44­171)973 90 90 Fa» (44­171)873 84 63 E­mail ipa enquiñes O Ineso co uk URL: hnp/'wwwthe­stationery­otricecouk ISLAND Bokabud Larusar Blonda! Skolavordustig 2 IS­101 Reykiavik Tel (354) 551 56 50 Fa» (354) 552 55 60 NORGE Swets Norge AS Oslonjoveion 18 Boks 6512 Erterstad NOOOeOskj Tel (47-22)97 45 00 Fa» (47­22) 97 45 45 SLHWEIZ. SU1SSESVIZ2ERA Euro Info Center Schweiz c/o OSEC Siamplenbachsuaße 85 PF 492 CH 8035 Zurich Tel (41-TJ365 53 15 Fa» (41­1)365 5411 E­mail ercsOosecch URL: hrtp/flivww osee ch/ecs B A L G A R UA Europress Euromedia L id 59. Hvd Vilosria BO· 1000 S o ta Tel (359­2)980 37 66 Fa» (359­2) 980 42 30 E­mail MilenaOmoox cil bg CESKA REPUBLIKA USIS NI S­prod3|na HsWesstMfi ?2 CZ­130 00 Praha3 Tei (420­2)24 2 3 14 86 Fax (420­2) 24 23 11 14 E m a il nfcpospOdecnis cz URL: http usiscr cz CYPRUS Cyprus Chamber of Commerce and Industry PO Bo» 1455 CV­1509 Nicosia Tel (357-2)66 95 00 Fa» (357­2) 66 10 44 E­mail demetrapOccn orgcy MALTA Miller Distributors Ltd Malia ime m abonai Airport PO Bo»25 Luqa LOA 05 Tel (356)66 44 88 Fa» (356167 67 99 E­mail gwirthOusa. net POLSKA Ars Polena KrakowskiP Prz ed míesete 7 Skr pocztowa 1001 PL­00­ 950 Warszawa Tel (­10­22)826 12 01 Fax (40­22) 826 62 40 E­mail ars_polObevy hsncom pi ROMANIA Sn G­tal Berthelol Nr 41 RO­70749 Bucuresti Tel (40­1)31544 03 Fax (40­1 314 22 66 CCEC 60­le try a Oktyabrya A v 9 '17312 Moscow Tel (7­095) 135 52 27 Fax (7­095) 135 52 27 SLOVAKIA Centtum vn SU Nam Stobody. 19 SK­81223 Bratislava Tel. (421­7)54 41 83 64 Fa» (421­7) 54 41 83 64 E­mail e u m p e t b bt sltk stuba sk URL http //www stik stuba Sk SLOVENIJA Gospodarskl Vestnlk Dunajska costa 5 SLO­l000L|ubl|an. i Tel (366)613 09 16 40 Fax 1386) 613 09 16 45 E­mail europCgvestnik si URL http//www gvesinik. si TURKIYE Dünya Infolel AS 100. Υ Ί Mahailossi 34440 TR-80050 Bagcilai-Istanbul Tel (90-2121629 4G 89 Fu (90-212) 629 46 27 E-mail mlotelOdun/a-gazeto corn tr AUSTRALIA Humer Ρ PO Bo» 404 3067 Abbots'ord. Vtctona Tel (61-3)94 17 53 61 Fa» (61-3) 94 19 71 54 E-mad jpdavtesOozomailcomau CANADA Les editions La Liberté Inc. 3020. chemin Sainte­Foy G1X 3V Samte­ Foy Quebec Tel (1­418)658 37 63 Fax (1­600) 567 54 49 E­mm irbenoOmedicmqcca Renout Publishing Co. Ltd 536P Chemin Canotek Road Unit 1 KU JJ3 Ottawa. Ontano Tel (1­613)745 2 6 65 Fax (1­613)745 76 60 E­mail order deptOreno­. fbooks com URL http //www renoufboofcs com EGYPT The Middle East Observer 41 S hent Street Cairo Tel (20­2)392 69 19 Fa» (20­2) 393 97 32 E­ma4 mafoudaOmeobservercomeg URL: http://www meobserver com eg EESTI INDIA Eesll Kau ban d us ­Tôoet us koda (Estonian Chamber of Commerce and Industry) Toom-Kook 17 EE­0001 Tillinn Tel ( 3 7 2 ) 6 46 02 44 Fu (372) 6 46 02 45 E­mail emloOkodaee URL http www koda ee EBIC India 3rd Floor, Υ B Chavan Ceñiré Gen J Bhosaie Marg 400 C21 Mumbai Tel (91­22)282 60 64 Fa» (91­22) 285 45 64 E­ma·! ebcOgtasbmOI vsnlnetir URL http //www eocindta com JAPAN PSI­Japan Asaht Sanbancho Plaza «206 7­1 Sanbancho Chrvoda­ku Tokyo 102 Tel. (81­3)32 34 69 21 Fu (81­3) 32 34 69 IS E­mail: booksOpS'iap in. co |P URL: http:/Awww psi­japan com EBIC Malaysia Level 7. Wisma Hong Leong || J. I. 111 P| :,,. 50450 KuaU Lumpur Tel 160­3) 262 62 96 Fa» (60­3) 262 61 98 E­mail ebtc­kltfmol net my Mundi Prensa Mexico. SA dt CV Rio Panuco No 141 Colonia Cuauhtemoc MX 06500 Mexico, DF Tel ( 5 2 - 5 ) 5 33 56 58 Fu (52-5) 514 67 99 E­mail 101545 2361 Ocompuserve com EBIC Philippines 19th Floor. PS Bank Tower Sen Gil J. Puvat Ave cor Tindalo Si Manali City Metro Manilla Tel (63­2) 759 66 80 F ai (63­2) 759 66 90 E­mail occpcomOglobo corn ph URL http //www eeep com EBIC Sri Lanka Tra n 5 Asw Hotel 115 Sir chittampalam A G. iidifMir Mawallia Colombo2 Tel (94­1)074 71 50 78 Fu (94­1) 44 87 79 E­rrail ebtcsJOilmincrxn EBIC Thalland 29 Vanisse Building. Hin Floor E ■< Chrdtom Ploenchit 10330 Bangkok Tel (66-2) 655 06 27 Fax (66­2) 655 06 2b E­m. ­il ebrcbfckOksc15th. com U R I: http/www ebicbkk. org UNITED STATES OF AMERICA Beman Associates 4611­F Assembly Drive Lanham MD20706 Tel (1­80O)"/"4 44 47 (mHrco telephone) Fax (1­800) 865 34 50 (toll free fax) E­mail queryObernancom URL. httpJnwww beman com Bine wenden Sie sich sn ein Büro Ihrer Wahl/ Please c o n t a ci the sale« office of your choice/ Veuillez vous »dresser ■u bureau de vente de voira choix 2. rue Mercier L­2965 Luxembourg Tel '352) 29 29­42455 Fu (352) 29 29­42756 F­maa inlo mfoOopoce cec. be URL http //eut­oo eu mt The MEANS collection: "Evaluating socio-economic programmes" This six-volume set gives wider public access to the lessons of the MEANS programme which the European Commission introduced with a view to impro­ ving and promoting evaluation methods. Its publication meets a growing need for reliable benchmarks to deal with the problems of evaluation encoun­ tered when implementing public measures in general and Structural Fund ope­ rations in particular. In this context, the set constitutes a unique methodological guide that can solve the technical and organisational problems arising in the course of evaluation work and for which the existing literature provides little help. The collection is aimed not only at professional evaluators, for whom it will provide a valuable handbook, but also at non-specialists, in particular manag­ ing authorities and structural programme Monitoring Committees. Evaluation design and management Each volume is designed to be independent of the others, yet they are also complementary. Together, they deal with all aspects of the evaluation of socio­ economic programmes: I. II. Selection and use of indicators for monitoring and evaluation III. Principal evaluation techniques and tools IV. Technical solutions for evaluation within a partnership framework V Transversal evaluation of impacts on the environment, employment and other intervention priorities VI. Glossary of 300 concepts and technical terms OFFICE FOR OFFICIAL PUBLICATIONS OF THE EUROPEAN COMMUNITIES L-2985 Luxembourg