Evaluating socio-economic programmes : volume 6, Glossary of 300 concepts and technical terms.
EC Structural Funds iluating socio-economic programmes Glossary of 300 concepts and technical terms n Commission EC Structural Funds Evaluating socio-economic programmes Glossary of 300 concepts and technical terms Volume 6 European Commission MEANS is a programme of the European Commission. It is aimed at improving Methods of Evaluating Structural Policies. The MEANS Programme and Collection was supervised by Phillipe Goybet and Miguel Angel Benito Alonso, Unit for the Coordination of Evaluation in the Directorate for Coordination and Evaluation of Operations of DG XVI Coordination-evaluation@fmb. dg16. cec. be The realisation of the Programme was entrusted to the Centre for European Evaluation Expertise (C3E) under the direction of Eric Monnier and Jacques Toulemonde. Responsibility for the opinions expressed in this book lies solely with the authors of the text: C3E - 13 bis Place Jules Ferry - F - 69006 Lyon - France - c3e@c3e. fr A great deal of add tional information on the European Union is available on the Internet. It can be accessed through the Europa server (http://europa. eu. int) and on the Inforegio Website (http://inforegio. cec. eu. int). Cataloguing data can be found at the end of this publication. Luxembourg: Office for Official Publications of the European Communities, 1999 ISBN 92-828-6626-2 © European Communities, 1999 Reproduction is authorized, except for commercial purposes, provided the source is acknowledged. PRINTED ON WHITE CHLORINE-FREE PAPER Printed in Italy Foreword Developing good evaluation practices has become a priority area for the Commission and is directly relevant to the effective use of Community resources. Indeed, the sound financial management principles which the Commission seeks to promote via its SEM2000 (Sound and Efficient Management) Programme rely, specifically, on the regular evaluation of the Community activities. fe^isi j| ___. Jk ^L ^ ^ / ¿È ^k ^L __t I _K__i The Directorates General responsible for structural policy have considerable expe- rience in this area. Since 1988 they have made evaluation an integral part of each stage in the process of Community Structural intervention. Thus, the programme development stage includes a prospective evaluation; the implementation stage a strengthening of the monitoring systems and, at the end of the intervention, evaluation establishes what has been achieved in terms of programme impacts. However, the capacity of evaluation to contribute to the effectiveness and transparency of structural interventions rests ultimately on the integrity of the methods adopted to verify the credibility of results. This is why, four years ago, on the initiative for the Directorate General for Regional Policy and Cohesion (DG XVI), the MEANS programme was initiated as a response to the demand for developing evaluation methodo­ logy. This also include practices combining both general scientific requirements and the specific requirements of Community interventions whilst, at the same time, remaining accessible to a large audience. One of the more tangible results of the MEANS programme has been the publication of this six volume set by a group of independent experts, focu­ sing on the principal elements of socio-economic programme evaluation. A short summary of this work can be found at the beginning of the present volume. This collection covers the various methodological approaches and innovations that have taken place in the context of structural funds eva­ luation. It will enable those responsible for evaluation to develop their approach in a flexible and independent manner and for the various programme partners to exercise greater control over the evaluation pro­ cess. Attending the conferences of evaluation specialists and "clients" convened in Brussels (1995), Berlin (1996) and Seville (1998), I gained some impression of the interest that new evaluation techniques can gene­ rate. I would stress that the development of evaluation needs to progress in tandem with efforts to incorporate evaluation findings in the decision making process. This implies evaluation as a means of establishing whe­ ther policies are equitable, effective and relevant to the identified needs. In satisfying these requirements, evaluation should be seen as a valuable instrument affording both Member State and Community authorities the necessary guarantees that Community funds are being deployed in the best interests of the EU citizen. This is a cost worth paying when the goals of Economic and Social Cohesion and the European Union enlargement are at stake. Finally, I wish to extend my thanks to the members of the Committee of Independent Experts established at the launch of the programme. Their expertise and standing have contributed greatly to the quality and relevan­ ce of the MEANS programme. I therefore thank: E. Chelimsky (USA), ex- President of the "American Evaluation Association"; K. Kennedy (IRL), Director of the "Economic and Social Research Institute" (ESRI); J. R. Cuadrado Roura (SP), President of the "European Regional Science Association"; C. Seibel (F), Director of the "Statistiques démographiques et sociales de l'Institut National de la Statistique et des Etudes Economiques" (INSEE); L. Senn (I), Director of the Department of Economie Regional of the University Bocconi; E. Stern (U. K. ), Director of the unit of evaluation of the Tavistock Institute; A. Sorber (t) (NL), Head of the "Policy Analysis Department of the Ministry of Finance"; L. I. Strom (SV), Statens Institut för Regionalforsking (SIR); L. Tsoukalis (GR. ), professor at the University of Athens; H. Wollmann (D), professor at the University of Humboldt - Berlin and founder member of the "European Evaluation Society". E. Landaburu The MEANS Collection: "Evaluating socio-economic programmes" The evaluation of European Union structural interventions has entered into a phase of maturity, attested by the extension of work carried out by both the Member States and the Commission. Regulations based on decisions by the Council have largely contributed to the development of evaluation practice and have stimulated methodological research. initiated by the Commission Over the past ten years, the Commission has striven to promote and improve methods for evaluating Structural Funds. The main methodological conclusions have been debated and disseminated as part of the MEANS in 1991. programme Achievements to date provide enough material for a complete and accessible publication intended for a broad public: a six volume Collection covering all aspects of the evaluation of socio-economic programmes. These Volumes are written with a perspective that is broader than the current administrative framework, for this is likely to evolve more rapidly than professional standards. information on In general, the Collection is aimed at a readership of non- specialists seeking essential the evaluation of socio-economic development programmes. These programmes have multiple objectives, a partnership dimension and a multi-sectoral content - all characteristics which call for suitable evaluation methods and solutions not yet provided by the international technical literature. The complete Collection consists of 6 volumes designed to be complementary and to provide essential for professionals wanting to independently develop their own evaluation practices. Each volume can, however, be read and understood without having to refer to the other volumes in the Collection. recommendations information and Glossary of 300 concepts and technical terms The first Volume in the MEANS Collection entitled "Evaluation design and management" serves as an introduction both to the Collection and to the evaluation of socio-economic programmes. It enables readers to understand the implications of evaluation and to know what they can and cannot expect from an evaluation at a given time in a particular context. It covers all the elements needed to steer an evaluation project from A to Z: sharing of responsibilities, defining of the aims and method of an evaluation, establishing a schedule and budget, assessing the quality of the work performed and, finally, managing the use of conclusions. The second Volume focuses on the "Selection and use of indicators for monitoring and evaluation". It deals with the subject of indicators used for monitoring and evaluation. It shows how indicators can be used to support policy decisions, improve management and measure the overall performance of programmes. The volume proposes a methodological framework and a practical guide for the use of indicators in these different situations. The third Volume presents a selection of "Principal evaluation techniques and tools" applicable to the evaluation of socio-economic programmes. These tools are taken from relevant scientific disciplines: economics, sociology, management, geography, etc. Each one of the 23 tools is described in the form of an information sheet featuring a general description, the principles of its implementation, a list of its strengths and weaknesses for the evaluation of Structural Funds, an example of its application and a list of references. A number of methods adapted to the evaluation of socio­ economic programmes are the object of a more detailed description in the fourth Volume entitled "Technical solutions for evaluation within a partnership framework". Several pilot evaluations were implemented within the MEANS programme to design and test the following four innovations, the application of which is described in The MEANS Collection detail in this Volume: impact mapping, cross-impact matrix, score cards and multi-criteria analysis. The fifth Volume deals with "Transversal evaluation of impacts on the environment, employment and other intervention priorities" of the European Union and Member States. Among the questions addressed are impacts in terms of equal opportunities between men and women and the competitiveness of SMEs. The Volume provides the elements required for an analysis of those impacts which are not necessarily part of the explicit objectives of all the actions evaluated. Finally, the last Volume is a "Glossary of concepts and technical terms". The compilation of the Collection has necessitated the rigorous standardisation and definition of the terms used. Each of the first five volumes has an index of terms which, along with hundreds of definitions, are brought together in the glossary. Table of contents I Introduction II Evaluation Issues Utility of evaluation Types of evaluation Activities related to evaluation III Organisation of evaluation What is evaluated Stakeholders in evaluation Organisation of evaluation Preparation of the evaluation Quality of the evaluation IV Criteria and Indicators Internal logic of an intervention Evaluation criteria Measures and information systems Indicators V Impacts Causality analysis Socio-economic effects Effects of synergy Employment effects Economic and social development VI Techniques Evaluation methods Structuring techniques Information gathering techniques Analytical techniques Judgement techniques VII Appendix Bibliography French Alphabetical Index English Alphabetical Index 9 15 17 25 29 31 33 37 41 45 47 51 53 61 67 73 77 79 85 89 91 93 97 99 101 103 107 111 115 117 119 127 I Introduction Raison d'être of this Volume Evaluation is an exercise consisting of numerous interactions between the commissioner and the evaluation team, on the one hand, and between the various institutions concerned, on the other. The quality and utility of the evaluation depend to a large degree on the quality of these interactions. In­ teraction can, however, easily lead to misunderstanding if the vocabulary is not precise and accurate. Within the context of the European Union's socio-economic programmes, problems of vocabulary are frequently encountered because evaluation is performed in an international and ¡ntersectoral context, between people who share neither the same mother tongue language nor the same professional references. Moreover, due to the recent development of evaluation, many actors are unfamiliar with the standard vocabulary which is not always stabilised. The resulting communication problems can waste time and energy, and even compromise the quality or the utility of evaluation work. The purpose of this Volume is to contribute towards the stabilisation and clarification of the vocabulary. In the long run, it is also intended to bring the concepts used in different European languages closer together. The involved in commissioning, targeted public consists of all persons performing and using evaluations situated in the context of European economic and social cohesion policy. The book applies mainly to evaluation of a socio-economic development programme, but may also be of interest to readers who conduct or perform thematic evaluations or in-depth evaluations in more limited domains. That is why the term "public intervention" is systematically used in this Volume, in a generic sense, to denote any evaluation object. The choices that were made This glossary originated from the need to translate the work of the MEANS Programme (Evaluation Methods for Actions of a Structural Nature) into several languages. A first version in five languages was compiled in 1994 in the form of an internal Commission document. This initial effort was welcomed and attracted the attention of evaluation experts in various countries. Many of them spontaneously contributed to a second version created in 1996, with improved definitions in four additional languages. By demanding a rigorous effort to clarify and standardise the terms used, Glossary of 300 concepts and technical terms publication of the MEANS Collection, afforded an opportunity for expanding the glossary, enhancing its coherence and for publishing this Volume. Initially the Commission chose to favour the coherence of the glossary rather than the number of languages covered. This Volume is therefore a first step which covers English and French, and which includes a systematic translation from one language to the other. The work can subsequently be extended to other European languages. The words selected for this volume belong essentially to the evaluation vocabulary. An important place was also reserved for socio-economic impacts. Each of the first five volumes in the MEANS Collection is followed by an index which refers to this sixth volume. the European Commission and One of the difficulties encountered in the construction of this work derives from the fact that several definitions of the same word often co-exist, both at in scientific and professional communities. In these instances, a single, clear choice has consistently been made so that a coherent whole can be achieved. However, to facilitate the acceptance of the definitions chosen and to reduce misunderstandings, the main competing definitions and related terms currently used in other sectoral or national contexts are also mentioned. The definitions in this volume are presented thematically rather than alphabetically, along the same lines as the first versions of this glossary which were much appreciated by users. The themes covered are listed in the contents. An alphabetical index in French and in English can be found in the annex. The glossary can therefore be used in three different ways, as shown in the following box. Introduction How to use the glossary For readers who want to find the definition of a term >· Look for the term in the English index at the back of the book and go to the page on which that term is defined. For readers who want to translate a term from English to French or vice versa >■ Look for the term in the English or French index at the back of the book and go to the page on which that term is defined. The English terms are listed in italics. For readers who want to find their way around the vocabulary by starting with a theme > Look for the theme on the contents page and scan the pages concerning that theme. Λ",*-' Evaluation issues This part defines evaluation and its different forms, with an emphasis on the utility of the exercise and its value-added compared to similar exercises such as monitoring and control. For a complete presentation of these themes, the reader is referred to Volume 1 of the Collection, particularly the introduction and first part devoted to evaluation issues in the context of European economic and social cohesion policy. Utility of evaluation Evaluation Evaluation Judgement on the value of a public intervention with reference to criteria and explicit standards (e. g. its relevance, its efficiency). The judgement primarily concerns the needs which have to be met by the intervention, and the effects produced by it. The evaluation is based on information which is specially collected and interpreted to produce the judgement. For example: evaluation of the effectiveness of a programme, cost-benefit evaluation of a project, evaluation of the validity of a policy, and evaluation of the quality of a service delivered to the public. Certain definitions exclude the judgement dimension and limit evaluation to the measurement of the intervention's effects. Other, more restrictive definitions, limit evaluation to the ex post estimation of effects. In certain contexts, evaluation focuses not on a public intervention but on a public organisation (e. g. evaluation of a university or a hospital). More generally, the term is used in human resource management for the evaluation of a person (e. g. annual evaluation interview) or in the financial domain to estimate the value of an enterprise. Journalistic use of the word is often in the sense of a vague estimation (" the number of victims is estimated at 1,000 people") or a monetary estimation (" this table is valued at 1 million euros"). Related term(s) : Appraisal Assessment Appréciation Examen ■ Formative evaluation Evaluation endoformative Evaluation which is intended for managers and direct protagonists, in order to help them improve their action (feedback). Glossary of 300 concepts and technical terms Formative evaluation applies mainly to a public intervention during its implementation (on-going or intermediate evaluation). It focuses essentially on implementation procedures and their effectiveness and relevance. The distinction between formative and summative evaluation has wittily been summarised in the following way: "When the cook tastes the soup, it's formative; when the guests taste the soup, its summative". Related term(s) : Evaluation formative S u m m a t i ve evaluation Evaluation récapitulative S Evaluation performed for actors who are not directly involved in the management of the public intervention (political or professional institutions, foundations, the press, etc. ). It produces a global and distant assessment. Evaluation is said to be summative or "recapitulative" when it aims at accounting for actions, and when it serves both for judging a public intervention and for helping the authorities responsible decide on its launching, maintenance, postponement, suppression and reproduction. In such instances, the users are external funding bodies and possibly other social actors who may decide for their own purposes on the appropriateness of adopting better practices identified during the programme evaluation. Related term(s) : Evaluation sommative Managerial evaluation Evaluation manageriale ■ An evaluative approach integrated into the management of public interventions, and aimed at recommending changes related either to decision-making (feedback, instrumental use) or to the behaviour of the actors responsible for the implementation of the intervention. The general approach of managerial evaluation is similar to that of new public management, and is aimed at addressing the problem of stagnating Evaluation issues public revenue. The underlying question can be formulated as follows: how can the trade-off between the different sectoral policies be justified? The dominant approach here which occurs within the administration, is that of " optimisation " of budgetary resources. The political dimension of the evaluation act is overlooked. The quality of managerial evaluation is based on two principles: the independence of the evaluator and the objectivity of the method. For example, during the 1980s, most functions of the British administration were privatised or delegated to semi-independent agencies. Demands for effective services grew at the same time. This led to the development of Value For Money evaluations. Pluralistic evaluation Evaluation pluraliste Evaluative approach designed as a collective problem-solving process involving all the parties concerned. On the basis of reliable information acceptable to all, value judgements are formulated by seeking agreement within an evaluation authority consisting of political and administrative officials, as well as spokespersons for the groups concerned. In France, ad hoc evaluation authorities are frequently set up to manage evaluation work. Thus, in the framework of national evaluation of the prevention of natural hazards, the evaluation authority was composed of actors from diverse origins: ministries, local authorities, public agencies, associations, etc. Within the framework of policies co-financed by the State and the regions, the steering of certain evaluations is entrusted to political and administrative officials at both in partnership with representatives of the groups concerned. levels, Related term(s) : Partnership evaluation Stakeholder evaluation Evaluation partenariale Evaluation participative Glossary of 300 concepts and technical terms D e m o c r a t ic evaluation Evaluation démocratique Evaluative approach aimed at enhancing the quality and transparency of democratic debate, whether through the promotion of an awareness and understanding of the logic and effects of public intervention, or through a judgement of its raison d'être and effectiveness. Evaluation is defined as a "judgement on the value of the public intervention" by democratically elected personalities. Evaluation professionals are at the service of this authority and have an exclusively technical role. The process is public. This model applies in the case of an experimental intervention which, in view of the results obtained after the first few years, must be confirmed by a vote. Cognitive aim Finalité cognitive The fact that evaluation enables its users to become aware of new information on an intervention and its effects. The cognitive contribution of an evaluation is particularly visible when the reader of the evaluation report is surprised by what s/he reads, when s/he discovers something new. It generally corresponds to the production of new information (primary data), but may also relate to an evaluation which affords its users access to secondary data about which they were not previously aware. is the A dimension of the cognitive contribution of an evaluation transformation of the way in which users perceive the intervention, its effects and relations of cause and effect (impact explaining theory). In that sense it is a conceptual contribution. Irrespective of its aim, a successful evaluation always has a visible and recognised cognitive role. Related term(s) : Conceptual. Apport conceptuel. Evaluation issues Instrumental aim Finalité instrumentale The fact that the evaluation results directly in decisions to improve, extend or interrupt a public intervention. For example, an intermediate evaluation report recommended a series of twenty-six amendments to components of public intervention. A few months after the evaluation, the authorities responsible for the intervention agreed on a series of adjustments. Of the decisions taken, thirteen corresponded to recommendations made by the evaluation. realistic This contribution is optimised when the evaluation team and steering group formulate recommendations and when decision-makers appropriate them easily. In this spirit, the status of the recommendations is clearly explained by answering the following questions: Does the implementation of this recommendation have a financial implication? Can it intervene in the short term? Is there consensus on the recommendation? An instrumental use is difficult to prove because many other factors intervene at the time of decision-making. Normative aim Finalité normative The fact that the evaluation proposes arguments to form or modify judgements on an intervention. An evaluation may have a normative role when it helps to clarify criteria and define comparative norms, or when it identifies good practice and exemplary projects. This contribution is essential when the evaluation serves to account to political authorities and the general public. Users compare the conclusions of the evaluation with their own opinions. These conclusions therefore either maintain, support or change their personal judgement on the intervention. However, not all evaluations serve this purpose because public authorities are not always prepared to call into question their personal convictions. Feedback Rétroaction Feedback exists when the observation of results and impacts in the field is used to adjust the implementation of an intervention, or to make more Glossary of 300 concepts and technical terms radical changes, including calling intervention. into question the existence of the Feedback is the main purpose of evaluation when it has a managerial and/or formative perspective. It corresponds to an instrumental use. Related term(s) : Feed-back Organisational learning Apprentissage organisationnel The fact that lessons drawn from experience are accepted and retained by institutions or organisations responsible for intervention. The learning goes beyond feedback in so far as the lessons are capitalised on and can be applied to other interventions. Direct learning (called " single loop learning ") exists when the users learn that the intervention has been a success or failure. This can lead them, for example, to reduce the budgets allocated least successful interventions. to the Indirect learning (called " double loop learning ") exists when users realise that they must call into question their basic assumptions (their action theory). This may cause them, for example, to reorganise implementation of the least successful interventions. Related term(s) : Single loop learning Double loop learning Apprentissage en simple boucle Apprentissage en double boucle Accountability Obligation, for the actors participating in the introduction or implementation of a public intervention, to provide political authorities and the general Rendre des comptes Evaluation issues public with information and explanations on the expected and actual results of an intervention, with regard to the sound use of public resources. From a democratic perspective, accountability is an important dimension of evaluation. Public authorities are progressively their requirements for transparency vis-à-vis taxpayers, as to the sound use of funds they manage. In this spirit, evaluation must help to explain simply where public money was spent, what effects it produced and how the spending was justified. The addressees of this type of evaluation are obviously political authorities and, in fine, citizens via the media. increasing For example, a training organisation reports on the number of trainees who benefited from its services and the qualifications obtained. A managing authority reports on the cost per net job created due to the intervention. The Commission publishes a report on progress made in terms of economic and social cohesion. Related term(s) : Imputabilité Redevabilité, Rendu-compte Responsabilité Types of evaluation Overall evaluation Evaluation d'ensemble Evaluation of an intervention in its totality. Overall evaluation focuses on all actions financed as part of a public intervention. It encompasses all the tools used, all groups targeted, all eligible territories, all expected impacts and all relevant themes. Overall evaluation often constitutes the first step in the screening of an evaluated field, before a second stage of focalisation, which takes the form of in- depth or thematic evaluations. Related term(s) : Global evaluation Screening Radioscopie Thematic evaluation Evaluation thématique Evaluation which transversally analyses a particular point (a theme) in the context of several interventions within a single programme or of several programmes implemented in different countries or regions. The theme may correspond to an expected impact (e. g. competitiveness of SMEs) or to a field of interventions (e. g. R&D). The notion of thematic evaluation is similar to that of an in-depth study (e. g. impact of support for R&D on the competitiveness of SMEs), but it is a large scale exercise when conducted on a European scale. In-depth evaluation Evaluation approfondie Consists of focusing evaluation or a part of an evaluation precisely on a category of outputs, a group or on category of impacts. This permits a more detailed analysis, unlike overall evaluation which aims at being exhaustive. Glossary of 300 concepts and technical terms Overall evaluation of a programme may be accompanied by an in-depth analysis of one or more questions. A decision may, however, be taken to launch an evaluation devoted entirely to the in-depth analysis of a single question. By focusing on a specific question it is possible to use more demanding and therefore more reliable evaluation techniques, particularly for observing the behaviour of addressees, for observing a comparison group or for analysing net effects. Since in-depth analysis focuses essentially on the observation of effects in the field, these studies concern intermediate and ex post evaluation. Related term(s) : Scoping Focalisation '& Ex ante evaluation Evaluation ex ante ■ Evaluation which is performed before implementation. For an intervention to be evaluated ex ante, it must be known with enough precision; in other words, a plan, at least, must exist. If the intervention still has to be planned from scratch, one would refer to a diagnosis of needs. This form of evaluation helps to ensure that an intervention is as relevant and coherent as possible. Its conclusions are meant to be integrated at the time decisions are made. Ex ante evaluation mainly concerns an analysis of context. It provides the relevant authorities with a prior assessment of whether development issues have been diagnosed correctly, whether the strategy and objectives proposed are relevant, whether there is incoherence between them or in relation to Community policies and guidelines, whether the expected impacts are realistic, etc. Moreover, it provides the necessary basis for monitoring and future evaluations by ensuring that there are explicit and, where possible, quantified objectives. Related term(s) : Prior appraisal Needs assessment Appréciation ex ante Evaluation a priori, Diagnostic Evaluation issues Mid-term evaluation Evaluation à mi-parcours ■ Evaluation which is performed towards the middle of the period of implementation of the intervention. This evaluation critically considers the first outputs and results, which enables it to assess the quality of the monitoring and implementation. It shows the translation into operational terms of initial intentions and, where relevant, points out de facto amendments to objectives. Through comparison with the initial situation, it shows the evolution of the general economic and social context, and judges whether the objectives remain relevant. It examines whether the evolution of policies and priorities of other public authorities raises problems of coherence. It also helps to prepare adjustments and reprogramming, and to argue them in a transparent manner. Mid-term evaluation not only relies strongly on information derived from the monitoring system, but also on Information relating to the context and its evolution. Mid-term evaluation has a formative character: it provides feedback on interventions of which it helps to improve the management. Mid-term evaluation is a form of intermediate evaluation. Other intermediate evaluations may be performed during the first or last years of implementation. Related term(s) : Evaluation intermédiaire On-going evaluation Evaluation chemin-faisant ■ Evaluation which extends throughout the period of implementation of an intervention. This form of evaluation accompanies the monitoring of outputs and results. It is too often confused with monitoring. The advantage of on-going evaluation is that it allows for good collaboration between the evaluation team and programme managers, which in turn favours a better appropriation of conclusions and recommendations. Glossary of 300 concepts and technical terms On-going evaluation may be seen as a series of in-depth studies, comprising successive analyses of evaluative questions which have appeared during the implementation. For example, an on-going evaluation of development support for tourism has successively considered the following questions: "how has the public reacted to the proposed support?" (first year); "is the evolution of the tourist market making the assistance useless?" (third year); "which of the assisted projects can be considered cases of best practice to reproduce?" (fourth year). Related term(s) : Rolling evaluation Evaluation in itinere Ex post evaluation Evaluation ex post I Evaluation which recapitulates and judges an intervention when it is over. It aims at accounting for the use of resources, the achievement of expected effects (effectiveness) and of unexpected effects (utility), and for the efficiency of interventions. It strives to understand the factors of success or failure, as well as the sustainability of results and impacts. It also tries to draw conclusions which can be generalised to other interventions. For impacts to have the time to materialise, ex post evaluation must be performed two to three years after implementation of an intervention. For the purpose of analysing impacts, ex post evaluations are likely to involve field surveys and to take place over long periods of time. Related term(s) : Evaluation a posteriori M e t a - e v a l u a t i on Méta-évaluation Evaluation of another evaluation or of a series of evaluations. The criteria for judgement are generally reliability, credibility and utility. The term is sometimes applied to the auditing of an evaluative function of an organisation (verifying that the rules concerning evaluation have been applied professionally). Sometimes meta-evaluation is used to refer to a synthesis based on a series of evaluations. Activities related to evaluation Audit Controle Verification of the legality and regularity of the implementation of resources. Auditing makes a judgement in terms of general criteria and standards, known and clarified beforehand, applicable to an entire political or professional field, and not specific to a public intervention. For example, in the case of assistance to a SME, an audit will check whether eligibility criteria have been met and whether the beneficiary firms have complied with the rules governing the use of assistance. The auditor is thoroughly familiar with the standards (e. g. legal or accounting knowledge) and his or her intervention is mandatory and inspires respect and authority. The main purpose of an audit is to find mistakes and evidence of dysfunctioning. The idea is to obtain a dissuasive effect. With time, the terms "control" and "audit" have extended further afield to encompass more activities than those corresponding to the original definition provided above. For example, certain audits or controls check whether the outputs have been produced with an adequate degree of efficiency and quality. Others examine whether the results and performance are sufficient compared to those obtained by other similar interventions. Control and audit also apply to the verification of the existence and good functioning of monitoring and evaluation procedures. Related term(s) : Control Audit Monitoring An exhaustive and regular examination of the resources, outputs and results of public interventions. Monitoring is based on a system of coherent information including reports, reviews, balance sheets, indicators, etc. Monitoring system information is Suivi ■ Glossary of 300 concepts and technical terms obtained primarily from operators and is used essentially for steering public interventions. When monitoring includes a judgement, this judgement refers to the achievement of operational objectives. Monitoring is also intended to produce feedback and direct learning. It is generally the responsibility of the actors charged with implementation of an intervention. For example: monitoring of the consumption of budgets, monitoring of the meeting of deadlines, monitoring of the percentage of SMEs in beneficiary firms, monitoring of the level of qualifications obtained by trainees. Related term(s) : Management audit Performance audit Audit de performance Contrôle de gestion Pilotage Monitorage Additionality Additionnante The fact that Community support for economic and social development is not substituted for efforts by national governments; in other words, the fact that the beneficiary State's own financing remains, globally, at least equal to that which existed before the Structural Funds' contribution. Verification of the implementation of this principle is carried out at the national level in the context of financial control and not of evaluation as such. This term must not be confused with additionality which applies to the evaluation of the net effects of an intervention. Organisation of evaluation The following pages cover the main terms relating to the preparation of an evaluation, from the decision to launch it to the terms of reference. The question of the quality of evaluation is also addressed. For a complete presentation of these themes, the reader is referred to the second part of Volume 1 of the Collection, devoted to the preparation of the different forms of evaluation, and to the fourth part, which proposes an approach to the assessment of evaluation quality. What is evaluated Scope of evaluation Champ Precise definition of the evaluation object, of what is evaluated. The scope of the evaluation must be defined in at least four respects: operational (all or part of the domains of intervention, one or several related policies), institutional (all or part of the authorities), temporal (period taken into consideration) and geographical (one or more territories or parts of territories, a particular region, town, nature reserve, etc. ). In the context of European cohesion policy, the evaluation object may be a programme, a measure or a project. To remain sufficiently general, this book uses the term public intervention and applies it, generically, to any evaluation object. Related term(s) : Evaluand Evaluanda ■ Policy Politique A set of different activities (programmes, procedures, laws, rules) directed towards a single goal or general objective. These activities are often accumulated incrementally through the years. European economic and social cohesion policy is scheduled according to a precise time-frame, with a pluri-annual budget. This is not the case for the majority of policies, in the traditional sense of the term. Programme Programme Organised set of financial, organisational and human resources mobilised to achieve an objective or set of objectives in a given lapse of time. Glossary of 300 concepts and technical terms A programme is delimited in terms of a schedule and a budget. Programme objectives are defined beforehand; an effort is then made systematically to strive for coherence among these objectives. The three main steps in the life-cycle of a programme are design, implementation and ex post evaluation. A programme is always under the responsibility of an authority or several authorities who share the decision-making. Within the framework of European social and economic cohesion, programmes are generally broken down into axes, measures and projects Related term(s) : Axis Programme cycle Axe Cycle de programmation M e a s u re Mesure Within the framework of European economic and social cohesion policy, the basic unit of programme management, consisting of a set of similar projects and disposing of a precisely defined budget. Each measure has a particular management apparatus. Measures generally consist of projects. Many measures are implemented in the form of calls for projects. Related term(s) : Action Action ■ Project Non divisible operation, delimited in terms of schedule and budget, and placed under the responsibility of an operator. For example: creation of a new training branch, extension of a purification network, carrying out of a series of missions by a consultancy firm. Within Projet m Organisation of evaluation the framework of European economic and social policy, the operator requests assistance which, after a selection procedure, is either attributed or not by the managers of the programme. Particularly careful ex ante evaluations are made of major infrastructure projects, using the cost-benefit analysis technique. Intervention Intervention ■ Any action or operation carried out by public authorities regardless of its nature (policy, programme, measure or project). Within the framework of European Economic and Social Cohesion policy, interventions take the following forms: Operational Programmes and Community Initiatives and Support Programmes for technical assistance measures and innovatory actions. Means of interventions employed are grants, loans, subsidised interest rates, guarantees, participation in equity and risk capital schemes or other forms of financing. Finally, the major domains of intervention are basic infrastructure, the productive environment and human resources. In this Volume, the term intervention is systematically used to designate the object of evaluation. Stakeholders in evaluation Stakeholder Protagoniste Individuals, groups or organisations with an interest in the evaluated intervention or in the evaluation itself, particularly: authorities who decided on and financed the intervention, managers, operators, and spokespersons of the publics concerned. The stakeholders may have legitimate interests which must be taken into account in an evaluation. They may also have purely private interests which are not legitimately part of the evaluation. For example, in the case of an intervention which subsidises the creation of new hotels, the stakeholders are the funding authorities/managers, the new hoteliers (direct addressees), other professionals in tourism, former hoteliers facing competition from the assisted hotels, tourists, nature conservation associations, and building contractors. Related term(s) : Partner Partenaire Partie prenante ■ Funding authority Financeur Public institution which helps to finance an intervention. By extension, the term funding authority is also used for people who intervene on behalf of these institutions in the evaluation process: European Commission desk officers, officials from a national ministry; elected representatives from a regional or local authority. When private people or organisations are subsidised by the intervention, they also contribute part of the funds (leverage effect). These people or organisations are not, however, funding authorities but addressees. Glossary of 300 concepts and technical terms Related term(s) : Decision-maker Décideur Maître d'ouvrage M a n a g er Gestionnaire Public (sometimes private) organisation responsible for implementing an intervention. Within the framework of European economic and social policy, managers exist at the programme and measure levels. By extension, the term manager is also used for people who intervene in the evaluation process on behalf of these organisations. Related term(s) : Maitre d'oeuvre Operator Opérateur Organisation which implements, closest to the field, the resources allocated to an intervention. The operators are the last links in the chain of implementation of an intervention. They are at the source of information feeding into the include: a local administration which monitoring system. Examples distributes direct support to enterprises, a private company which has built and manages a telephone network, a training institute which trains people funded by interventions. Within the framework of European Union socio-economic programmes, the term final beneficiary is used to denote public or private organisations responsible for commissioning work or for distributing public assistance. Final beneficiaries are required to report on their actions regularly and systematically. They are therefore programme operators in the sense defined in this book. Related term(s) : Final beneficiary Bénéficiaire final Organisation of evaluation Project promoter Porteur de projet Public or private person or organisation which requests and possibly obtains assistance in the framework of an intervention for a given project (e. g. rehabilitating a run down urban site; creating a new training branch). A project promoter must be considered to be an operator if it receives public funds every year and if it has to report regularly and permanently on the project. In contrast, it must be considered an addressee if it receives limited funding for a single project. Direct addressee Destinataire direct Person or organisation directly affected by the intervention. Addressees receive support, services and information, and use facilities created with the support of the intervention (e. g. a family which uses a telephone network that has been improved with public intervention support, or a firm which has received assistance or advice). Some people may be addressees without necessarily belonging to the group targeted by the intervention. Similarly, the entire eligible group does not necessarily consist of addressees. The term "beneficiary" is often used in the sense given here to the term "direct addressee". In this glossary, the term "addressee" was chosen for two reasons: (1) the term "final beneficiary" has a different meaning in the evaluation system of European economic and social cohesion policy, and (2) the term "beneficiary" is associated with the idea of a positive effect, which can be misleading because the evaluation team may identify negative impacts when investigating effects on direct addressees. Related term(s) : End-user Recipient Bénéficiaire Usager Glossary of 300 concepts and technical terms Eligible public Public eligible ■ All the people and organisations for which an intervention is directly intended (e. g. people unemployed for over a year; entrepreneurs). An eligible public is composed of all potential direct addressees. Only some of these are affected by the intervention (see the notion of coverage rate). This concept is more limited than the term "concerned public" which also includes foreseen or unforeseen addressees. Related term(s) : Target group Public cible Indirect addressee Destinataire indirect i A person, group of persons or organisation which has no direct contact with an intervention, but which is affected by it via direct addressees (e. g. firms which have used technology transfer networks set up by a public intervention to innovate). Indirect effects are produced in the short or medium term, positively or negatively. The raison d'être of an intervention is to produce positive change for all its direct and indirect addressees. C o n c e r n ed group Public concerné All the persons or organisations which are affected or potentially affected, voluntarily or not, directly or indirectly, by the intervention. This concept is broader than the term "eligible public" (all the potential direct addressees) because it also includes indirect addressees. Related term(s) : Winners and losers Gagnants et perdants Organisation of evaluation Self-evaluation Auto-évaluation Evaluation of a public intervention by the organisation which participates directly in its implementation. This mode of organisation is rarely found in the context of programmes financed by the European Union. Due to cultural and geographic distance, it is more frequent in projects to assist developing countries. Internal evaluation Evaluation interne Evaluation of a public intervention by an evaluation team belonging to the administration responsible for the programme. Internal evaluation may be independent if the evaluation team has no hierarchical relation with the actors implementing the intervention. Otherwise, it would be self-evaluation. Related term(s) : In-house evaluation External evaluation Evaluation externe ■ Evaluation of a public intervention by people not belonging to the administration responsible for its implementation. For example, a team composed of private consultants, researchers or people belonging to public organisations unrelated to those responsible for the intervention. Commissionner Commanditaire Person or organisation which decides (or co-decides) to launch an evaluation. Glossary of 300 concepts and technical terms A commissioner has the advantage of making his or her decision visible by establishing a mandate. A commissioner may decide to steer the work of an evaluation team her/himself, or to constitute a steering group or evaluation authority for this purpose. Related term(s) : Mandant ■ Evaluation team Equipe d'évaluation M The people who perform the evaluation. An evaluation team selects and interprets secondary data, collects primary data, carries out analyses and produces the evaluation report. An evaluation team may be internal or external. It may consist of a group of several organisations (consortium) or contract out the work. It may even consists of a single person. In this Collection, the term "evaluation team" has been used rather than "evaluator". The latter term, found in the management model, cannot easily be applied to the other two models (pluralistic and democratic evaluation) in so far as it is an evaluation authority which makes the final judgement on the basis of the work produced by an evaluation team. Related term(s) : Evaluator Evaluateur Chargé d'évaluation Steering group Groupe de pilotage Limited group composed of officials from those administrations most directly concerned by an intervention, sometimes accompanied by experts. The steering group meets frequently and its working procedures are fairly informal. The steering group acts as a mediator between the commissioners and the evaluation team, but also between the different partners concerned by an intervention. Organisation of evaluation Evaluation C o m m i t t ee Instance d'évaluation Steering group extended to include the main stakeholders in an evaluated intervention. An evaluation may involve any person who is potentially a user of its recommendations, any person who has an interest in the information produced, and any person who is likely to win or lose in the process. The main categories of stakeholders are funding authorities, managers, operators and concerned groups. Stakeholders invited to join an evaluation authority express their hopes and fears ; they improve the relevance of the questions asked and their presence makes the evaluation more credible. A wider diversity of points of view facilitates their confrontation within the steering group, and helps to raise the degree of consensus in the interaction and the robustness of interpretations. Provided it can take advantage of the different points of view, pluralistic evaluation enhances the richness and credibility of its conclusions. When the expectations of several partners are taken into account, conclusions are more readily accepted and viewed with greater interest. Evaluation setting Dispositif d'évaluation All the modes of organisation of a given evaluation (relations established between the commission, the steering group and the evaluation team, involvement of stakeholders, organisation of work, schedule and evaluation method). The term evaluation protocol applies more restrictively to only the technical aspects of the system, without referring to its organisational aspects. The setting put in place for a given evaluation is integrated into evaluation system which is applied to the institutional context of the intervention evaluated. Evaluation system Système d'évaluation All the rules, institutions, procedures and resources which organise the evaluative function in a given institutional context. Glossary of 300 concepts and technical terms The evaluation system of European economic and social cohesion policy specifies the moment at which each programme must be evaluated, as well as the responsibility at the different evaluation stages. The system varies depending on the country. Each evaluation may be subject to a particular setting provided that the basic principles of the system as a whole are respected. Preparation of the evaluation M a n d a te Mandat Document signed by the commissioners to formalise their wish to evaluate an intervention. By means of a mandate, a commissioner delegates the performance of an evaluation to a steering group or evaluation authority. The mandate specifies what will be evaluated (the scope of the evaluation), the reasons for which the evaluation has been launched, and the expected use. The commissioner specifies the nature of the expected recommendations resulting from an evaluation. S/he states the main questions that the evaluation will have to answer, which generally gives an initial idea of the evaluation criteria. S/he fixes a final date for submitting the report and may also fix a budget. Several mandates may be prepared at the same time for each of the exercises—ex ante, intermediate and ex post—planned in the framework of a pluri-annual evaluation plan. Related term(s) : Evaluation plan Plan d'évaluation Pre-evaluation Préévaluation Rapid prior study, aimed at identifying stakeholders' expectations, at choosing and specifying evaluative questions, at checking whether it will be possible to answer them, at establishing the evaluation setting and at drawing up terms of reference. Where relevant, a pre-evaluation may recommend that an evaluation be cancelled or postponed, if the conditions for its smooth operation and its utilisation cannot be met. The technical part of a pre-evaluation is called an evaluability assessment. Glossary of 300 concepts and technical terms Evaluability a s s e s s m e nt Erude d'évaluabilité Technical part of the pre-evaluation, which takes stock of available knowledge and assesses whether technical and institutional conditions can be met in order for reliable and credible answers to be given to the questions asked. Concretely, it consists of checking whether an evaluation team using appropriate evaluation tools will be capable, in the time allowed and at a cost compatible with existing constraints, to answer evaluative questions with a strong probability of reaching useful conclusions. Terms of reference Cahier des charges The terms of reference define the work and the schedule that must be carried out by the evaluation team. It recalls the regulatory framework and specifies the scope of an evaluation. It states the main motives for an evaluation and the questions asked. It sums up available knowledge and outlines an evaluation method. It describes the distribution of the work and responsibilities among the people participating in an evaluation process. It fixes the schedule and, if possible, the budget. It specifies the qualifications required from candidate teams as well as the criteria to be used to select an evaluation team. It is generally completed by administrative annexes. Related term(s) : Job description Termes de référence Evaluative question Question evaluative Question asked by the commissioner in the terms of reference and which the evaluation team will have to answer. Theoretically, questions have (what happened?), causal (to what extent is what has happened really an effect of the intervention?) and normative (is the effect satisfactory?). An evaluation generally has several questions. three dimensions: descriptive Quality of the evaluation Credibility Crédibilité Quality of the results and conclusions of an evaluation when they are logically supported by empirical facts and justified by an analysis of valid data. Credibility depends on several factors, including: reliability of data, soundness of the method, but also the reputation of the evaluation team. Related term(s) : Trustworthiness Impartiality Impartialité Quality of conclusions and recommendations of an evaluation when they are justified by explicit judgement criteria and have not been influenced by personal or partisan considerations. An impartial evaluation takes into account the expectations, interpretations and judgement criteria of all legitimate stakeholders, Including those who have very little power or ability to express themselves, in order to give an opinion on the "Res publica". Impartiality is an essential element of the quality of an evaluation. Related term(s) : Neutrality Neutralité Reliability Quality of the collection of evaluation data when the protocol used makes it possible to produce similar information during repeated observations in identical conditions. Fiabilité Glossary of 300 concepts and technical terms Reliability depends on compliance with the rules of the art as regards to sampling and tools used for the collection and recording of quantitative and qualitative information. Moreover, sound reliability implies exhaustive data collection and the appropriateness of the evaluative questions asked. This notion is important not only for primary data but also for secondary data, the reliability of which must be carefully checked. Related term(s) : Objectivity Soundness Objectivité Robustesse Construct validity Validité de construction Quality of an evaluation method which faithfully reflects the changes or needs that are to be measured or identified. impacts are expressed Construct validity means that expected in sufficiently precise terms and that observations in the field allow for a reliable and sensitive analysis of the object of observation. For example, if the impact of support for innovation must be estimated by means of a survey on regional SMEs, the notion of innovation must have been defined precisely and the questions asked must correspond precisely to these definitions (e. g. number of new products or production procedures). Internal validity Validité interne Quality of an evaluation method which, as far as possible, limits biases imputable to data collection or processing techniques. For example, an intervention which, in directly or indirectly assisted SMEs, led to the doubling of the innovation rate per year for 1,000 jobs. This type of conclusion is sounder if the collection and analysis of data have very precisely taken into account all the specific aspects of the intervention and its context: categories of assisted firms, forms of innovation specific to the region, etc. To obtain better internal validity, it is necessary to strictly control Organisation of evaluation a series of parameters, otherwise an artificial situation may be created which would limit the possibility of generalising the conclusions (less external validity). External validity Validité externe Quality of an evaluation method which makes it possible to obtain conclusions that can be generalised to contexts (groups, areas, periods, etc. ) other than that of the intervention being evaluated. For example, evaluation makes it possible to conclude (1) that in directly or indirectly assisted SMEs the intervention helped to double the innovation rate and (2) that similar assistance, attributed to SMEs in other regions, would probably have exactly the same effect. Only strong external validity allows one to extrapolate from lessons learned during the implementation of the evaluated intervention. It is particularly sought after when the evaluation aims at identifying and validating best practice. External validity is also required when the evaluation uses conclusions of evaluations performed on similar interventions. Related term(s) : Representativeness Généralisabilité Représentativité IV Criteria and Indicators The following terms concern the logic of the intervention, from the needs it meets to the inputs it mobilises, the outputs it accomplishes and, finally, the impacts it produces. This is the logic which structures evaluation criteria and makes it possible to construct judgement criteria and indicators. For a complete presentation of these themes, the reader is referred to Volume 1 of the Collection, particularly the beginning of the third part entitled "evaluating the global logic of a programme", as well as Volume 2, especially the first part devoted to indicator-related terminology. Internal logic of an intervention Need Besoin Problem or difficulty affecting concerned groups, which the public intervention aims to solve or overcome. Ex ante evaluation verifies whether the needs used to justify an intervention are genuine. Mid-term evaluation may involve a survey on addressees, to reveal their needs and opinions. Needs are the judgement reference of evaluations which use relevance and usefulness criteria. Strategy Stratégie Selection of priority actions according to the urgency of needs to be met, the gravity of problems to be solved, and the chances of actions envisaged being successful. In the formulation of a strategy, objectives are selected and graded, and their levels of ambition determined. Not all territories and groups are concerned by the same development strategy. Ex ante evaluation examines whether the strategy is suited to the context and its probable evolution. Context Contexte The socio-economic environment in which an intervention is implemented. The term is used in its broadest sense. For example, in the case of interventions in favour of SMEs, the context includes the macro-economic situation and the framework conditions (tax laws, company law, etc. ). Related term(s): Framework condition Socio-economic environment Condition-cadre Environnement socio-économique Glossary of 300 concepts and technical terms Objective Objectif Clear, explicit and initial statement on the effects to be achieved by a public intervention. If the objectives are not stated implicitly, an evaluation (and particularly ex ante evaluation) may help to clarify them. A quantitative objective is stated in the form of indicators and a qualitative objective in the form of descriptors, e. g. : 30% of all outputs must be accomplished by the end of the third year; the public intervention must first benefit the long-term unemployed. Specific objectives concern the results and impacts of an intervention on direct addressees. A global objective corresponds to the aim of the intervention. The aim of an intervention is to produce an impact expressed in global terms, e. g. reducing regional disparities in development levels. Objectives may also be intermediate. Objectives which specify outputs to be produced are called operational objectives. If the objectives of a public intervention have not been clearly defined beforehand, the evaluation can try to clarify them afterwards. In that case, it is preferable to refer to implicit objectives. Related term(s): Aim Goal Global objective Intermediate objective Specific objective Operational objective Implicit objective But Finalité Objectif global Objectif intermédiaire Objectif spécifique Objectif opérationnel Objectif implicite Criteria and indicators Policy priority Priorité politique The funding authorities' wish that evaluation should examine certain impacts which were not stated as objectives when the intervention was launched, but which represent political priorities at that level. For example, in the framework of its economic and social cohesion policy, the European Union demands that evaluations systematically take into account impacts on the environment, on the competitiveness of SMEs, on the creation and maintenance of jobs, and on equal opportunities between men and women. Verifiable objective Objectif verifiable An objective stated in such a way that it will subsequently be possible to check whether or not it has been achieved. A way of making an objective verifiable is to quantify it by means of an indicator linked to two values (baseline and expected situation). An objective may also be verifiable if it is linked to a descriptor, i. e. a clear and precise qualitative statement on the expected effect. Related term(s): Quantitative objective Objectif quantitatif Implementation Mise en oeuvre The operational process needed to produce expected outputs. In the context of European Union socio-economic programmes, implementation comprises all or part of the following tasks: mobilising and distributing allocated inputs; assigning management responsibilities to operators; selecting calls for tenders for project promoters; and, lastly, selecting and financing projects. To monitor and improve implementation, a monitoring committee is set up, a system of information monitoring is launched, and audits and evaluations are performed. Glossary of 300 concepts and technical terms Related term(s): Implementation ■ Input Ressource Financial, human, material, organisational and regulatory means mobilised for the implementation of an intervention. For example, sixty people worked on implementing the programme; 3% of the project costs were spent on reducing effects on the environment. Monitoring and evaluation focus primarily on the inputs allocated by public authorities and used by operators to obtain outputs. In this Collection, private inputs mobilised by assisted firms, for example, are considered to be results of public intervention. The above definition gives a relatively broad meaning to the word "input". Some prefer to limit its use to financial or budgetary resources only. In this case, the word "activity" can be applied to the implementation of human and organisational resources. The term "financial outputs" is sometimes used in the sense of consumption of budgetary inputs. Related term(s): Activity Financial realisation Means Resource Activité Intrant Moyen Réalisation financière Effect Socio-economic change resulting directly or indirectly from an implemented intervention. Effet U Criteria and indicators Effects include the results and impacts of an intervention, whether positive or negative, expected or not. In certain cases, the term "effect" is wrongly used to include outputs. Output Réalisation ■ That which is financed and accomplished (or concretised) with the money allocated to an intervention. A project promoter undertakes to produce an output in immediate exchange for the support granted. If this is not accomplished, the support is withheld or must be partly or entirely refunded. Operators are responsible for outputs and must regularly and systematically report on them to the monitoring committee. Outputs may take the form of facilities or works (e. g. building of a road, rehabilitation of an urban wasteland; purification plant; tourist accommodation). They may also take the form of immaterial services (e. g. training, consultancy, information). Related term(s): Product Realisation Extrant Produit Result Résultat Advantage (or disadvantage) which direct addressees obtain at the end of their participation in a public intervention or as soon as a public facility has been completed. Results can be observed when an operator completes an action and accounts for the way in which allocated funds were spent and managed. At this point s/he may show, for example, that accessibility has been improved due to the construction of a road, or that the firms which have received advice claim to be satisfied. The operators may regularly monitor results. They have to adapt the implementation of the intervention according to the results obtained. Glossary of 300 concepts and technical terms Related term(s): Immediate outcome I m p a ct Impact A consequence affecting direct addressees following the end of their participation in an intervention or after the completion of public facilities, or else an indirect consequence affecting other addressees who may be winners or losers. impacts (specific Certain impacts) can be observed among direct addressees after a few months or in the longer term (e. g. the monitoring of assisted firms after two years). In the field of development support, these impacts are usually referred to as sustainable results. Some impacts appear indirectly (e. g. turnover generated for the suppliers of assisted firms). Others can be observed at the macro-economic or macro-social level (e. g. improvement of the image of the assisted region); these are global impacts. Evaluation is frequently used to examine one or more intermediate impacts, between specific and global impacts. Impacts may be positive or negative, expected or unexpected. Related term(s): Global impact Intermediate impact Outcome Specific impact Sustainable result Impact global Impact intermédiaire Impact spécifique Résultat durable U n e x p e c t ed effect An impact which is revealed by evaluation but was not foreseen when an intervention was launched. Effet inattendu Criteria and indicators inductive analysis Unexpected effects are revealed by techniques (particularly case studies). They are not part of the objectives. A positive unexpected effect may become an implicit objective. Negative unexpected effects are also called perverse effects. For example, if assistance is granted for the development of a territory and, for that purpose, the territory is classified "disadvantaged", it will become less attractive for business as a result. Related term(s) : Perverse effect Side effect Effet pervers Retombée Evaluation criteria Criterion Critère M Character, property or consequence of a public intervention on the basis of which a judgement will be formulated. For example, an employment incentive programme may be judged in terms of "costs per job created" or "percentage of support benefiting the long- term unemployed" (in the latter case, it is assumed that the higher the percentage, the better the intervention). An evaluation criterion must be explicit, that is, it must clearly show why the intervention will be judged better or worse. Criteria frequently used in evaluation are performance, effectiveness, equity and sustainability. Thus, evaluation criteria may refer to different social values. To be used in an evaluation, a criterion must be accompanied by a norm (level of success at which an intervention will be considered good in terms of this criterion). An intervention is generally judged in terms of several criteria. Related term(s): Social value Valeur sociale Norm Level that the intervention has to reach to be judged successful, in terms of a given criterion. For example, the cost per job created was satisfactory compared to a national norm based on a sample of comparable interventions. Norme Related term(s) : Standard Glossary of 300 concepts and technical terms Weighting Pondération Used to state that one criterion is of more or less importance than another one in the formulation of a global judgement on an intervention. The weighting of criteria can be formalised by expressing it as a percentage (the total being 100%). Multicriteria analysis also uses weighting. Rationale Raison d'être The fact that an intervention can be justified in relation to needs to satisfy or socio-economic problems to solve. Ex ante evaluation verifies the real existence of these needs and problems, and ensures that they cannot be met or solved by existing private or public initiatives. Thus, the inadequacy or shortcomings of other initiatives (whether private or public) are a fundamental element in the programme rationale, by virtue of the principle of subsidiarity. Related term(s) : Raison d'etre Bien-fondé Relevance Pertinence Appropriateness of the explicit objectives of an intervention, with regard to the socio-economic problems the intervention is meant to solve. Questions of relevance are particularly important in ex ante evaluation because the focus is on the strategy chosen or its justification. Within the framework of mid-term evaluation, it is advisable to check whether the this socio-economic context has evolved as expected and whether evolution calls into question the relevance of a particular initial objective. Related terms: Appropriateness Adéquation Criteria and indicators Eligibility Eligibilité The fact that a region, project or group of people has the required characteristics to benefit from an intervention or, more precisely, to receive assistance. For example, European regions eligible for support attributed to lagging development areas must have a GDP lower than 75% of the European Union average. Eligibility criteria follow directly from the rationale of the intervention. Disparity Disparité The fact that a region or group of people are in a situation which differs significantly from others. In general, the rationale of programmes financed by the Structural Funds consists of reducing socio-economic disparities between regions or social groups. Related term(s) : Inequality Inégalité Effectiveness Efficacité The fact that expected effects have been obtained and that objectives have been achieved. Effectiveness can be assessed by answering the following questions, for example: "Could more effects have been obtained by organising the implementation differently?" or "Which are the most successful operators or measures?". An effectiveness indicator is calculated by relating an output, result or impact indicator to a quantified objective. For example: the objective in terms of number of firms created was as high as 85%; the placement rate obtained by operator A is better than that obtained by operator B. For the sake of clarity, it may be useful to specify whether one is referring to the effectiveness of outputs, results or impacts. Glossary of 300 concepts and technical terms Related term(s): Efficacy Success Effectivité Réussite Succès Efficiency Efficience The fact that the effects were obtained at a reasonable cost. Efficiency may be assessed by answering the following questions, for example: "Could more effects have been obtained with the same budget?" or "Have other interventions obtained the same effects at a lower cost?". An indicator of efficiency inputs mobilised by the quantity of effects obtained. For example: the average cost of training a person who has been jobless for a long time is 2,000 euro; the intervention should achieve a cost per job created of less than 30,000 euro. is calculated by dividing the budgetary For the sake of clarity, it could be useful to specify whether the efficiency referred to relates to outputs, results or impacts. The efficiency of outputs is called the unit cost. Related term(s): Cost effectiveness Unit-cost. Value for money Coût-efficacité Coût unitaire Sustainability Durabilité The ability of effects to last in the middle or long term. Effects are sustainable if they last after the funding granted by the intervention has ceased. They are not sustainable if an activity is unable to generate its own resources, or if it is accompanied by negative effects, particularly on the environment, and if that leads to blockages or rejection. Criteria and indicators Related term(s) : Viabilité Pérennité Performance Performance The fact that effects were obtained at a reasonable cost and that the addressees are satisfied with them. For example: a water purification programme has a high performance rate if the per capita cost is limited compared to similar interventions, if the purification plants built comply with quality standards, and if the rivers concerned are less polluted. include qualitative dimensions. Efficiency and performance are two similar notions, but the latter extends, more broadly, to In certain contexts, performance concerns outputs and results but not impacts. In other contexts, the term applies mainly to either outputs (World Bank - Operations Evaluation Department) or impacts. The meaning of the word performance is not yet stable; it is therefore preferable to define it whenever it is used. Utility Utilité Í : The fact that the impacts obtained by an intervention correspond to society's needs and to the socio-economic problems to be solved. Utility is a very particular evaluation criterion because it disregards all reference to stated objectives of an intervention. It may be judicious to apply this criterion when objectives are badly defined or when there are many unexpected effects. The criterion must, however, be used with caution to avoid the evaluation team being influenced by personal considerations in their selection of important socio-economic needs or problems. Some authors have argued for a form of goal-free evaluation. Related term(s): Goal free evaluation Evaluation affranchie des objectifs Glossary of 300 concepts and technical terms Subsidiarity Subsidiante The principle which justifies that a public authority decides to implement an intervention rather than to leave it up to private initiative or another public authority. The principle of subsidiarity justifies public intervention when there are shortcomings in the private sector and when other public administration levels would not have been as effective. Subsidiarity is part of the rationale of an intervention. In the European context, subsidiarity means, for example, that the Community acts in those cases where an objective can be achieved better at the European level than at the level of Member States taken alone. This corresponds to the notion of Community value added which is frequently considered a criterion to take into account during the evaluation of programmes. Related term(s): Community value added Valeur ajoutée communautaire Measures and information systems Indicator Indicateur Measurement of an objective to achieve, a resource mobilised, an output accomplished, an effect obtained or a context variable (economic, social or environmental). The information provided by an indicator is of a quantitative nature and is used to measure facts or opinions (e. g. percentage of regional enterprises which have been assisted by public intervention; percentage of trainees who claim to be satisfied or highly satisfied). An indicator must, among other things, produce simple information which is easy to communicate and easily understood by both the provider and the user of the information. It must help the managers of public intervention to communicate, negotiate and decide. For that purpose, it should preferably be linked to a criterion on the success of the intervention. It must reflect precisely whatever it is meant to measure (validity of construction). The indicator and its measurement unit must be sensitive, that is to say, the quantity measured must vary significantly when a change occurs in the variable to be measured. Indicators may be specially constructed by the evaluation team and quantified by means of surveys or statistical data. They are often borrowed from the monitoring system or statistical series. An indicator may be elementary or derived from several other indicators in the form of ratios or indexes. Related term(s) : Quantitative data Sensitivity Donnée quantitative Sensibilité Measurement unit Used to observe a phenomenon, change or variable, and to place it on a quantitative scale. Unité de mesure Glossary of 300 concepts and technical terms A measurement unit allows for quantification. An elementary indicator is associated with a measurement unit and has only one dimension (e. g. km of motorway; number of training courses). Some measurement units are divisible and others not (e. g. 20. 3km were built; 30 trainees were qualified). Measurement units must be harmonised if indicators are to be comparable. Related term(s): Quantification Quantification Descriptor Descripteur A description, in the form of a concise, clear and stable statement, of an objective to achieve or an impact obtained. For example, one can examine the impact of a measure in terms of equal opportunities, and conclude by choosing a pre-established descriptor such as: (1) "the measurement enabled most women addressees to qualify for jobs considered to be reserved for men", or (2) "the measure attracted a large proportion of women addressees who wanted to qualify for jobs considered to be reserved for men". The organisation of descriptors in the form of a structured grid may constitute the first step in the construction of an indicator. If several descriptors have been established beforehand, they can be used to construct an observation grid. By means of this grid a phenomenon or change can be observed and described in a qualitative and structured way. Evaluation cannot afford to exclude from its scope of analysis an important objective or impact simply because it is difficult to measure quantitatively when in fact it is considered to be important. In that case, it is preferable to collect qualitative data and to structure them by means of descriptors. Related term(s) Observation grid Qualitative data Statement Donnée qualitative Enoncé Grille d'observation Criteria and indicators Scoring Notation ■ Choice of a level on a scale graduated in quantitative measurement units (e. g. a scale of 0 to 100 or -3 to +3) in order to represent the significance of an effect, need or element of quality. It is possible to construct an observation grid which is sufficiently structured to directly produce a score. The person who chooses the score is called the scorer or the assessor. Statistical analysis of results studies scoring systems. Related term(s): Rating Monitoring information system Système d'information de suivi Measures taken so that operators can collect and store information on inputs, outputs and results, and so that they can process and transmit this information regularly to the managers (e. g. progress reports). The monitoring information system also includes the monitors' syntheses and aggregations, periodically presented to the authorities responsible for the implementation (reviews, operating reports, indicators, etc. ). In European Union socio-economic programmes, the key element in an information system is a system of indicators. Related term(s): System of indicators Système d'indicateurs Operating report Tableau de bord Presentation of essential information for the monitoring of an intervention, in the form of a small number of periodically quantified indicators. The operating report concerns the inputs mobilised, the outputs achieved and the results observed. It also consists of a few statistical indicators used Glossary of 300 concepts and technical terms to monitor important elements In the context (provided the information can be obtained without too much delay). The operating report is part of the monitoring information system. It is often produced by means of data base management software. The advantage of an operating report is that it allows for the continuous visualisation of the main parameters of public intervention as well as rapid feedback. Statistical t e am Observatoire The organisation or system used to regularly gather or disseminate statistical information on a given theme and over a long period (e. g. statistics on equal opportunities; environmental statistics in a nature reserve). Due to its permanence, a statistical team can produce chronological series, something which an evaluation that takes place in a limited timeframe cannot do. When they exist, statistical teams are interesting sources of secondan/ data for evaluations. Comparability Comparabilité Quality of an indicator which uses the same measurement unit to quantify the needs, objectives or effects of several different interventions. to that of similar interventions). Efforts made to Comparability is useful for establishing norms for judgement (e. g. the average cost of jobs created by the intervention can be favourably improve compared comparability involve the harmonisation of measurement units and result, initially, in the definition of standard indicators, i. e. indicators that can be used in several regions with the same definition for the same sector of intervention (e. g. number of SMEs assisted, defined and calculated in a comparable way). Secondly, comparability can be extended to key indicators, that is, indicators which can be used in several regions and sectors of intervention. Related term(s) Harmonisation Standard indicator Harmonisation Indicateur standard Criteria and indicators Field of intervention Champ d'intervention A set of interventions which are similar enough for their indicators to be harmonised and for comparisons to be made between different evaluations. For example, the same programme can predict outputs in the field of research and development, in that of transport infrastructure, in that of training, and so on. Within the framework of European cohesion policy, fields of intervention are grouped together into three main categories: basic infrastructure, productive environment and human resources. Related term(s): Secteur d'intervention Indicators Context indicator Indicateur de contexte Measurement of an economic, social or environmental variable concerning an entire region, sector or group in which public intervention takes place (e. g. per capita GDP, annual number of jobs created in the region). Context indicators may describe a basic situation before an intervention and a desired situation after intervention. They are generally quantified on the basis of data from statistics offices or statistical teams. They apply to an entire territory or group, unlike programme indicators which apply only to addressees actually affected by an intervention. Baseline Situation de base State of the economic, social or environmental context, at a given time (generally at the beginning of the intervention), and from which changes will be measured. The basic situation is described by context indicators which describe the economy, socio-economic environment, concerned groups, etc. Related term(s): Point zéro Situation de référence P r o g r a m me indicator Indicateur de programme Indicator which concerns the inputs and outputs of the intervention as well as the results and impacts on its direct and indirect addressees. Programme indicators may include derived indicators measuring efficiency, effectiveness or performance. They are quantified by monitoring information systems and also by evaluation when it produces primary data. When they concern effects, programme indicators measure only those which affect direct and indirect addressees. Glossary of 300 concepts and technical terms Generic indicator Indicateur générique Indicator which uses the same measurement unit to quantify the impacts resulting from several outputs of various kinds (e. g. value-added generated information in the domains of fishing, tourism and new by outputs technologies). Key indicator Indicateur clé Indicator likely to play an important part in comparisons between different interventions and in the synthesis of conclusions of several evaluations. Within the framework of European cohesion policy, key indicators are, for example, the budgetary absorption rate or number of SMEs assisted (for the monitoring of implementation), or the cost per job created or per capita GDP (for the evaluation of impacts). Key indicators must be both standard and generic indicators. Absorption rate Taux de consommation budgétaire Budgetan/ inputs mobilised in proportion to the inputs initially allocated. Related term(s): Taux d'absorption Completion rate Taux de réalisation Percentage of initially planned outputs and which have been completed. The completion rate of a major infrastructure project is calculated in terms of the stages of work which were initially planned and were actually completed. The completion rate of an intervention is the average of the completion rates of the projects comprising that intervention. The completion rate is a key indicator for monitoring the implementation of an intervention. Related term(s): Taux d'avancement Criteria and indicators C o v e r a ge rate Taux d'atteinte du public eligible Percentage of the eligible group which was actually affected by an intervention. The coverage rate is a result indicator which is important to quantify for monitoring purposes. The exposure rate will be referred to indicate the portion of the population targeted, which received information on the programme. Related term(s): Exposure rate Taux de couverture Taux d'exposition ν Impacts The following pages list terms relating to the impacts of public interventions and, more particularly, those which contribute towards socio-economic development. The vocabulary, partly borrowed from economics, generally uses the term "effect" rather than "impact". Since the two terms are very similar, the original economic vocabulary has been retained. For a complete presentation of these subjects, the reader is referred to the third part of Volume 1 of the Collection, devoted to socio-economic impacts at the micro and macro levels, the third part of Volume 4 devoted to effects of synergy, and Volume 5 which covers impacts on employment, the environment and other Community priorities. Causality analysis Gross effect Effet brut Change observed following a public intervention, or an effect reported by the direct addressees. A gross effect appears to be the consequence of an intervention but usually it cannot be entirely imputed to it. The following example shows that it is not sufficient for an evaluation merely to describe gross effects: Assisted firms claimed to have created 500 jobs owing to the support (gross effect). In reality, they would in any case have created 100 jobs even without the support (deadweight). Thus, only 400 jobs are really imputable to the intervention (net effect). Net effect Effet net Effect really imputable to the public intervention and to it alone, as opposed to apparent changes or gross effects. To evaluate net effects, based on gross effects, it is necessary to subtract the changes which would have occurred in the absence of the public intervention, and which are therefore not imputable to it since they are produced by confounding factors (counterfactual situation). For example, the number of employees in assisted firms appears to be stable (change or gross effect equal to zero). However, it is estimated that without support there would have been 400 redundancies (counterfactual situation). Thus, 400 jobs were maintained (net effect). Related term(s): Additional Effect Effef additionnel Effet propre D e a d w e i g ht the public Change observed among direct addressees intervention, or reported by direct addressees as a consequence of the public intervention, that would have occurred, even without the intervention. following Effet d'aubaine Glossary of 300 concepts and technical terms For example: a farmer received assistance for the building of a self-catering cottage. In the survey he stated that the support had enabled him to create better quality facilities, but that he would have built the cottage, even without support. Thus, there is deadweight since the the construction of the cottage cannot be imputed entirely to the intervention. Deadweight can account for as much as 50% of the gross effect. The estimation of deadweight necessitates a survey of direct addressees, preferably with a comparative analysis of non-participants. Related term(s): Perte sèche Poids mort D i s p l a c e m e nt effect Effet de déplacement Effect obtained in an eligible area at the expense of another area. Displacement effects may be intended (e. g. displacement of a public administration from the capital to a region undergoing redeployment) or unintended (e. g. 10% of the jobs created by a regional development programme resulted in the disappearance of jobs in other eligible regions; a firm used programme assistance to move its premises from the centre to the outskirts of a town). When they are not intended, displacement effects must be subtracted from gross effects to obtain net effects. Substitution effect Effet de substitution Effect obtained in favour of a direct addressee but at the expense of a person or organisation that does not qualify for the intervention. For example, a person unemployed for a long time found a job owing to the intervention. In reality, this job was obtained because someone else was granted early retirement. If the objective was the redistribution of jobs in favour of disadvantaged groups, the effect can be considered positive. An evaluation determines, with regard to the objectives of the intervention, whether the substitution effect can be considered beneficial or not. When it is not beneficial, the substitution effect must be subtracted from gross effects. Impacts Counterfactual situation Situation contrefactuelle A situation which would have occurred in the absence of a public intervention. For example, a firm was assisted so that its employees could be retrained in new technologies. No redundancies were recorded in the following two years. It is estimated that without the assistance (counterfactual situation) 50 jobs would have been lost. By comparing the counterfactual and real situations, it is possible to determine the net effects of the public intervention. Various tools can be used for the construction of the counterfactual situation: shift-share analysis, comparison groups, simulation using macro-economic models, etc. At the baseline, the real situation and the counterfactual situation are identical. If the intervention is effective , they diverge while it is underway. Related term(s): Policy-off situation Scénario hors intervention Confounding factor Facteur concurrent Factors independent of a public intervention which are partly or entirely the cause of changes observed among addressees (gross effects). A confounding factor can hinder or amplify an expected impact. For example, employment increased by 10% in a group of assisted firms. However, this increase could not be imputed entirely to the public intervention because the assisted firms also benefited from a favourable macro-economic context (confounding factor). In another example, close to 80% of the trainees found a job after one year. However, this high placement rate was partly imputable to the initial qualifications of the trainees which were very high at the time of their recruitment (skimming-off effect). When a causality analysis is carried out as part of an evaluation, it distinguishes between effects imputable to the public intervention (net effects) and those imputable to confounding factors. Glossary of 300 concepts and technical terms Related term(s): Concurrent factor Extraneous factor Exogenous factor Co-facteur Facteur exogène Facteur contextuel Causality analysis Analyse de causalité The study of relations of cause and effect which link a public intervention to its impacts. Causality analysis may be inductive. In this case, it investigates the mechanisms likely to produce impacts, as well as confounding factors likely to have an influence. Causality analysis may also be deductive (or hypothetico-deductive). In this case, it examines whether assumptions about impacts are not contradicted by the facts. It may also supply a quantitative estimation of impacts. Related term(s): Deductive analysis Inductive analysis Analyse deductive Analyse inductive T h e o ry of action Théorie d'action All the hypotheses used by funding authorities and managers to explain how a public intervention is going to produce its impacts and achieve its aim. The theory of action consists of relations of cause and effect linking outputs, results and impacts. It is often implicit, or at least partly so. Evaluation helps to clarify the theory and for that purpose relies on a logical diagram of expected impacts. Impacts Related term(s): Programme theory Referentiel Modèle de causalité Explanatory theory All the assumptions likely to explain changes observed following the public intervention (gross effects). Théorie explicative likely to explain gross effects, The scope of explanatory theory is far wider than that of the theory of action. Like the theory of action, it encompasses relations of cause and effect between outputs, results and impacts. It also covers any other causes factors. Evaluation relies on a list of explanatory assumptions established with the help of experts, based on research and evaluation in similar fields. Inductive analysis the construction or improvement of explanatory theory. techniques also contribute i. e. all confounding towards Extrapolation coefficient Coefficient d'extrapolation Ratio used to estimate an impact through the use of monitoring data. For example, if the amount of investments by assisted SMEs is known, an extrapolation coefficient can be used to estimate the annual value added generated. In this case, the extrapolation coefficient may be the ratio of annual value added per euro invested. This ratio will be drawn from relevant regional or sectoral statistics. If the evaluation team uses existing extrapolation coefficients, it is important for it to explain and justify its choices. A coefficient can be deduced from a questionnaire survey (provided the sample is representative) conducted as part of the same evaluation or a preceding one. A coefficient may also be taken from an evaluation or research on similar interventions, provided the conclusions can be generalised (external validity). A coefficient can be established for the needs of an evaluation by an expert panel or by a network of experts, using the Delphi panel technique. Socio-economic effects Direct effect Effet direct U Effects of a public intervention on its direct addressees, excluding all repercussions on other groups. For example: investment support has direct effects on assisted businesses (production costs and capacity; creation or maintenance of jobs). New infrastructure has a direct effect on the people and enterprises which use it. Indirect effect Effet indirect I Effect which spreads throughout the economy, society or environment, beyond the direct addressees of the public intervention. Indirect "internal" effects, which are spread through market-based relations (e. g. effect on suppliers or on the employees of an assisted firm), are distinguished from external effects or "externalities" which are spread through non-market mechanisms (e. g. noise pollution; cross-fertilisation within an innovation network). Related term(s): Effet induit First round effect Effet primaire Effect of a public intervention on direct addressees and on the inner circle of indirect addressees. For example, the impact on assisted firms and their customers, suppliers and employees. First round effects may be analysed by means of evaluation surveys. They are, by nature, micro-economic, unlike secondary effects which concern circles of increasingly indirect addressees. Glossary of 300 concepts and technical terms Related term(s): Knock-on effect Impact de premier ordre S e c o n d a ry effect Effet secondaire All the impacts produced by an intervention, over and above its first round effects, that is, apart from direct addressees and the first circle of indirect addressees. Potentially, the mechanisms of secondary effects stretch in concentric circles throughout the entire economy. They are generated by supplier effects, income multiplier effects, impulsion effects, etc. The evaluation of secondary effects is based on macro-economic reasoning and techniques. Externality Externante Effect of a private action or public intervention which is spread outside the market. For example: a firm pollutes a river and causes an economic loss for a fish farm downstream; an engineer leaves the firm in which he or she was trained and applies his or her know-how in a new firm which he or she creates. By their very nature, externalities trigger private choices which cannot be optimised through the mechanisms of market competition. Only collective and often public decisions are able to promote positive external effects and prevent negative ones. A large proportion of financial support allocated within the framework of European cohesion policy is aimed at promoting positive external effects which businesses do not seek to create themselves spontaneously. L e v e r a ge effect Effet de levier Propensity for public intervention to induce private spending among direct addressees. In cases where public intervention subsidises private investments, leverage effects are proportional to the amount of private spending induced by the Impacts subsidy. Leverage effects must not be confused with additional effects (see net effect). Nor do they refer to the principle of additionality which applies to European social and economic cohesion policy (see additionality). D e m a n d - s i de effect Effet de demande Effect which spreads through growing intermediate consumption of enterprises (supplier effect) and through the income distributed within the assisted region, and which, in turn, generates spending by households (multiplier effect). When public financing ceases, demand side effects disappear. Supplier effect Effet fournisseur Secondary effect on companies supplying goods and services businesses which are the direct addressees of a public intervention. to There may be several cycles of supplier effects, if the analysis is extended to the suppliers of suppliers, etc. Input-output analysis is used to estimate all supplier effects in a given territory. Income multiplier effect Effet multiplicateur de revenu Secondary effect resulting from generated by the public intervention. increased income and consumption Multiplier effects are cumulative and take into account the fact that part of the income generated is spent again and generates other income, and so on in several successive cycles. In each cycle, the multiplier effect diminishes due to purchases outside the territory. The effect decreases much faster when the territory is small and when its economy is open. Supply-side effect Efref d'offre Secondary effect which spreads through the increased competitiveness of businesses and thus of their production. Glossary of 300 concepts and technical terms The main mechanisms at play are increased productive capacity, increased productivity, reduced costs, and the diversification and reinforcement of other factors of competitiveness such as human capital, public facilities, the quality of public services, innovation networks, etc. Structuring effect Effet structurant Sustainable changes in socio­economic structures following a public intervention. Structuring effects are changes which last after the public spending has ceased. They include sustainable effects at the micro­economic level and supply­side effects, but not demand­side effects. Structuring effects must not be confused with structural adjustment, which strives for the convergence of the macro­economic variables of a country towards international standards, particularly in terms of public finances and inflation. Related term(s): Lasting effect Effet structurel ■ Impulsion effect Effet d'entraînement ■ Secondary effect which spreads through investments induced upstream and downstream from the sector affected by the intervention. For example, the construction of a large infrastructure project generates the creation of new businesses in the region. These continue to expand after the work has ceased. ■ N e t w o rk effect Effet de réseau Secondary effect which spreads through special and lasting relations between businesses in the same territory or branch, and between those same businesses and public or para­public organisations such as research centres, universities, etc. Network effects are partially based on non­market interpersonal relations. Thus, they are in part "externalities". Effects of synergy Internal coherence Cohérence interne Correspondence between the different objectives of the same intervention. Internal coherence implies that there is a hierarchy of objectives, with those at the bottom logically contributing towards those above. Related term(s): Consistency External coherence Cohérence externe Correspondence between the objectives of an intervention and those of other public interventions which interact with it. If a national policy and a European Union socio-economic programme are implemented in a complementary manner in the same territory for the purpose of developing SMEs, it can be said that there is external coherence. Complementarity Complémentarité The fact that several public interventions (or several components of an intervention) contribute towards the achievement of the same objective. Complementarity may be functional, if the objective is functional (e. g. developing a sector of activity, creating a network). It may be territorial, if the objective concerns a territory (e. g. integrated local development policy). Synergy Synergie The fact that several public interventions (or several components of an intervention) together produce an impact which is greater than the sum of the impacts they would produce alone (e. g. an intervention which finances Glossary of 300 concepts and technical terms the extension of an airport which, in turn, helps to fill tourist facilities, also financed by the intervention). Synergy generally refers to positive impacts. However, phenomena which reinforce negative effects, negative synergy or anti-synergy may also be referred to (e. g. an intervention subsidises the diversification of enterprises while a regional policy helps to strengthen the dominant activity). Employment effects Employment effect Improvement structural modification of the labour market, following an intervention. in employability, creation and maintenance of jobs, or Effet d'emploi Job created or maintained Emploi créé ou maintenu New jobs created owing to the public intervention, or threatened jobs which the public intervention helped to safeguard. Employability Employabilité An individual's ability to find or keep a job in a given socio-economic environment. Employability concerns the appropriateness of skills in relation to the requirements of the labour market, so that the individual concerned can keep his or her job or find a (new) job in reasonable conditions in a reasonable length of time. A public intervention in favour of employability concerns human resource development and particularly training. The employability of an individual can be examined indirectly on the basis of pre-established factors (e. g. qualifications, experience, mobility, existence of job offers). Transition rate Taux de transition Percentage of addressees whose social integration has improved, following the intervention. The improvement may correspond to the people's recruitment (placement rate) or to the continuation of training on a higher level. The transition rate is evaluated after a period of months (for example, 12 months after the end of a training course). Glossary of 300 concepts and technical terms Related term(s): Placement rate Taux de placement Skimming-off effect Consequence of the recruitment of participants for a training course, which favours those most likely to obtain the best results. Effet d'écrémage is an unintended effect which This the measurement of performance rather than the satisfaction of the needs of those who are most disadvantaged. favours P e r m a n e nt job Emploi permanent A job generated by an intervention, which lasts after public support has ceased. By contrast, a temporary job lasts only until the end of an intervention. The creation of permanent jobs is a sustainable effect of an intervention. Full-Time Equivalent (FTE) Equivalent temps plein (ETP) Conversion of part-time jobs into quantities corresponding to full-time jobs. For example, an intervention to support exports enables firms to find new markets and to create 8 full-time and 22 part-time jobs. The impact in terms of full-time equivalents is 19. Economie and social development Economic a nd social cohesion Cohésion économique et sociale The fact that there are unacceptable gaps between regions, territorial categories (e. g. urban / rural) or social groups, from the point of view of their level of economic development and social integration. Economic and social cohesion relates primarily to disparities in regional income and quality of life, as well as to all differences between social groups' access to employment. It is the main objective of the European policy of the same name (see Article 130 A of the Treaty). Sustainable d e v e l o p m e nt Développement durable ■ Increase in economic activity which respects the environment and uses natural resources harmoniously so that future generations' capacity to meet their own needs is not compromised. By contrast, unsustainable development is characterised by the destruction of natural resources. This has negative repercussions on long-term development potential. Related term(s): Développement viable ■ Mitigation Action aimed at reducing the potential negative effects of a public intervention on the environment. Atténuation Endogenous d e v e l o p m e nt Développement endogène Increase in economic activity based on internal competitive advantages within a region or territory. Glossary of 300 concepts and technical terms The main factors of endogenous development are human capital, enterpreneurial spirit, local savings and local innovation networks. By contrast, exogenous development favours the inward transfer of capital, technology, know-how and skills. Competitiveness Compétitivité The capacity of a firm, in a competitive socio-economic environment, to defend or increase its market share over the medium and longer term, and to generate wealth. The notion of competitiveness can apply to a single firm or to all the businesses in a sector or region. It is directly related to the notion of competitive advantage, an advantage which an enterprise, region or sector must possess or develop if it is to be competitive in a segment of a particular market. SWOT analysis is used to identify existing advantages (called "strengths" or success factors) and disadvantages that need to be reduced (called "weaknesses"). Diversification Diversification Deployment of the production of a firm or a region beyond its traditional specialisation, with a view to creating new activities and/or setting up in new market segments. Policies which create incentives and support for diversification are intended for all enterprises established in market segments which are threatened or in crisis. Equal opportunities Egalité des chances Equal access for women and men to employment, at the same level of remuneration and social advantages, in a given socio-economic context. This impact relates to the principle of equal rights and equal treatment of women and men. It means, first, that everybody is free to develop their Impacts personal aptitudes and to make choices without being limited by stereotyped gender roles and, secondly, that particular differences in behaviour, aspirations and needs, between women and men, are not to be valued too highly nor considered too critically. The principle of equal opportunities may for discrimination. The evaluation of impacts on equal opportunities includes the mainstreaming of gender at all stages. require unequal to compensate treatment M a i n s t r e a m i ng Intégration Systematically taking into account the specific priorities and needs of women and men in all dimensions of an intervention, from the design and implementation stage to monitoring and evaluation. For example, equal opportunities is mainstreamed in evaluation in so far as evaluation systematically distinguishes men and women among the direct and indirect addressees of a public intervention, and among operators and decision-makers, when this distinction concerns needs, results and impacts. By extension, the principle of integration may also apply to other impacts considered to be priorities. Related term(s): Intégration stratégique Environmental I m p a ct A s s e s s m e nt (EIA) Evaluation d'Impact sur l'Environnement (EIE) Study of all the repercussions of an individual project on the natural environment. EIA is a compulsory step in certain countries in the selection of major infrastructure projects. By contrast, strategic environmental assessment (SEA) refers to the evaluation of programmes and policy priorities. EIA consists of two steps: screening, which refers to an initial overall analysis to determine the degree of environmental evaluation required before the implementation is approved; and scoping which determines which impacts must be evaluated in depth. The evaluation of environmental impacts examines expected and unexpected effects. The latter are often more numerous. VI Techniques lists the main This part the design and implementation of an evaluation. For a complete presentation of the main evaluation techniques, the reader is referred to Volume 3 of the Collection, which includes 23 detailed information sheets. terms used technical in Evaluation methods M e t h od Méthode Complete plan of evaluation team's work. A method is an ad hoc procedure, specially constructed in a given context to answer one or more evaluative questions. Some evaluation methods are of low technical complexity, while others include the use of several tools. Evaluation design Protocole d'évaluation Technical part of the evaluation plan, designed for data collection and analysis. M e t h o d o l o gy Méthodologie Strictly speaking, this is the science of the construction of evaluation methods. The technical design of an evaluation is, however, a matter of know-how and expertise rather than a purely scientific procedure. In practice, the term methodology is often used in the sense of a method, as defined above. Tool Outil Standardised procedure used to regression analysis or questionnaire survey). fulfil a function of evaluation (e. g. Evaluation tools serve to collect quantitative or qualitative data, synthesise judgement criteria, explain objectives, estimate impacts, and so on. A tool has a specific name, is described in a manual, and can be taught. Its use may be free or may involve payment of a fee if the tool is protected by a registered trademark. An evaluation may use one or more tools. Glossary of 300 concepts and technical terms Related term(s): Technique Technique Structuring techniques Strengths, W e a k n e s s e s, Opportunities, Threats (SWOT) Forces, faiblesses, opportunités, menaces Analysis of internal factors which can be relied on (strengths) or which need to be compensated for (weaknesses), as well as external factors which are favourable (opportunities) or unfavourable (threats). This is an evaluation tool which is used to check whether a public intervention is suited to its context. The tool structures debate on strategic orientations. Logical f r a m e w o rk Cadre logique Tool used to structure the logic of a public intervention. It is based on a matrix presentation of the intervention, which highlights its outputs, results, and specific and global impacts. Each level of objective is associated with one or more verifiable indicators of success, as well as with influencing success or failure (confounding the conditions and risks factors). This tool is frequently used for the evaluation of development support projects. By contrast, it is ill-suited to the evaluation of socio­ economic programmes because its level of simplification is too high. Objective tree Arbre des objectifs Hierarchical classification of the objectives of a public intervention, linking each specific objective to its global objective. The objectives tree of a programme makes the overall logic explicit. It is used for clarification. Concept mapping of impacts Cartographie conceptuelle des impacts Tool used for the clarification of explicit and implicit objectives, based on the identification, grouping together and grading of expected impacts. Glossary of 300 concepts and technical terms The concept mapping of impacts is implemented in a pluralistic and participatory way, so that a large number of participants can be involved. It may result in the selection of indicators that are associated with the main expected impacts. Colour Vote Vote coloré Technique used to run meetings, based on a visual presentation of opinions in a group. In an evaluation situation in which several stakeholders are involved, it can be used to structure discussion and collective reflection by facilitating the expression of all points of view. It is therefore a tool likely to enhance the impartiality of the evaluation in the context of a pluralistic approach. It may be used to clarify and grade evaluative questions, to choose evaluation criteria, to validate conclusions, and to formulate recommendations. Some variations of this tool require computer support. Related term(s) Abaque de Régnier Matrix of cross impacts Matrice des impacts croisés Tool used to highlight potential complementarity, duplication and conflict between the impacts of the different components of an intervention, or between the impacts of different interventions. For example, the tool presents the different measures of a programme in lines and columns. In each cell a score between -3 and +3 is given to describe the potential synergy between a measure situated on a line and a measure situated in a column. Information gathering techniques Primary data Données primaires Data collected directly in the field, by means of a survey carried out by the evaluation team on the groups concerned by the intervention. Primary data play an important role in the cognitive contribution of the evaluation. They are added to data already available at the start of the evaluation (e. g. former research and evaluations, monitoring data, statistics). Primary data are particularly useful for analysing impacts and needs in depth. S e c o n d a ry data Données secondaires Existing information, gathered and interpreted by the evaluation team. Secondary data consists of information drawn from the monitoring system, produced by statistics institutes and provided by former research and evaluations. Cross sectional data Données par coupe transversale Data collected in a comparative way, at a given time, on several categories of individuals or facts. For example: comparative unemployment rates in European regions; comparative placement rates for women and men. ■ T i me series Série temporelle Data collected on the same population, in a comparative way, at regular intervals during a given period. Overall variations in the characteristics of a given population are observed over time. Statistics institutes and statistical teams are the main sources of time series. Glossary of 300 concepts and technical terms Related term(s): Série chronologique Longitudinal data Données longitudinales Time series relating to repeated comparative observations of the same individuals, at regular intervals, during a given period. In the evaluation context, these data are obtained by observing the evolution of a sample of addressees. Related term(s): Données de cohorte S a m p le Echantillon Part of a group selected for a survey. In its statistical sense, the word "population" applies to all the people, projects or objects likely to be observed, e. g. : all the assisted firms or individuals; all the firms in a region; or an entire targeted group. If the sample is representative, the survey results can be generalised to the entire surveyed population (internal validity) or to similar populations (external validity). A sample is representative if it has the same distribution of characteristics as the population from which it is taken, and if it is large enough (generally at least a thousand individuals). Questionnaire survey Enquête par questionnaire A set of structured questions addressed to a sample of people or, exhaustively, to a group of people. A questionnaire consists of a list of questions of which the sequence is determined in advance. Depending on whether the questions are open or closed, the interviewee is either free to formulate his or her answers as he or she wishes, or is given predefined statements (descriptors) from which Techniques to choose. A questionnaire can be undertaken by post, telephone, e-mail or face-to-face interview. If the questionnaire survey is exhaustive or if it concerns a representative sample, and provided that most of the questions are closed, it may be used for gathering quantitative data and for informing indicators. Individual interview Entretien individuel Technique used to collect qualitative data and the opinions of people who are concerned or potentially concerned by the intervention, its context, its implementation and its effects. Several types of individual interview exist, including informal conversations, semi-structured interviews and structured interviews. The latter is the most rigid approach and resembles a questionnaire survey. A semi-structured interview consists of eliciting a person's reactions to predetermined elements, without hindering his or her freedom to interpret and reformulate these elements. Focus group Entretien de groupe Survey technique based on a small group discussion. Often used to enable participants to form an opinion on a subject with which they are not familiar. The technique makes use of the participants' interaction and creativity to enhance and consolidate the information collected. It is especially useful for analysing themes or domains which give rise to differences of opinion that have to be reconciled, or which concern complex questions that have to be explored in depth. C a se study Etude de cas In-depth study of data on a specific case (e. g. a project, addressee, town). The case study is an appropriate tool for the inductive analysis of impacts and particularly for innovative interventions for which there is no explanatory theory of impacts. A case study is concluded with a monograph presented Glossary of 300 concepts and technical terms in a narrative form. A series of case studies can be carried out concurrently, in a cumulative or iterative way. The latter consists of conducting several series of case studies, carefully selected to verify an assumption. Ethnographic observation Observation ethnographique In situ, non-disruptive observation of the daily activity of actors and/or addressees of the evaluated intervention. tries to understand the situation "from the The researcher inside". Ethnographic observation is useful in little known situations or when access to the field is difficult. It is used to collect very detailed information, with audio-visual recordings where relevant, on a few typical situations of implementation of the intervention. It also serves to identify all the effects of the intervention and the influence of the context. Analytical techniques C o m p a r i s on group Groupe de comparaison Group of people or organisations which has not participated in a public intervention and which is compared with a group of participants for the purpose of analysing the net effects of an intervention. If the two groups are randomly selected samples, the comparison group is called the control group because it is comparable, in every way, to the so­ called "experimental" group. Irrespective of the mode of selection of the comparison group, its characteristics must be as similar as possible to those of the direct addressees. For example, the evaluation must make a "before­after" comparison between a group of direct addressees and another group constituted by means of the matching pair technique. Related term(s): Analyse comparative ■ Control group Groupe témoin ti Comparison group consisting of eligible people or organisations which have been excluded from all participation in the intervention by a process of random selection. Apart from its non­participation in the intervention, the control group is, from every point of view, comparable to the group of participants. It has also been exposed to the same variations in the socio­economic context (confounding factors). When a group of participants and a control group are compared, the influence of confounding factors is the same on both sides (provided the two groups are large enough). Related term(s): Affectation aléatoire Groupe de contrôle Glossary of 300 concepts and technical terms M a t c h i ng pair Binome apparié Technique for constructing a comparison group. For each individual in the sample of direct addressees being surveyed, an individual is chosen who resembles it as closely as possible and who has not participated in the intervention. Regression analysis Analyse de régression Statistical tool used to make a quantitative estimation of the influence of several explanatory variables (public intervention and confounding factors) on an explained variable (an impact). Regression analysis is a tool for analysing deductive causality. It is based on an explanatory logical model and on a series of preliminary observations. The tool can be used in varying ways, depending on whether the variables of the model are continuous or discrete and on whether their relations are linear or not. Covariation Covariance The fact that two variables systematically evolve in the same direction or in opposite directions. If there is little covariance, there is no statistically reliable relationship between them. A large degree of covariance between A and Β indicates an assumption of causality but does not prove it. (Is A the cause of B? or is Β the cause of A? or are A and Β the consequence of something else?). Related term(s): Corrélation Factor analysis Statistical analysis tool used to identify all correlation within a large quantity of data concerning many individuals (e. g. survey results), and to represent it in a simplified way in the form of a two- or three-dimensional space. Analyse factorielle Techniques The tool reveals groupings and suggests relations of cause and effect. It is an inductive causality analysis tool. Shift-share analysis Analyse structurelle-résiduelle Tool for evaluating regional policy, which estimates the counterfactual situation by projecting national economic trends onto the economy of a given region. The basic assumption of this technique is that, in the absence of regional policy, the evolution of economic variables in the region would have been similar to that of the country as a whole. Comparison between the policy- off and policy-on situations is concluded with an estimation of the macro- economic impact of regional policy. The optimum conditions for using this tool rarely exist. Input-output analysis Analyse Entrées-Sorties Tool which represents the interaction between sectors of a national or regional economy in the form of intermediate or final consumption. Input-output analysis serves to estimate the repercussions of a direct effect in the form of first round and then secondary effects throughout the economy. The tool can be used when a table of inputs and outputs is available. This is usually the case at the national level but more rarely so at the regional level. The tool is capable of estimating demand-side effects but not supply-side effects. M a c ro e c o n o m ic model Modèle macro-économique Tool used to simulate the main mechanisms of a regional, national or international economic system. A large number of models exist, based on widely diverse macro-economic theories. This type of tool is often used to simulate future trends, but it may also serve as an evaluation tool. In this case, it is used to simulate a counterfactual situation, and thus to quantitatively evaluate net effects on Glossary of 300 concepts and technical terms most of the macro-economic variables influenced by public actions, i. e. : growth, employment, investment, savings, etc. The models are generally capable of estimating demand-side effects far more easily than supply-side effects. Delphi panel Enquête Delphi Procedure for iterative and anonymous consultation of several experts, aimed at directing their opinions towards a common conclusion. The Delphi panel technique may be used in ex ante evaluation, for estimating the potential impacts of an intervention, for example. Judgement techniques Expert panel Panel d'experts Work group which is specially formed for the purposes of the evaluation and which meets several times. The experts are recognised independent specialists in the evaluated field of intervention. They collectively pronounce a judgement on the value of the public intervention and its effects. An expert panel serves to rapidly and inexpensively formulate a synthetic judgement which integrates the main information available on the programme, as well as information from other experiences. Related term(s): Peer review Evaluation par les pairs B e n c h m a r k i ng Etalonnage Qualitative and quantitative comparison of the performance of an intervention, with that which is reputed to be the best in the same domain of intervention or in a related domain. Benchmarking is facilitated when, at the national or regional level, there are league tables of good and not so good practice. Related term(s): Benchmark Best practice League table Classement Etalon Meilleure pratique Glossary of 300 concepts and technical terms Multicriteria analysis Analyse multicritère Tool used to compare several interventions in relation to several criteria. Multicriteria analysis is used above all in the ex ante evaluation of major projects, for comparing variants. It can also be used in the ex post evaluation of an intervention, to compare the relative success of the different components of the intervention. Finally, it can be used to compare separate but similar interventions, for classification purposes. Multicriteria analysis may involve weighting, reflecting the relative importance attributed to each of the criteria. It may result in the formulation of a single judgement or synthetic classification, or in different classifications reflecting the stakeholders' different points of view. In the latter case, it is called multicriteria-multijudge analysis. Related term(s): Multicriteria-multijudge analysis Analyse multicritère-multijuge Cost-effectiveness analysis Analyse coût-efficacité Evaluation tool for making a judgement in terms of efficiency. This tool consists of relating the net effects of the intervention (which must be determined separately) to the financial inputs needed to produce those effects. The judgement criterion might, for example, be the cost per unit of impact produced (e. g. cost per job created). This unit cost is then compared to that of other interventions chosen as benchmarks. Cost-benefit analysis Analyse coût-avantages Evaluation tool for judging the advantages of the intervention from the point of view of all the groups concerned, and on the basis of a monetary value attributed to all the positive and negative consequences of the intervention (which must be estimated separately). Techniques When it is neither relevant nor possible to use market prices to estimate a gain or a loss, a fictive price can be set in various ways. The first consists of estimating the willingness of addressees to pay to obtain positive impacts or avoid negative impacts. The fictive price of goods or services can also be estimated by the loss of earnings in the absence of those goods or services (e. g. in cases of massive unemployment, the fictive price of a day's unskilled work is very low). Finally, the fictive price can be decided on directly by the administrative officials concerned or the steering group. Cost-benefit analysis is used mainly for the ex ante evaluation of large projects. VII Appendix Bibliography GAO (1991), Designing Evaluations, Washington : United States General Accounting Office, Program Evaluation and Methodology Division. Guba E. et Lincoln Y. (1989), Fourth Generation Evaluation', Newbury Park: Sage Publications. Hadji C. (1990), L'évaluation, règles du jeu, Paris : ESF Editeur (collection Pédagogies). Monnier E. (1992), Evaluation de l'action des pouvoirs publics, Paris : Economica. OECD (1986) 'Glossaire sur l'évaluation des politiques internationales de développement de l'OCDE'; in : Methods and Procedures in Aid Evaluation : A Compendium of Donor Practice and Experience, Paris: OECD. Patton M. Q. (1987), Utilization Focused Evaluation', Newbury Park : Sage Publications. Rossi P. H. et Freeman H. E. (1993), Evaluation - A Systematic Approach, Newbury Park : Sage Publications. Scriven M. (1991), Evaluations Thesaurus, Newbury Park : Sage Publications. Sohm E. D. (1978), Glossary of Evaluation Terms, Genève : UNO. Vedung E. (1997), Public Policy and Program Evaluation, New Brunswick : Transaction Publishers. Index Alphabétique Français French Alphabetical Index French alphabetical index Abaque de Régnier Voir Vote coloré Action Voir Mesure Activité Voir Ressource Additionnante 30 Adéquation Voir Pertinence Affectation aléatoire Voir Groupe Analyse comparative témoin Voir Groupe de comparaison Analyse coût-avantages 112 Analyse coût-efficacité 112 Analyse de causalité 82 Analyse de régression 108 Analyse deductive Voir Analyse de causalité Analyse Entrées-Sorties 109 Analyse factorielle 108 Analyse inductive Voir Analyse de causalité Analyse multicritère 112 Analyse multicritère-multijuge Voir Analyse multicritère Analyse structurelle-résiduelle 109 Apport conceptuel Voir Finalité cognitive Appréciation Voir Evaluation Appréciation ex ante Voir Evaluation ex ante Apprentissage en double boucle Voir Apprentissage organisationnel Apprentissage en simple boucle Voir Apprentissage organisationnel Apprentissage organisationnel 22 Arbre des objectifs 101 Atténuation 93 Audit Voir Contrôle Audit de performance Voir Suivi Auto-évaluation 41 Axe Voir Programme B Bénéficiaire Voir Destinataire direct Bénéficiaire final Voir Opérateur Besoin 53 Bien-fondé Voir Raison d'être Binôme apparié 108 But Voir Objectif Cadre logique 101 Cahier des charges 46 Cartographie conceptuelle des impacts 101 Champ d'évaluation 33 Champ d'intervention 71 Chargé d'évaluation Voir Equipe d'évaluation Classement Voir Etalonnage Coefficient d'extrapolation 83 Co-facteur Voir Facteur concurrent Cohérence externe 89 Cohérence interne 89 Cohésion économique et sociale 93 Commanditaire 41 Comparabillté 70 Compétitivité 94 Complémentarité 89 Condition-cadre Voir Contexte Contexte 53 Contrôle 29 Contrôle de gestion Voir Suivi Corrélation Voir Covariance Coût unitaire Voir Efficience Coût-efficacité Voir Efficience Covariance 108 Crédibilité 47 Critère 61 Cycle de programmation Voir Programme Glossary of 300 concepts and technical terms Décideur Voir Financeur Descripteur 68 Destinataire direct 39 Destinataire indirect 40 Développement durable 93 Développement endogène 93 Développement viable Voir Développement durable Diagnostic Voir Evaluation ex ante Disparité 63 Dispositif d'évaluation 43 Diversification 94 Donnée qualitative Voir Descripteur Donnée quantitative Voir Indicateur Données de cohorte Voir Données longitudinales Données longitudinales 104 Données par coupe transversale 103 Données primaires 103 Données secondaires 103 Durabilità 64 indirect inattendu Effet fournisseur 87 Effet inattendu 58 Effet indirect 85 Effet induit Voir Effet Effet multiplicateur de revenu 87 Effet net 79 Effet pervers Voir Effet Effet primaire 85 Effet propre Voir Effet net Effet secondaire 86 Effet structurant 88 Effet structurel Voir Effet structurant Efficacité 63 Efficience 64 Egalité des chances 94 Eligibilité 63 Emploi créé ou maintenu 91 Emploi permanent 92 Employabilité 91 Enoncé Voir Descripteur Enquête Delphi 110 Enquête par questionnaire 104 Entretien de groupe 105 Entretien individuel 105 Environnement socio-économique Echantillon 104 Effectivité Voir Efficacité Effet 56 Effet additionnel Voir Effet net Effet brut 79 Effet d'aubaine 79 Effet de demande 87 Effet de déplacement 80 Effet de levier 86 Effet de réseau 88 Effet de substitution 80 Effet d'écrémage 92 Effet d'emploi 91 Effet d'entraînement 88 Effet direct 85 Effet d'offre 87 Voir Contexte Equipe d'évaluation 42 Equivalent temps plein 92 Etalon Voir Etalonnage Etalonnage 111 Etude d'évaluabllité 46 Etude de cas 105 Evaluanda Voir Champ Evaluateur Voir Equipe d'évaluation Evaluation 17 Evaluation à mi-parcours 27 Evaluation a posteriori Voir Evaluation ex post Evaluation a priori Voir Evaluation ex ante Evaluation affranchie des objectifs Voir Utilité Evaluation approfondie 25 French alphabetical index Evaluation chemin-faisant 27 Evaluation démocratique 20 Evaluation d'ensemble 25 Evaluation d'Impact sur l'Environnement 95 Evaluation endoformative 17 Evaluation ex ante 26 Evaluation ex post 28 Evaluation externe 41 Evaluation formative Voir Evaluation endoformative Evaluation in itinere Voir Evaluation chemin-faisant Evaluation intermédiaire Voir Evaluation à mi-parcours Evaluation interne 41 Evaluation manageriale 18 Evaluation par les pairs Voir Panel d'experts Evaluation partenariale Voir Evaluation pluraliste Evaluation participative Voir Evaluation pluraliste Evaluation pluraliste 19 Evaluation récapitulative 18 Evaluation sommative Voir Evaluation récapitulative Evaluation thématique 25 Examen Voir Evaluation Externante 86 Extrant Voir Réalisation Facteur concurrent 81 Facteur contextuel Voir Facteur concurrent Facteur exogène Voir Facteur concurrent Feed-back Voir Rétroaction Fiabilité 47 Finalité Voir Objectif Finalité instrumentale 21 Finalité normative 21 Financeur 37 Focalisation Voir Evaluation approfondie Forces, faiblesses, opportunités, menaces 101 Gagnants et perdants Voir Public concerné Généralisabilité Voir Validité externe Gestionnaire 38 Grille d'observation Voir Descripteur Groupe de comparaison 107 Groupe de contrôle Voir Groupe témoin Groupe de pilotage 42 Groupe témoin 107 H Harmonisation Voir Comparabilité I Impact 58 Impact de premier ordre Voir Effet primaire Impact global Voir Impact Impact intermédiaire Voir Impact Impact spécifique Voir Impact Impartialité 47 Implementation Voir Mise en oeuvre Imputabilité Voir Rendre des comptes Indicateur 67 Indicateur clé 74 Indicateur de contexte 73 Indicateur de programme 73 Indicateur générique 74 Glossary of 300 concepts and technical terms Indicateur standard Objectif quantitatif Voir Comparabilité Inégalité Voir Disparité Instance d'évaluation 43 Intégration 95 Intégration stratégique Voir Intégration Intervention 35 Intrant Voir Ressource Voir Objectif verifiable Objectif spécifique Voir Objectif Objectif verifiable 55 Objectivité Voir Fiabilité Observation ethnographique 106 Observatoire 70 Opérateur 38 Outil 99 M Maître d'oeuvre Voir Gestionnaire Maître d'ouvrage Voir Financeur Mandant Voir Commanditaire Mandat 45 Matrice dimpacts croisés 102 Meilleure pratique Voir Etalonnage Mesure 34, 67 Méta-évaluation 28 Méthodes 99 Méthodologie 99 Mise en oeuvre 55 Modèle de causalité Voir Théorie d'action Modèle-macroéconomique 109 Monitorage Voir Suivi Moyen Voir Ressource N Neutralité Voir Norme 61 Notation 69 Impartialité Objectif 54 Objectif global Voir Objectif Objectif implicite Voir Objectif Objectif intermédiaire Voir Objectif Objectif opérationnel Voir Objectif Panel d'experts 111 Partenaire Voir Protagoniste Partie prenante Voir Protagoniste Pérennité Voir Durabilité Performance 65 Perte sèche Voir Effet d'aubaine Pertinence 62 Pilotage Voir Suivi Poids mort Voir Effet d'aubaine Point zéro Voir Situation de base Politique 33 Pondération 62 Préévaluation 45 Priorité politique 55 Produit Voir Réalisation Programme 33 Projet 34 Protagoniste 37 Protocole d'évaluation 99 Public cible Voir Public eligible Public concerné 40 Public eligible 40 Quantification Voir Unité de mesure Radioscopie Voir Evaluation d'ensemble French alphabetical index Raison d'être 62 Réalisation 57 Réalisation financière Voir Ressource Redevabillté Voir Rendre des comptes Referentiel Voir Théorie d'action Relevance 62 Rendre des comptes 22 Rendu-compte Voir Rendre des comptes Représentativité Voir Validité externe Ressource 56 Résultat 57 Résultat durable Voir Impact Retombée Voir Effet inattendu Rétroaction 21 Réussite Voir Efficacité Robustesse Voir Fiabilité Scénario hors intervention Voir Situation contrefactuelle Secteur d'intervention Voir Champ d'intervention Sensibilité Voir Indicateur Série chronologique Voir Série temporelle Série temporelle 103 Situation contrefactuelle 81 Situation de base 73 Situation de référence Voir Situation de base Stratégie 53 Subsidiante 66 Succès Voir Efficacité Suivi 29 SWOT 101 Synergie 89 Système d'évaluation 43 Système d'indicateurs Voir Système d'information de suivi Système dinformation de suivi 69 Tableau de bord 69 Taux de réalisation 74 Taux d'absorption Voir Taux de consommation budgétaire Taux d'avancement Voir Taux de réalisation Taux de couverture 75 Taux d'exposition Voir Taux d'atteinte du public eligible Taux de consommation budgétaire 74 Taux de couverture Voir Taux d'atteinte du public eligible Taux de placement 91 Taux de placement Voir Taux de transition Technique Voir Outil Théorie d'action 82 Théorie explicative 83 U Usager Voir Destinataire direct Utilisation cognitive 20 Utilité 65 Valeur ajoutée communautaire Voir Subsidiarité Valeur sociale Voir Critère Validité de construction 48 Validité externe 49 Validité interne 48 Viabilité Voir Durabilité Vote coloré 102 English Alphabetical Index English alphabetical index Absorption rate 74 Accountability 22 Action See Measure Activity See Input Additional Effect See Net effect Additionality 30 Aim See Objective Appraisal See Evaluation Appropriateness See Relevance Assessment See Evaluation Audit 29 Axis See Programme Β Baseline 73 Benchmarking 111 Case study 105 Causality analysis 82 Cognitive aim 20 Colour Vote 102 Commissionner 41 Community value added See Subsidiarity Comparability 70 Comparison group 107 Competitiveness 94 Complementarity 89 Completion rate 74 Concept mapping of impacts 101 Conceptual See Cognitive aim Concerned group 40 Concurrent factor factor See Confounding Confounding factor 81 Consistency See Internal coherence Construct validity 48 Context 53 Context indicator 73 Control See Audit Control group 107 Cost effectiveness See Efficiency Cost-effectiveness analysis 112 Counterfactual situation 81 Covariation 108 Coverage rate 75 Credibility 47 Criterion 61 Cross sectional data 103 D Deadweight 79 Decision-maker See Funding authority Deductive analysis See Causality analysis Delphi panel 110 Demand-side effect 87 Democratic evaluation 20 Descriptor 68 Direct addressee 39 Direct effect 85 Disparity 63 Displacement effect 80 Diversification 94 Double loop learning See Organisational learning 93 Economic and social cohesion Effect 56 Effectiveness 63 Efficacy See Effectiveness Efficiency 64 Eligibility 63 Eligible public 40 Employability 91 Glossary of 300 concepts and technical terms Employment effect 91 Endogenous development 93 End-user See Direct addressee Environmental Impact Assessment 95 Equal opportunities 94 Ethnographic observation 106 Evaluability assessment 46 Evaluand See Scope Evaluation 17 Evaluation Committee 43 Evaluation design 99 Evaluation setting 43 Evaluation system 43 Evaluation team 42 Evaluative question 46 Evaluator See Evaluation Ex ante evaluation 26 Ex post evaluation 28 Exogenous factor team See Confounding factor Expert panel 111 Explanatory theory 83 Exposure rate See Coverage rate External coherence 89 External evaluation 41 External validity 49 Externality 86 Extraneous factor See Confounding factor Extrapolation coefficient 83 Factor analysis 108 Feedback 21 Field of intervention 71 Final beneficiary See Operator Financial realisation See Input First round effect 85 Focus group 105 Formative evaluation 17 Framework condition See Context Full-Time Equivalent 92 Funding authority 37 Generic indicator 74 Global evaluation See Overall evaluation Global Impact See Impact Global objective See Objective Goal See Objective Goal free evaluation See Utility Gross effect 79 H Harmonisation See Comparability I Immediate outcome See Result Impact 58 Impartiality 47 Implementation 55 Implicit objective See Objective Impulsion effect 88 Income multiplier effect 87 In-depth evaluation 25 Indicator 67 Indirect addressee 40 Indirect effect 85 Individual interview 105 Inductive analysis See Causality analysis Inequity See Disparity In-house evaluation See Internal evaluation Input 56 Input-output analysis 109 Instrumental aim 21 Intermediate impact See Intermediate objective See Objective Internal coherence 89 Impact English alphabetical index Internal evaluation 41 Internal validity 48 intervention 35 Monitoring information system 69 Multicriteria analysis 112 Multicriteria-multijudge analysis See Multicriteria analysis Job created or maintained 91 N Key indicator 74 Knock-on effect See First round effect Lasting effect See Structuring effect League table See Benchmarking Learning 22 Leverage effect 86 Logical framework 101 Longitudinal data 104 M Macro-economic model 109 Mainstreaming 95 Management audit See Monitoring Manager 38 Managerial evaluation 18 Mandate 45 Matching pair 108 Matrix of cross impacts 102 Means See Input Measure 34 Measurement unit 67 Meta-evaluation 28 Method 99 Methodology 99 Mid-term evaluation 27 Mitigation 93 Monitoring 29 Need 53 Needs assessment See Ex-ante Evaluation Net effect 79 Network effect 88 Neutrality See Norm 61 Normative aim 21 Impartiality 0 Objective 54 Objective tree 101 Objectivity See Reliability Observation grid See Descriptor On-going evaluation 27 Operating report 69 Operational objective See Objective Operator 38 Outcome See Impact Output 57 Overall evaluation 25 Partner See Stakeholder Partnership evaluation See Pluralistic evaluation Peer review See Expert panel Performance 65 Performance audit See Monitoring Permanent job 92 Pertinence 62 Placement rate See Transition rate Pluralistic evaluation 19 Glossary of 300 concepts and technical terms Policy 33 Policy priority 55 Policy-off situation See Counterfactual situation Pre-evaluation 45 Primary data 103 Prior appraisal See Ex-ante evaluation Product See Output Programme 33 Programme cycle See Programme Programme indicator 73 Programme theory See Theory of action Project 34 Project promoter 39 Qualitative data See Descriptor Quantification See Measurement unit Quantitative data See Quantitative objective Indicator See Verifiable objective Questionnaire survey 104 Raison d'etre See Rationale Rating See Scoring Rationale 62 Realisation See Output Recipient See Direct addressee Regression analysis 108 Reliability 47 Representativeness See Reliability Resource See Input Result 57 Rolling evaluation See On-going evaluation Scope of evaluation 33 Scoping See In-depth evaluation Scoring 69 Screening See Overall evaluation Secondary data 103 Secondary effect 86 Self-evaluation 41 Sensitivity See Indicator Shift-share analysis 109 Side effect See Unexpected effect Single loop learning learning See Organisational Skimming-off effect 92 Social cost benefit analysis 112 Social value See Criterion Soundness See Reliability Specific impact See Impact Specific objective See Objective Stakeholder 37 Stakeholder evaluation See Pluralistic evaluation Standard See Norm Standard indicator See Comparability Stastitical team 70 Statement See Descriptor Steering group 42 Strategy 53 Structuring effect 88 Subsidiarity 66 Substitution effect 80 Success See Effectiveness Summative evaluation 18 Supplier effect 87 Supply-side effect 87 Sustainability 64 Sustainable development 93 Sustainable result See Impact SWOT 101 Synergy 89 System of indicators Sample 104 See Monitoring information system English alphabetical index Target group See Eligible public Technique See Tool Terms of reference 46 Thematic evaluation 25 Theory of action 82 Time series 103 Tool 99 Transition rate 91 Trustworthiness See Credibility u Unexpected effect 58 Unit-cost See Efficiency Utility 65 Value for money See Efficiency Verifiable objective 55 w Weighting 62 Winners and losers See Reached group Acknowledgements The preparation of this Volume took place over several years from 1993 to 1998. Preparatory works included the following steps : An initial version of the glossary in five languages was produced in 1993 in the form an internal document for the European Commission. A second version was produced in 1995 which included nine languages and took into account the comments of an international panel of experts. This Volume is the direct result of the work undertaken to ensure that the whole of the MEANS Collection is based on a coherent vocabulary. The production of this Volume was coordinated by Miguel Benito Alonso and Andrea Mairate (D. G. XVI. G. 2 : Coordination of Evaluation) and involved several Directorate- Generals of the European Commission, notably D. G. II, V, VI, XIV and XVI and in particular Philippe Goybet, Anastassios Bougas and David Sweet. This Volume has been written by the Centre for European Evaluation Expertise, notably with the participation of Kristian Colletis-Wahl. It was translated into English by Liz Libbrecht. The authors wish to thank all those who contributed, either directly or indirectly, to the production of this Volume, and in particular the following experts: Dr Ahonen (University of Tampere), X. Ballart (Universitad Autonoma de Barcelona), G. Becher (Prognos A. G), M. L. Bemelmans-Videc (University of Nijmegen), M. Meneguzzo (Universita Bocconi), Ch. Pollitt (Brunei University), Thanos Skouras (Athens University of Economics and Business), Markus Spinatsch (Switzerland) ), E. Vedung (Uppsala University). European Commission Evaluating socio-economic programmes Glossary of 300 concepts and technical terms Luxembourg: Office for Official Publications of the European Communities, 1999 1999 — 135 pp. — 14. 8 χ 21 cm ISBN 92-828-6626-2 Venta · Salg ■ Verkauf · Πωλήσεις · Sales · Vente · Vendita · Verkoop · Venda · Myynti · Försäljning 8ELG1QUE/BELGIE ÖSTERREICH Jean De Lannoy Avenue du Ro· 202/Komngsiaan 202 B­1190 Bruielles. Brussel Tel (32­2) 538 43 OS Fax (32­2) 538 08 41 E­mail |oan de lannoyOintoboard be URL: http //www« je an­de­lan no y be LB librairie européenne; De E u r o p e ·· Boekhandel Rue do la Lo· 244/Wetstraat 244 B­1040 Bmxe«es,_russei Tel (32­2)295 26 39 Fax (32­2) 735 08 60 E­mail mailOhbeurop be URL hrtpy/www iibourop be Moniteur belge/Belgisc h Staatsblad Roe de Louvam 40­4_leuvenseweg 40­42 B­1000 BruxeUes/Bmssei Tel (32­2)552 22 11 Fax (32­2) S1101 84 DANMARK J. H. Schultz Information A/S Herstedvang 10­12 DK­2620 Albertslund TH (45,43 63 23 00 Fax(45)43 63 19 69 E­mail Schultz O Schultz dk URL http //www schuiu. dk DEUTSCHLAND Β unde aanzei ger Verlag GmbH V ert »eOsabt _ lung Amsterdamer Strafle 192 D­5073S Koir­ Tci (49­221)97 6 6 80 Fax (49­221) 97 66 B2 78 E­Mail vertrieb O b undesanzeiger de URL hrp. '. 'www bundesanzeiger de EAAA_A/GREECE Q. C. Eleftheroudakis SA Intemationat Bookstore Paneptsnmiou 17 GR­10564 Athma Tel (30­1)331 41 8 0 Ί / _ _ 4 /5 Fax (30-1)323 98 21 Email etebooksOnetor gr ESPANA Bolelin Oficial del Estado Trafalgar. 27 E 28071 Madrid Tel (34)915 36 21 11 (Libros). 913 B4 17 15(Suscnp) Fax (34) 915 38 21 21 (Libros). 913 64 17 14<SuSCnp) Ε-max ctientesOcom boe es URL. http //www boe es Mundi Prensa Libros. SA Caste*d. 37 E-26001 Madnd Tel (34)914 36 37 00 Fax (34) 915 75 39 98 E-mail librerlaOmundiprensa os URL hnpy/wwwrnundprensacorn FRANCE Journal officiel Service des publications dos CE 26. rue Dosai« F-757_7 Pans Cedex 15 Τ« (33) 140 58 77 31 Fax (33) 140 56 77 00 URL http/. 'www ¡ournal­otriciel gouv tr IRELAND Govern m· ni Supplies Agency PuOkcaiions Section 4­5 Harcourt Road Dublin 2 Tel (353­1)661 31 11 Fax (353­1) 475 27 60 ITALIA U c o s a S pA Via Duca di Calabria, i/l Casana postale 552 1­50125 Firenze Tel (39) 055 64 83 1 Fax (39) 055 64 12 57 Ε­mai kcosaOtibcc ri URL hnpj/www ftbcc it/licosa LUXEMBOURG Messageries du livra SARL 5, rue Raiffeisen L 2411 Luxembourg Tel (352) 40 10 20 Fax (352) 49 06 61 E mail mailOmd) lu URL http//www m_ lu NEDERLAND SOU Servicecentrum Uitgevers C n π stoffel Planlijnstraal 2 Postbus 20014 2500 EA Den Haag Tel (31­70) 376 96 60 Fax (31­70) 376 97 63 E­mail sduOsdu nl URL: http //www sdu nl Manz sch« Verlags­ und Universitet sbuchfi i nd lung GmbH Kohlmarkl 16 A­1014 Wien Tet. (43­1)53 1611 00 Fax (43­1) 53 16 11 67 E­Mail bestellen O manz co ai URL' http //www manz at/Index htm PORTUGAL Distribuidora de Livros Bertrand L d. ' Grupo Bertrand, SA Rua das Tenas dos Vales. 4­A Apartado 60037 Ρ­2700 Amadora Tel (351­1)495 90 50 Fax (351­1)496 02 55 Imprensa Nacional­Casa da Moeda, EP Rua Marquas Sá da Bandeira. 16­A Ρ­1050 Lisboa Codex Tet. (351­1)353 03 99 Fax (351­1)353 02 94 E­mail, del incmOmaJ telepac pt URL: htlp7/www inem pt SUOMIFINLAND Akateeminen Kirjakauppa/ Akademiska Bokhandeln Keskuskatu i Ce ni raígala n ι PtVPB 128 FlN­00101 Helsinki Helsingfors Ρ Λη (358­9) 121 44 1Θ F. /fax (358­9) 121 44 35 Sähköposti akatilausOakaleeminencom URL: http ■. www akateeminen corn SVERIGE B T J AB Traktorvägen 11 S­221 82 Lund Ttn (46­46) 16 00 00 Fax (46­46) 30 79 47 E­posi btfeu­pubObt] se URL: httpJ/wwwbij se UNITED KINGDOM The Stationery Office Ltd International Sales Agency 51 Nine Elms Lane London SW8 SDR Tel (44­171)873 90 90 Fax (44­171)873 94 63 E­mail: tpa. enquinesetttesQ. couk URL: http //www. the­S lal ione ry­off ice co uk ISLAND Bokabud Lar u ur Blondel S* Ol av or Ou s! ig 2 IS­101 Reykjavik Tel (354)551 56 50 Fax (354) 552 55 60 NORGE S wets Norge AS Ostenioveien 1Θ Boks 6512 Etterstad N­O606OS1O Tel (47­22)97 45 00 Fax (47­22) 97 45 45 : ; o i W L : _ ! ; t j : : , Si SVIZZERA Euro Info Center Schwell co OSEC Stampfenbach Straße 85 PF 492 CH­8035 Zürich Tel (41­1)365 53 15 Fax (41­1) 365 54 11 E­mail eicsOoseccn URL http. '. Www osoc ctvoics BÁLGAHUA Europress Euromedia Ltd 59. btvd Vilosha BG­1000 Sotia Tel (359­2) 980 37 66 Fax (359­2) 980 42 30 E­mail MilenaOmbox eil bg C E SKA RE PUBLIKA ÚSIS NlS­prodejna Havelkova 22 C Z ­ l 3 O 0 0 P r a h a3 Tel (420­2)24 23 14 86 Fax (420­2) 24 23 11 14 Email nkpospOdec ms « URL: riMp usiscr cz CYPRUS HRVATSKA Medialraoe Ltd Pavia Haua ι HR­10000 Zagreb Tel. (385­1)491 94 11 Fax (385­1)481 94 11 MAGYARORSZAG Euro Info Service Europa Haz Margitszigel PO Box 475 H­1396 Budapest 62 Tel (36­1)350 80 25 Fax (36-1)350 90 32 E­mail euiomloOmail mala v hu URL hrtp//wwweuroinfo. hu/inaex. htm MALTA Miller Distributors Ltd Malta International Airport PO Box 25 Luqa LQA 05 Tel (356) 66 44 68 F a x ( 3 5 6 ) 67 67 99 E­mail, gwirth O usa net POLSKA Ars Polona Krakowskie Przedmiescie 7 Skr pocztowa 1001 PL­00­950 Warszawa Tel. (48­22)826 12 01 Fat (48­22) 826 62 40 E­mail: ars po' O bevy hsn com pi ROMANIA E u r o m e d ia Sir G­ralBerthek)lNr4l RO­70749 Bucuresti Tel (40­1)31544 03 Fax (40­1) 314 22 86 ROSSIYA CCEC 60­tetiya Ofctyabrya Av 9 117312 Moscow Tel ( 7 - 0 9 5 ) 1 35 52 27 Fax (7095) 135 52 27 SLOVAKIA Centrum VTI SR Nam Slcbody 19 SK­81223 Bratislava Tel (421­7)54 41 B3 64 Fax (421­7) 54 41 B3 64 E­mail europOtbbl sltk stuba sk URL http://wwwsrtk. _uba sk SLOVENUA GospodarsM Veslnlk Dunaiska cesta 5 SLO­1000 Ljublana Tet ( 3 8 6 ) 6 1 3 0 9 1 6 40 Fax ( 3 8 6 ) 6 1 3 0 9 1 6 45 E­mail europOgveslrwksi U RL http://www gvestmk si TURKIYE Du nya Infoisi AS 100. Yil Mahallessi 34440 TR­80050 Bagolar ­Istanbul Tel (90-212)629 46 89 Fax (90-212) 629 46 27 E­mail: inlotetOdunya­gazetecom Ir AUSTRALIA Hunter Publications PO Box 404 3067 Abboisford. Victoria Tel (61­3)94 1 7 5 3 61 Fax (61­3) 94 19 71 54 E­mail (pdaviesOozemail com au CANADA Les éditions La Libert* Inc. 3020. chemin Sainte­Foy G1X 3V Sainle­Foy. Ouébec Tel (1­418)658 37 63 Fax (1­800) 567 54 49 E­mail liberteOmedKVTi qcca Renouf Publishing Co. Ltd 5369 Chemin Canotek Road Unit 1 KU 9J3 Ottawa. Ontano Tel (1­613)745 26 65 Fax(1­613) 745 76 60 Email Order deptOrenoufbOOkS com URL. httpJ/www renoufbookj. com EGYPT Cyprus Chamber of Commerce end Industry PO Bo« 1455 CY­1509 Nicosia Tel ( 3 5 7 2 ) 66 95 00 Fax (357­2) 66 10 44 E­mail demetrapOccci orgcy Tel (20­2)392 69 19 Fax (20­2) 393 97 32 E­mail matoudaOmeobserver. com ι URL httpy/www meobserver. com eg EESTI INDIA Eesti Ksubandus­Toosluskoda (Estonian Chamber ol Commerce and Industry) Toom­Kook 17 E E­0001 TaH. nn Tel. (372)646 02 44 Fax (372) 646 02 45 E­mail emtoOkoda. ee URL: http //www koda ee EBIC India 3rd Floor. Y B Chavan Centre Gen J. Bhosale Marg 400 021 Mumbai Tet (91­22)282 60 64 Fax (91­22) 2B5 45 64 E­mail: ebicOgiasbm01. vsnl. net. in URL: http. //www ebcmdiacom ROY International 41. Misturar Ha>arden Street PO Box 13056 61130 Tet Aviv Tel (972-3)649 94 69 Fax (972­3) 648 60 39 E­mail royilOnetvision. net. il URL http '/www royint co il Sub­agoni lor the Palestinian Authority: Index Information Services PO Box 19502 Jerusalem Tel (972­2)627 16 34 Fax (972­2) 627 12 19 JAPAN PSI­Japan Asahi Sanbancho Plaza «206 7­1 Sanbancho. Chiyoda­ku Tckyo 102 Tel (81­3)32 34 69 21 Fax (81­3) 32 34 69 15 E­mail: booksOpsi­japancojp URL http_'n*ww psi­japan com EBIC Malaysia Level 7. Wisma Hong Leong 18 Jalan Perak 50450 Kuala Lumpur Tel (60­3)262 62 98 Fax [60­31 262 61 98 E­mail: e_c­klOmol. net my Mundi Prensa Mexico, SA de CV Rio Panuca No 141 Colonia Cuauhtemoc MX-06500 Mexico. DF Tel (52-5) 533 56 58 Fax (52­5) 514 67 99 E­mail 101545 2361 Ocompuservö com EBIC Philippines 19tti Floor. PS Bank Tower Sen. Gii J Puyat Ave cor Tindalo SI Makali City Metro Manilla Tel. (63-2) 759 66 30 Fax (63­2) 759 66 90 E­mail. eccpcomOglobe. com ph URL http^/www eccp com EBIC Sri Lanka Tr ans Asia Holel 115 Sir chittompalam A. Gardtner Mawalha C_ombo2 Tel (94­1)074 71 50 78 Fax (94­1) 44 87 79 E­mail: ebtcsIOitmincom EBIC Thailand 29 Varassa Building. 8lh Floor Soi Ch idiom Ploenchit 10330 Bangkok Tol (66-2)655 06 27 Fax (66­2) 655 06 26 E­mail e_cbVkOksc15. lhcorrt URL: http/www. ebtcbkk org UNITED STATES OF AMERICA Beman Associates 4611 F Assembly Dnvo U n h am MD20706 Tel (1­600) 274 44 47 (toll (ree telephone) Fax ( 1 ­6001 865 34 50 (loll free fax) E­mail: queryObeman. com URL: MtpVwww beman com Bitte wenden Sie sich an ein Büro Ihrer Wahl/ Please contaci the sales off Ico ol your choice/ Veuillez vous adresser au bureau de vente de votre choix O f l ke for Official Publications of lhe Europeen Communities 2. 'm­· Mercier L­2985 Luxembourg T_ (352)29 29­42455 Fax (352) 29 29­42758 E­mail: info infoOopoce cec be URL. http://eur­op. eu int The MEANS collection: "Evaluating socio-economic programmes" This six-volume set gives wider public access to the lessons of the MEANS programme which the European Commission introduced with a view to impro­ ving and promoting evaluation methods. Its publication meets a growing need for reliable benchmarks to deal with the problems of evaluation encoun­ tered when implementing public measures in general and Structural Fund ope­ rations in particular. In this context, the set constitutes a unique methodological guide that can solve the technical and organisational problems arising in the course of evaluation work and for which the existing literature provides little help. The collection is aimed not only at professional evaluators, for whom it will provide a valuable handbook, but also at non-specialists, in particular manag­ ing authorities and structural programme Monitoring Committees. Evaluation design and management Each volume is des gned to be independent of the others, yet they are also complementary. Together, they deal with all aspects of the evaluation of socio­ economic programmes: I. II. Selection and use of indicators for monitoring and evaluation III. Principal evaluation techniques and tools IV. Technical solutions for evaluation within a partnership framework V. Transversal evaluation of impacts on the environment, employment and other intervention priorities VI. Glossary of 300 concepts and technical terms OFFICE FOR OFFICIAL PUBLICATIONS OF THE EUROPEAN COMMUNITIES L-2935 Luxembourg