Evaluating socio-economic programmes : volume 3, Principal evaluation techniques and tools.
EC Structural Funds Evaluating socio-economic programmes Principal evaluation techniques and tools 1 MEANS Collection Volume 3 II an Commission EC Structural Funds Evaluating socio-economic programmes Principal evaluation techniques and tools Volume 3 European Commission MEANS is a programme of the European Commission. It is aimed at improving Methods of Evaluating Stmctural Policies. The MEANS Programme and Collection was supervised by Phillipe Goybet and Miguel Angel Benito Alonso, Unit for the Coordination of Evaluation in the Directorate for Coordination and Evaluation of Operations of DG XVI Coordination-evaluation@fmb. dg 16. cec. be The realisation of the Programme was entrusted to the Centre for European Evaluation Expertise (C3E) under the direction of Eric Monnier and Jacques Toulemonde. Responsibility for the opinions expressed in this book lies solely with the authors of the text: C3E - 13 bis Place Jules Ferry - F - 69006 Lyon - France - c3e@c3e. fr A great deal of additional information on the European Union is available on the Internet. It can be accessed through the Europa server (http://europa. eu. int) and on the Inforegio Website (http://inforegio. cec. eu. int). Cataloguing data can be found at the end of this publication. Luxembourg: Office for Official Publications of the European Communities, 1999 ISBN 92-828-6626-2 © European Communities, 1999 Reproduction is authorized, except for commercial purposes, provided the source is acknowledged. PRINTED ON WHITE CHLORINE-FREE PAPER Printed in Italy Foreword is directly Developing good evaluation practices has become a priority area for the relevant Commission and to the effective use of Community resources. Indeed, the sound financial management principles which the Commission seeks to promote via its SEM2000 (Sound and Efficient Management) Programme rely, specifically, on the regular evaluation of the Community activities. The Directorates General responsible for structural policy have considerable expe­ rience in this area. Since 1988 they have in the process of integral part of each stage made evaluation an Community Structural intervention. Thus, the programme development stage includes a prospective evaluation; the implementation stage a strengthening of the monitoring systems and, at the end of the intervention, evaluation establishes what has been achieved in terms of programme impacts. However, the capacity of evaluation to contribute to the effectiveness and transparency of structural interventions rests ultimately on the integrity of the methods adopted to verify the credibility of results. This is why, four years ago, on the initiative for the Directorate General for Regional Policy and Cohesion (DG XVI), the MEANS programme was initiated as a response to the demand for developing evaluation methodo­ include practices combining both general scientific logy. This also requirements and the specific requirements of Community interventions whilst, at the same time, remaining accessible to a large audience. One of the more tangible results of the MEANS programme has been the publication of this six volume set by a group of independent experts, focu­ sing on the principal elements of socio-economic programme evaluation. A short summary of this work can be found at the beginning of the present volume. This collection covers the various methodological approaches and innovations that have taken place in the context of structural funds eva­ luation. It will enable those responsible for evaluation to develop their approach in a flexible and independent manner and for the various programme partners to exercise greater control over the evaluation pro­ cess. Attending the conferences of evaluation specialists and "clients" convened in Brussels (1995), Berlin (1996) and Seville (1998), I gained some impression of the interest that new evaluation techniques can gene­ rate. I would stress that the development of evaluation needs to progress in tandem with efforts to incorporate evaluation findings in the decision making process. This implies evaluation as a means of establishing whe­ ther policies are equitable, effective and relevant to the identified needs. In satisfying these requirements, evaluation should be seen as a valuable instrument affording both Member State and Community authorities the necessary guarantees that Community funds are being deployed in the best interests of the EU citizen. This is a cost worth paying when the goals of Economic and Social Cohesion and the European Union enlargement are at stake. Finally, I wish to extend my thanks to the members of the Committee of Independent Experts established at the launch of the programme. Their expertise and standing have contributed greatly to the quality and relevan­ ce of the MEANS programme. I therefore thank: E. Chelimsky (USA), ex- President of the "American Evaluation Association"; K. Kennedy (IRL), Director of the "Economic and Social Research Institute" (ESRI); J. R. Cuadrado Roura (SP), President of the "European Regional Science Association"; C. Seibel (F), Director of the "Statistiques démographiques et sociales de l'Institut Nat onal de la Statistique et des Etudes Economiques" (INSEE); L. Senn (I), Director of the Department of Economie Regional of the University Bocconi; E. Stern (U. K. ), Director of the unit of evaluation of the Tavistock Institute; A. Sorber (t) (NL), Head of the "Policy Analysis Department of the Ministry of Finance"; L. I. Strom (SV), Statens Institut för Regionalforsking (SIR); L. Tsoukalis (GR. ), professor at the University of Athens; H. Wollmann (D), professor at the University of Humboldt - Berlin and founder member of the "European Evaluation Society". E. Landaburu The MEANS Collection: "Evaluating socio-economic programmes" of European Union The evaluation structural interventions has entered into a phase of maturity, attested by the extension of work carried out by both the Member States and the Commission. Regulations based on decisions by the Council have largely contributed to the development of evaluation practice and have stimulated methodological research. initiated by the Commission Over the past ten years, the Commission has striven to promote and improve methods for evaluating Structural Funds. The main methodological conclusions have been debated and disseminated as part of the MEANS in 1991. programme Achievements to date provide enough material for a complete and accessible publication intended for a broad public: a six volume Collection covering all aspects of the evaluation of socio-economic programmes. These Volumes are written with a perspective that is broader than the current administrative framework, for this is likely to evolve more rapidly than professional standards. information on In general, the Collection is aimed at a readership of non- specialists seeking essential the evaluation of socio-economic development programmes. These programmes have multiple objectives, a partnership dimension and a multi-sectoral content - all characteristics which call for suitable evaluation methods and solutions not yet provided by the international technical literature. The complete Collection consists of 6 volumes designed to be complementary and to provide essential for professionals wanting to independently develop their own evaluation practices. Each volume can, however, be read and understood without having to refer to the other volumes in the Collection. recommendations information and Principal evaluation techniques and tools The first Volume in the MEANS Collection entitled "Evaluation design and management" serves as an introduction both to the Collection and to the evaluation of socio-economic programmes. It enables readers to understand the implications of evaluation and to know what they can and cannot expect from an evaluation at a given time in a particular context. It covers all the elements needed to steer an evaluation project from A to Z: sharing of responsibilities, defining of the aims and method of an evaluation, establishing a schedule and budget, assessing the quality of the work performed and, finally, managing the use of conclusions. The second Volume focuses on the "Selection and use of indicators for monitoring and evaluation". It deals with the subject of indicators used for monitoring and evaluation. It shows how indicators can be used to support policy decisions, improve management and measure the overall performance of programmes. The volume proposes a methodological framework and a Dractical guide for the use of indicators in these different situations. The third Volume presents a selection of "Principal evaluation techniques and tools" applicable to the evaluation of socio-economic programmes. These tools are taken from relevant scientific disciplines: economics, sociology, management, geography, etc. Each one of the 23 tools is described in the form of an information sheet featuring a general description, the principles of its implementation, a list of its strengths and weaknesses for the evaluation of Structural Funds, an example of its application and a list of references. A number of methods adapted to the evaluation of socio­ economic programmes are the object of a more detailed description in the fourth Volume entitled "Technical solutions for evaluation within a partnership framework". Several pilot evaluations were implemented within the MEANS programme to design and test the following four innovations, the application of which is described in The MEANS Collection detail in this Volume: impact mapping, cross-impact matrix, score cards and multi-criteria analysis. The fifth Volume deals with "Transversal evaluation of impacts on the environment, employment and other intervention priorities" of the European Union and Member States. Among the questions addressed are impacts in terms of equal opportunities between men and women and the competitiveness of SMEs. The Volume provides the elements required for an analysis of those impacts which are not necessarily part of the explicit objectives of all the actions evaluated. Finally, the last Volume is a "Glossary of concepts and technical terms". The compilation of the Collection has necessitated the rigorous standardisation and definition of the terms used. Each of the first five volumes has an index of terms which, along with hundreds of definitions, are brought together in the glossary. Contents Introduction 9 I Presentation of evaluation tools 15 ■ Tool, function, method 17 ■ Properties of evaluation tools 19 ■ Advantages and risks of technical complexity 22 ■ Steps and functions of an evaluation 24 ■ Evaluation situations 28 II Tools for the overall evaluation of a programme 33 Examples of the evaluation of multi­sectoral programmes 35 Tools for structuring an evaluation 39 H SWOT Analysis (Strengths­Weaknesses­Opportunities­Threats) 41 ■ Concept mapping of impacts 47 ■ Colour vote 55 Tools for observing changes in the field 59 ■ Individual interview 61 ■ Focus Group 67 ■ Case study 73 Tools for analysing data 81 ■ Geographical Information Systems 83 ■ Shift­Share analysis 89 ■ Input­Output Model 95 ■ Macro­economic model 103 Tools for making a judgement 115 ■ Expert panels 117 ■ Multicriteria analysis 123 III Tools for in­depth treatment of an evaluative question 129 Examples of evaluations with in­depth treatment of a question 131 Tools for structuring an evaluation 135 ■ Logical framework 137 METAPLAN® 145 Principal evaluation techniques and tools Tools for observing changes in the field 151 ■ Questionnaire survey 153 ■ Ethnographic observation 161 Tools for analysing data 167 ■ Factor analysis 169 ■ Delphi Survey 175 ■ Comparison groups 179 ■ Regression analysis 187 Tools for making a judgement 193 ■ Cost-Effectiveness Analysis 195 ■ Benchmarking 201 ■ Cost-benefit analysis 207 IV Recommendations for selecting and assembling tools 217 Guide for identifying the most appropriate tools 219 Recommendations for assembling several tools 225 Conclusion 231 Index 233 Introduction A professional who wants to know what the evaluation toolbox contains, has to consult numerous texts and manuals devoted to particular techniques in the economic and social sciences. Fortunately, some reference documents present a range of tools for evaluation purposes. Among the classics are Rossi & Freeman (1993), as well as the Program Evaluation Kit collection1. These books feature "technical information sheets" illustrated by means of examples and all are organised according to each author's own typology. The most frequent typological distinctions separate tools in terms of their nature, i. e. quantitative or qualitative, deductive or inductive, etc. A particularly interesting typology is presented by C. Pollitt (Box 1). Is there any need for a new book on evaluation tools and, if so, what might its contribution be? The main need that this Volume has to fulfil is to make existing knowledge readily available to readers working in the field of socio­ economic programmes. In fact, most documents on evaluation tools are illustrated with examples of social programmes or development projects. The present document provides a link between general methodological works and the context of European Structural Funds. is It is secondly necessary to adapt available knowledge to specific difficulties in the evaluation of socio-economic programmes. Most texts on evaluation tools refer to simple programmes, aimed at a well-defined target group and at producing precise change within that group. However, a socio-economic development programme typically multi-sectoral, with multiple objectives. Its purpose is to produce economic and social transformations on the scale of an entire territory. To achieve this goal, an integrated strategy is adopted, oriented towards different publics (firms, workers, the unemployed, communities, etc. ). Its global impact is produced by means of a combination of multiple specific impacts on various targets. In this context, evaluation encounters complex technical problems for which the existing first methodological guide to record and present a number of relevant solutions. few solutions. This book literature provides the is 1 Rossi RH, Freeman H. E. (1993) Evaluation: a Systematic Approach (5th edition) Newbury Park CA: Sage. Program Evaluation Kit, Center for the Study of Evaluation, University of California. Newbury Park CA: Sage Principal evaluation techniques and tools Box 1- Four approaches to evaluation Experimental approaches treat phenomena of causality without analysing them directly. The effects of programmes are demonstrated by comparing a treatment group with a control group. Evaluations carried out in this framework are typical of American practice in the 1960s and 70s. These evaluations are long, cumbersome, and are not always conclusive. They are often criticised, sometimes severely, by their commissioners who want operational conclusions that can be used rapidly for improving programmes. Economic approaches are based on an individualistic view of society, inspired by welfare economics. From this point of view, the value of a public action is the sum of the benefits it provides to individuals. All individuals are supposed to have a system of references and it is assumed that these systems of reference can be aggregated. These hypotheses are used to give an objective character to techniques. Despite their elegance, evaluations carried out by economic economic techniques have been strongly criticised, in both practical and theoretical terms. Naturalistic or pluralistic approaches are based on the idea that the political and social world is a collective construction. This construction results from the interaction of differing social groups that have their own interpretations of the same phenomena and issues. Consequently public programmes are seen as temporary compromises between groups of actors. By means of appropriate techniques, based to a large extent on group work, evaluation acts as a mediator between the different points of view. It is a tool for promoting conciliation between the stakeholders, and its conclusions are all the more robust when they are a product of consensus. Pragmatic approaches adopt simplified views of the processes that they have to describe. Theoretical references are less pure and more eclectic. Evaluation objectives are of an essentially managerial rather than scientific nature. Evaluations carried out in this spirit, using techniques inspired by management approaches, are oriented more towards the implementation and improvement of efficiency. Their cognitive dimension is less important than in other forms of evaluation. Some of the technical suggestions in this Volume would not exist without the numerous studies and experiments carried out by the European Commission. To adapt available knowledge to the difficult field of socio­ economic programmes, substantial methodological investments were required, including the precise definition of the basic concepts (e. g. tools, methods), the definition of functions likely to be fulfilled by the tools, the construction of new tools (e. g. specialised macro-economic models, impact mapping) or the proposal of new ways of using classical tools. Introduction These investments have benefited the entire professional evaluation community. This is the third and last reason for the existence of this manual. The manual contains a detailed presentation of twenty­three evaluation tools, in the form of technical information sheets. The choice of tools is a compromise between the most frequently used tools and others that, although rarer, have significant potential for the evaluation of socio­ economic programmes. Since all compromises are to some extent arbitrary, certain frequently used tools are only mentioned in passing, without having technical information sheets devoted to them (e. g. variance analysis). Box 2 ­ For a rapid pre­selection of appropriate tools S Θ High technical complexity Medium Low technical technical complexity complexity O O Retrospective Prospective use use Φ Low cost (è) @% V^~"»// νΛ*·»*JJ) Medium High cost cost Θ 0 Stabilised use Innovative use O Overall evaluation A In­depth evaluation Each information sheet provides data on the functions of the tool, the conditions of its use, and the main steps in its implementation. Possibilities for application in the framework of Structural Fund evaluations are suggested, and the strengths and weaknesses of the tool in practice are highlighted. An example taken from a Structural Fund evaluation is described, if the tool has been used in that context. Each sheet also contains a commented bibliography of books and articles that refer to the tool. The information sheets are accompanied by pictograms that enable readers to rapidly select the tools applicable to a given situation. The meaning of these pictograms is given in Box 2. Principal evaluation techniques and tools This Volume is divided into the following four main sections: • Presentation of evaluation tools • Tools for the overall evaluation of a programme (12 tools) • Tools for more in-depth treatment of an evaluative question (11 tools) • Recommendations for selecting and assembling the tools I Presentation of evaluation tools In this Volume the terms "tools" and "techniques" are taken to be synonymous and applicable to standardised procedures which any evaluator can learn to use and apply in carrying out her/his missions. The evaluation "toolbox" contains over 23 techniques, presented in this Volume. The evaluator's art therefore involves the choice of an appropriate tool and the rejection of inappropriate ones, depending on the function they are meant to fulfil in a particular situation. Presentation of evaluation tools Tool, function, method Evaluation practitioners use various words to describe their instruments or procedures: technique, tool, approach, procedure, method, design, etc. It is essential to agree on certain definitions before reading this Volume. The following example will be used to establish these definitions. Box 3- Example of a tool(s)-based evaluation In 1989 the British National Audit Office evaluated eight regional development programmes in Wales. The evaluation calculated the cost per job created, in other words, it was based on cost-effectiveness analysis, a standard evaluation tool. To evaluate the eight programmes, the NAO adopted an approach in several stages: 1 - Selection of a single evaluation criterion: net jobs created. 2 - Construction of an indicator for measuring net jobs, in terms of the following definition: net jobs = gross jobs - deadweight effects - displacement effects + supplier effects. 3 - Selection of a sample of about 300 assisted businesses. 4 - Questionnaire survey by displacement coefficients. telephone, estimation of deadweight and 5 - Estimation of total jobs created by extrapolation from the coefficients obtained from all jobs reported by all the assisted businesses. 6 - Estimation of supplier effects by extrapolation from studies of similar programmes. 7 - Recapitulation of the public funds mobilised for the programmes, and cost- effectiveness analysis in terms of cost per job created. 8 - Comparison with the cost per job created in the framework of similar programmes - here, a regional development programme in Scotland. For further information on this evaluation, see: S. Roberts and Ch. Pollitt, (1994) Audit or Evaluation? a National Audit Office VFM Study, Public Administration In this example, there are at least t wo tools commonly used in evaluation: • the questionnaire survey (4th step) • cost-effectiveness analysis (7th step) It should also be noted that the evaluation work involved far more than simply the use of t wo tools. At each of the eight stages, a decision was made on how to carry out the work. Each decision depended on the 'specific context of this evaluation. The NAO thus constructed an ad hoc evaluation method for fulfilling its mission. Principal evaluation techniques and tools This example will be used to define three of the basic terms in this Volume: "tool", "method" and "function". A tool is used to carry out standard treatment and may be applied to any evaluation requiring such treatment. For example: • a questionnaire survey is used to gather the opinions of a group on one or several points defined in advance; • a cost-effectiveness analysis calculates the public funds spent to obtain a given effect; the result is presented in a form, which allows programmes and projects to be compared. A technique is so similar to a tool that in this Volume they are often used as synonyms. However, a tool can be considered to be an object (e. g. a shovel) and a technique the way in which this object is used (e. g. the mani­ pulation of the shovel). A method is an ad hoc procedure, specially constructed for a given evaluation. It may include one or more tools, or none. In its evaluation of regional development programmes in Wales, the NAO constructed a method adapted to the questions raised (Box 3). By contrast, for the prior appraisal of programmes proposed by the Member States, the Commission requests appraisals that are carried out within a few weeks. In general, the methods applied to these evaluations comprise no tools. Thus defined, the term "method" is similar to "design". In the American literature, "evaluation design" is applied to the way in which a given evaluation is conducted. However, the term is often applied to the analysis of causality, that is to say, to only one of the functions that an evaluation method must fulfil. The terms "approach" and "procedure" are also similar. They apply to a given method and to the spirit in which this method has been constructed. The method constructed for carrying out a particular evaluation can always be broken down into several steps. Each of these steps consists of a task and fulfils an elementary function. The same basic elementary functions can be found in most evaluations. A function may be fulfilled with or without a tool. For example: • in the NAO example (Box 3) the data collection function is carried out by means of a questionnaire survey; • in certain evaluations of the Commission, the estimation of effects is carried out by means of a macro-economic model. Presentation of evaluation tools In this Volume, methodology is defined as the "science" of the construction of methods. The aim of this Volume is to promote the development of this science. In reality, however, professional practice is still closer to expertise than to a science. The word methodology is therefore somewhat ambitious at this stage. Properties of evaluation tools A tool is used to process raw materials (inputs) to transform them into products (outputs). Inputs and outputs are information, data, facts, opinions, judgements, etc. Inputs and outputs can be different in nature. For example, a questionnaire survey collects opinions and produces quantitative data about these opinions. All types of information feed into an expert panel to produce a judgement, etc. The tool can be defined because the mode of processing is stable, that is to say, inputs have a standard form, outputs are always of the same nature, and transformations occur according to the same principles. Yet a tool always has a degree of flexibility so that it can be adapted to use in specific situations. These "adjustments" are limited so that the tool remains identifiable. Box 4 - The road worker's shovel and cost-effectiveness analysis The road worker's shovel is used to fulfil a standardised function, consisting of moving a loose material from one place to another, for example, by filling a wheelbarrow with earth. In this example, the "inputs" are the user's physical strength and the earth to be moved, and the "output" is the earth that has been moved in the wheelbarrow. A priori, there are not many types of shovel or different ways of using them. One could say that this tool performs a stable transformation. The main variants are the size of the shovel, the distance of the movement and the nature of the material to be moved (i. e. how loose it is). The case of the shovel may be transposed, term for term, to the field of evaluation, for example to cost- effectiveness analysis. The inputs in cost-effectiveness analysis are twofold. It is necessary first to measure the impact obtained (in a single dimension, e. g. net jobs created) and then the amount spent. The output is simple; it is the cost ratio per unit of impact. The transformation is stable; it consists of a simple division. The degree of freedom concerns, above all, the way of taking cost Into account: budgetary cost for all the public partners, budgetary cost for the only public actor that commis­ sions the evaluation, opportunity cost of public funds, cost limited to the duration of the programme or cost extended to all public support, etc. Principal evaluation techniques and tools A tool may be used freely by anyone who needs it. Most of the tools used in evaluation are free of charge but some have to be bought. For example, a tool used for conducting meetings, called METAPLAN®, presented in this Volume, is a registered trademark. A tool has instructions for use, described in one or more manuals. Training courses on tools and their use are offered for beginners. Quality standards ensure that tools are used professionally. To summarise, a tool can be recognised from the following characteristics, featured in Box 5, it can be identified by a name, it is used to carry out a process with standard inputs and outputs, it can be adapted but in a very limited number of ways, it has a manual and quality standards. All these properties make the tool transferable from one evaluation to another and from one professional to another. In particular, its use is public and not limited to its inventor. Box 5 ­ Features of an evaluation tool Standards inputs O Name O O O V '/ Adjustments ■=>¡ Standard outputs Manual Quality standard This diagram corresponds perfectly to some of the tools presented in this Volume, as the example of regression analysis developed below shows. In this example, the basic characteristics of a tool can be recognised, i. e. : • standard inputs (quantitative data on apparent effects, on participation in the programme and on the other explanatory variables since these data are available for a large number of individuals); • some adjustments (number and nature of variables, nature of relations, thresholds of validity); Presentation of evaluation tools • standard outputs (distribution of apparent or gross effects between the net effects attributable to the programme and the effects of the other explanatory variables). Moreover, the tool is identified by means of a name (regression analysis), irrespective of the variants corresponding to the different adjustments (linear regression, logistic regression, etc. ). Finally, the tool is described in numerous manuals and in all European languages. Box 6 - Regression analysis as an evaluation tool Data for numerous individuals : Apparent effect Participation in the programme 0 Other explanatory variables Regression analysis 0 % of apparent effects which are attributable to the programme Several of the 23 information sheets presented in this Volume have only some of the characteristics of a real "tool". For example, some have not been fully standardised (e. g. SWOT analysis), others do not have a proper manual (e. g. colour vote, METAPLAN) and others have no stabilised quality standards (e. g. multicriteria analysis). Principal evaluation techniques and tools Apart from those tools developed specifically for evaluation (e. g. cost benefit analysis, multicriteria analysis), most tools are borrowed from the main scientific disciplines. The questionnaire survey from sociology, regression analysis was created by statisticians, the input-output model comes from economics, the expert panel draws largely upon legal information system and ethnographic practices, and the geographic observation emerged from their corresponding sciences. is taken When a technique is "borrowed" from a discipline for use as an evaluation tool, it is often adapted and simplified. From a purely scientific point of view, there are always regrets about the loss of the "canonical" form of the original technique. However, evaluation is above all a pragmatic activity. The technical information sheets presented in this Volume should not be judged with the academic rigour of their original disciplines, but in terms of their usefulness for evaluation practitioners. Advantages and risks of technical complexity The use of a tool has numerous advantages in terms of guaranteeing results, evaluation content, comparability of results, and quality and cost control. It also provides a guarantee as to the form of results, on condition that the use of the tool complies with the established procedures. Thus, a questionnaire survey conducted properly (e. g. confidentially, by telephone) can serve to accurately estimate net effects. The choice of certain tools makes it necessary to structure the entire evaluation. For example, cost-effectiveness analysis means that the evaluation will favour a principal effect (e. g. employment). The tool will, in this case, offer the commissioner a guarantee of this main effect being prioritised, if that is what is desired. Use of a known tool facilitates the estimation of the time and budget required for an evaluation because practitioners have references for grounding their estimations. Moreover, the fact that an evaluation is based on standardised tools facilitates control of the conditions of its execution. Several evaluations based on the same tools produce results that are easier to compare. In the example in Box 3, it was possible to compare the costs per job created in several regions because all the evaluations had used cost-effectiveness analysis. Presentation of evaluation tools In short, a more technical evaluation is often simpler to commission, easier to control, more credible for readers of the report and more predictable for the commissioner. All these advantages obviously make evaluation tools highly attractive, but they are no without certain risks. The risks of "tool(s)-based" evaluation can be summed up in the following two sentences: • "With a hammer in hand, I see nails everywhere" - risk of choosing the wrong tool • "Because I have a hammer, I can build the entire house" - risks of over­ estimating the use of a tool. A tool must always be adapted to its context of use and to the functions it is to fulfil. The risk is that someone with a hammer might try to use it to hammer in a screw. In an example studied in the framework of the MEANS programme, the evaluation commissioner and team decided to use regression analysis. After a few weeks, they realised that the available data were insufficient and that they were unable to apply the chosen tool. They were forced to quickly reorganise the evaluation work. To know whether an evaluation tool is appropriate in the context, it is necessary to examine the object to be evaluated, the available data, the evaluation budget, and the deadline for presentation of the results. Choosing a tool without taking all these factors into account could have serious consequences: absence of conclusions or irrelevant conclusions, deadlines that are not met, conclu­ sions that are not credible, etc. The weaknesses of the tools used must also be recognised and accepted. It is better to carry out a less technical evaluation by specifying the limits of the work, than to base conclusions on the use of sophisticated tools which were not skilfully used. Since technical complexity is tempting, there is often a tendency to over­ value tools used in evaluation. Thus, the NAO evaluation (Box 3) was presented as an exercise in "value for money", that is, cost-effectiveness analysis. The entire evaluation was given the name of one of the tools implemented and this tool became emblematic. This phenomenon has become widespread and applies to numerous evaluations which, depending on the case, may be called cost-benefit evaluation, econometric evaluation, quality/price evaluation, etc. Equating an evaluation to its emblematic tool is always a mistake. An evaluation tool is used to fulfil a function and, most of the time, one function Principal evaluation techniques and tools only. Favouring one function amounts to neglecting all the other functions required by evaluation, and thus undermining the overall quality of the work. Steps and functions of an evaluation In general, a tool fulfils the function for which it was designed, and does not fulfil the others. An example of a specific tool is a macro-economic model, which can fulfil no other function than that of estimating impacts on the scale of a country or region. Yet tools do exist which are capable of fulfilling several evaluation functions. These are generic tools. It will be seen in this Volume that the case study technique can be used for collecting data but also for other functions. Similarly, an expert panel can be set up for making judgements and for estimating impacts. All evaluations have several steps with which one or more functions are associated. In the example of the evaluation carried out by the NAO, it can be seen that there were at least five functions. Box 7 - Several steps in an evaluation (Evaluation of economic development programmes in Wales) Choosing the effects to be evaluated (employment) Defining an observation tool (indicator of net jobs created) Collecting data (300 telephone interviews) Estimating effects (cost-effectiveness analysis) ι \ 1 1 ι STRUCTURING OBSERVING ANALYSING JUDGING By analysing a large number of evaluations in the same way, the most common functions can be identified. These are presented below in eight basic steps (Box 8), grouped together into four main stages: structuring the Presentation of evaluation tools evaluation, observing changes in the field, analysing data, and judging the programme. Box 8 ­ Standard steps of an evaluation Structuring ; Clarifying and grading the effects ; to _be_e_va[ua_ted,_defi_njng_ enterra. I Choosing observation l_ instruments ___. Observing ; Defining the observation field %~», r Analysing ; Collecting data I Comparing data Judging J Estimating effects ,«·»' | Judging in terms of the """■ ' different criteria ι Rirmulating a synthetic judgement The diamond­shaped diagram is borrowed from Scriven M. (1980) The Logic of Evaluation, Inverness CA: Edgepress The structuring steps normally occur before the evaluation work as such (before the terms of reference are drawn up). Their place in the present Volume may therefore be questioned. However, the evaluation team frequently has to revert to these steps at the start of its work, for example for explaining the judgement criteria in more detail. It will also be seen that tools exist which may facilitate structuring work. It is therefore logical for this Volume to cover evaluation as a whole, including the very first steps. Clarifying and grading the effects to be evaluated, defining criteria The first step consists of listing the questions to which the evaluation must provide answers. For example: Are the taxpayers getting value for their money? What are the most effective parts of the programme? Is the pro­ gramme justified? Does the programme have negative unexpected effects? Each question can be subdivided into the effects to be evaluated, e. g. job creation, investments induced, market share gained, increase in employability of a category of the population or increase in the attractiveness of a region. Each question also refers to a judgement criterion, e. g. is the effect sought and/or obtained relevant, consistent with the objectives, efficient, lasting and fair? Principal evaluation techniques and tools The tools applicable in this step are used to list questions, clarify them, grade them and select those for which the evaluation must provide answers (see the information sheets relating to the following tools: logical framework, impact mapping, SWOT analysis, METAPLAN®, colour vote). Defining observation instruments This second step consists of the translation of evaluation questions into indicators or descriptors observable in the field. In the example of regional development programmes in Wales (Box 3), the question was as follows: "Is the employment effect of the programmes sufficient compared to the money spent?" It was translated in the form of an efficiency indicator, as follows: "cost to the national budget of a net job created". This Volume proposes no tools for this step. The reader is referred to Volume 2 devoted to indicators, and to Volume 4 for more information on the construction and use of descriptors. Defining the field of observation This third step consists of defining a field of observation that is limited enough not to generate excessive costs, large enough to produce credible information, and contrasted enough to reveal the effects to be evaluated. It is therefore a compromise involving choices such as: Should all categories of addressees be observed? Should one or more populations which are equivalent but which have not participated in the programme also be observed? How can cases be selected to obtain the fullest picture possible of the programme? Should cases of best practice be selected in order to understand the key factors of success? The field of observation, that is to say, the definition of "groups" to be observed (businesses, trainees, inhabitants, farmers, etc. ), should not be confused with the field of evaluation (definition of the intervention to be evaluated). Collecting data The fourth step consists of collecting primary data in the field. Numerous survey tools exist for this purpose. They are borrowed from various disciplines in the social sciences that collect data: sociology, political scien­ ce, ethnography, etc. Presentation of evaluation tools This step is essential if the evaluation cannot rely entirely on secondary data such as the evaluation of previous research, statistical data and monitoring data. The tools applicable to this step allow the collection of facts and opinions and to reconstitute them in a qualitative or quantitative form (see the information sheets relating to the following tools: individual interview, focus group, questionnaire survey, ethnographic observation). Comparing observations The fifth step consists of the presentation of primary and/or secondary data in such a way as to suggest conclusions and comparisons that can be made intuitively or inductively. If the example of the evaluation of a training course for the long-term unemployed is taken, the comparison of observations will consist of breaking down the rate of return to work in terms of occupational sectors, to see whether there are significant differences and to better understand the factors of success. The basic tools used for this function are graphs and comparative tables. This Volume also proposes several more sophisticated tools (see the information sheets relating to the following tools: geographical information systems, factor analysis). Estimating effects The sixth step is the one in which the net effects of the programme are estimated. This step requires an analysis of causality which almost always follows a deductive approach, i. e. an approach based on the verification of hypotheses of cause and effect. The technical literature on evaluation often favours quantitative analysis tools used to fulfil this function (e. g. comparative analysis, regression analysis). to qualitative and quantitative analysis In this respect, this Volume differs from the existing literature. It attributes tools by the same value demonstrating the strengths and weaknesses of each one. Furthermore, it does not over-value the analysis of causality, often considered as being the single role of evaluation. The main assumption is that the functions of observation and judgement are at least as important as the function of estimating net effects. Principal evaluation techniques and tools Tools applicable to this step allow the net effects of programmes to be estimated, both in the past and the future, and at both the micro and macro-economic level (see the information sheets relating to the following tools: shift-share analysis, input-output models, macro-economic models, Delphi survey, comparison groups, regression analysis). Judging effects in terms of each criterion The evaluation must be able to judge whether the effects that the programme has produced or will produce are sufficient or insufficient. In the example of regional development programmes in Wales (Box 3), the evaluation team had to judge job creation in terms of the efficiency criterion. The tool used was cost-efficiency analysis. It was thus possible to conclude that programmes were more satisfactory when the cost per job created was low. For the evaluation to be complete, it was also necessary to be able to say whether the cost per job created was sufficiently low. The standard chosen was the average cost per net jobs created by comparable programmes in Scotland. By comparing Wales and Scotland it was possible to conclude that the evaluated programmes were sufficiently efficient. This Volume presents several tools which can be used in the judgement step (see the information sheets relating to the following tools: cost- effectiveness analysis, benchmarking). Formulating a synthetic judgement The eighth and last step in an evaluation consists of formulating conclusions by making a final synthesis of the partial judgements on the different criteria. This function is necessary when several criteria have been selected. To make a synthetic judgement of a programme, it is necessary to combine, weight or grade the different effects that the evaluation has assessed. In the framework of socio-economic programmes, this function is particularly difficult since there are always several partners who do not attribute the same value to the criterion. There are several tools which totally or partially fulfil this function (see the information sheets relating to the following tools: expert panels, multicriteria analysis, cost-benefit analysis). Evaluation situations A tool is designed to fulfil a given function. There is therefore a strong connection between tool and function. However, several tools are often Presentation of evaluation tools able to fulfil the same function (see Box 9). When several evaluation tools are available, the most suitable tool for the situation has to be selected. Box 9- Significance of the situation for the choice of a tool The shovel is a tool of which the function is to move a loose material from one place to another. To fill a 50-litre wheelbarrow, an ordinary shovel is used. To fill a 25-tonne truck, it would definitely be better to use a mechanical shovel. In both examples, the function of the tool is the same, but the situation is very different. The tool must be adapted to both its function and the situation in which it is to be used. For example, in a situation of massive public intervention on the scale of a large territory, the function of estimating effects will be fulfilled by a macro- economic model. Thus, the HERMIN model was constructed and successfully used to estimate the macro-economic impacts of Structural Funds in Ireland, Portugal and Spain. In a different situation, for example if the amounts involved are small compared to the economy of the territory under consideration, or if the territory is small, the macro-economic model is not suited to the estimation of effects. As the example of the HERMIN model suggests, one of the first differences is that of the scale of effects to be evaluated, which may be micro (a small group of potential addressees) or macro (a large region or a country). The evaluation of an impact on the national economy requires the use of specific tools, particularly macro-economic models, and disqualifies others such as case studies. Another important difference is that of partnership. In an evaluation involving several decision-makers, the choice of effects to be evaluated and of evaluation criteria is of major importance and has to respect the differing points of view. Evaluations involving a multiplicity of decision-makers require the use of open tools, such as group-work techniques or multicriteria analysis. On the other hand, they are ill suited to closed techniques such cost-effectiveness analysis. is that of the use of evaluation: formative or The third difference recapitulative. A case study, which makes it easier to understand and to learn quickly from an experience, is a suitable tool for formative evaluation. By contrast, this tool produces conclusions that are difficult to synthesise for decision-makers and the public. As a result, it is less appropriate for recapitulative evaluation. Principal evaluation techniques and tools Box 10 - The three phases of evaluation In the framework of European Structural Fund programmes for the period 1994- 99, evaluation is carried out in three stages. Ex ante evaluation focuses on the planned programme. It concerns the coherence and relevance of the project, and the realism of the expected effects. Intermediate evaluations focus on the effects of the very first outputs of the programme. They are used by decision-makers at all levels. They are complementary to monitoring and are intended to reorient the programme mid­ way. Ex post evaluations are intended primarily to report on the effects of programmes. The fourth difference is that of the schedule (see Box 10). The evaluation cannot use the same tools to evaluate those effects which have already been obtained and those which are still to be produced. The fifth difference is one of the most important for the choice of tools. It concerns the complexity of the programme. The majority of socio­ economic programmes are multi-sectoral, that is to say, they intervene in several different forms, are aimed at several different publics and produce multiple kinds of impact. However, most evaluation tools have traditionally been used in straightforward situations (a single sector of intervention, a single public, and a predominant expected impact). The latter difference is very important for the choice of tools. It is also peculiar to socio-economic programmes, which is why it is a focal point in this Volume. The presentation of tools is therefore set out to correspond to two main situations: • Global evaluation of an entire programme. In this situation a regional multi-sectoral programme has to be evaluated globally, in its entire complexity. The evaluation of impacts at a macro level is required. • More in-depth consideration of an evaluative question. This situation is similar to the evaluation of a simple programme, described in most evaluation manuals. In general, the greater the complexity, the more attention must be paid to the functions of clarification and structuring, the higher the cost of empirical observation, and the more difficult the analysis of causality. Normal practice consists of breaking down the programme into categories of homogeneous interventions. Each intervention is evaluated in terms of its relevance, effectiveness or efficiency. The evaluation then makes a synthesis at the Presentation of evaluation tools programme level. The difficulty of this approach is that the different components of the programme (measures, actions, etc. ) must be evaluated in terms of criteria common to the entire programme. If all the criteria were specific the synthesis would be impossible. A variant of this situation is one in which macro-economic effects on a regional or countrywide scale must be estimated. This requires first an estimation of the micro effects of each component of the programme according to a common analytical grid, then the introduction of these effects into all the macro-economic mechanisms in order, finally, to deduce the global effect which is not equal to the sum of all the micro-economic effects. Various issues may need to be considered in greater depth: • Evaluations focused on an elementary component of the programme, for example a priority axe, a measure or an action. In this case, the outputs are homogeneous and the addressees may easily be identified and questioned. The collection and analysis of primary data are easy, as is judgement by comparison with similar interventions in other regions. • Evaluations focused on a specific public such as SMEs or the long-term unemployed. This case is very similar to the preceding one. The only difference is that several elementary components of the programme may affect the same public. • Evaluations focused on a priority theme. In this case, the relevance or impact of all the components of the programme on a particular theme, such as employment, the environment or equal opportunities, is evaluated. The difficulty in this type of situation is that the interventions are multi-sectoral, as are the results and impact mechanisms. This situation therefore involves almost as many difficulties as the overall evaluation of a complex programme. It is considered in Volume 5 of this Collection. • Evaluations focused on a specific territory. In this case, the relevance or effectiveness of all the measures of the programme on a limited part of the area covered by the programme (e. g. problem neighbourhoods, the most underprivileged areas) is evaluated. This situation presents exactly the same difficulties as the overall evaluation of a complex programme. Throughout this Volume, the term "in-depth evaluation" will refer to only the first two sub-divisions that produce simple situations, in which traditional evaluation tools are fully appropriate. Tools for the overall evaluation of a programme Texts concerning evaluation tools usually refer to simple programmes aimed at a well-defined public and at producing limited change within that public. By contrast, a socio-economic development programme is typically multi- sectoral and multi-objective. Its purpose is to create economic or social transformations on a scale that covers an entire territory. To achieve this goal, an integrated strategy is implemented, aimed at various publics (businesses, employees, the jobless, communities, etc. ). It has to produce a global impact by means of the right combination of multiple specific im pacts on its different In this context, evaluation encounters technical problems for which the existing literature provides few solutions. This section presents twelve tools adapted to this situation. Many of them are innovative or are employed in an innovative way. targets. Examples of the evaluation of multi-sectoral programmes The following three examples describe situations in which the programme to be evaluated was clearly multi-sectoral. The evaluations used one or more tools very successfully. In the first example, the programme provided substantial funds to a region that was lagging in its development. The programme financed basic infrastructure as well as assistance to businesses. The objectives were, however, numerous and not always precise. One of the questions answered by the team responsible for intermediate evaluation was the following: "Is the regional economic context developing in such a way that the goals will be achieved?" The evaluation made it possible to ascertain whether regional economic problems were improving or worsening. Box 11 - Intermediate evaluation of a regional programme in Spain In 1996, the Generalität de Valencia launched an intermediate evaluation of a regional socio-economic development programme (Objective 1 Operational Programme) co-financed by the European Structural Funds. The programme comprised 25 measures intended to act, together or separately, on several categories of target population. The numerous programme objectives were rarely graded and quantified. The evaluation started with a series of meetings during which the partners established a list of the ten main expected impacts. This list was obtained by means of the concept mapping of impacts technique. Each expected impact was given a name (e. g. "diversification and efficiency of energy resources"). Each of the ten impacts was associated with one or more context indicators (e. g. "number of regional firms that use natural gas"). The evaluation team then gathered statistical data to establish the overall evolution of these indicators at a regional level. The underlying evolution of each indicator was analysed by means of the before-after comparison technique. In this example, the concept mapping of impacts technique was used to "clarify and grade the effects to be evaluated" (first evaluative function), while the before-after comparison technique was used to compare data in order to show any interesting developments (fifth function). In the second example, the situation was that of a programme financing the development of a French region. Here too, interventions are multi-sectoral Principal evaluation techniques and tools and multi-objective. The team responsible for the intermediate evaluation wanted to show the spatial dimension of the outputs and results. Box 12 - Intermediate evaluation of a regional programme in France The decision to launch the Integrated Mediterranean Programmes (IMP) was taken in the 1980s. The aim was to help those regions facing strong competition due to the entry of Spain and Portugal into the European Community. The IMP of the Languedoc-Roussillon region in France consisted of several sub-programmes, oriented towards the different sectors: agriculture and fishing, industry, crafts, services, forestry and the environment, and tourism. Over half the funds mobilised were allocated to agriculture and fishing, since these sectors were hit hardest by the integration of Spain and Portugal into the European Community. An intermediate evaluation was carried out on the Languedoc-Roussillon IMP. Since the IMPs were designed to be complementary to regional policies, the evaluation had primarily to analyse the synergy between these two policy levels. This evaluation focused above all on the location and spatial integration of the different programme measures. The evaluation team gathered together all the monitoring data as well as certain statistical indicators in the database, and noted their location. A geographical Information system was used on this occasion for the first time for evaluative purposes. The evaluation team presented its results in the form of maps, some of which revealed numerous discrepancies between the location of funds and the most threatened rural areas. In this example, the geographical information system technique was used to compare data, thus producing interesting conclusions (fifth function). The third example concerns the global evaluation of Structural Fund assistance in the Republic of Ireland. The situation is that of a set of programmes covering numerous sectors and providing massive funding countrywide. The global objectives of the programme are of a macro- economic nature. There are few of them and they are very clearly defined. The team responsible for ex ante evaluation has to estimate likely macro- economic impacts. In this example, the evaluation team used a macro-economic model to estimate effects (sixth function). It should be noted, however, that the tool used includes a series of options concerning the variables to be modelled or excluded. Thus, the tool implicitly served to clarify and grade the effects to be evaluated (first function). Examples of the evaluation Box 13 - Ex ante evaluation of a set of programmes in the Republic of Ireland For the period 1989-93, all European development assistance for Ireland was co­ ordinated within a "Community Support Framework" (CSF). The European Commission and the national government jointly commissioned an ex ante evaluation of this highly complex set of aid. The evaluation had to estimate the potential impacts of the CSF on the Irish economy. An important question was to know whether a massive intervention (about 5% of the Irish GDP) would not indirectly cause a worsening of the state of public finances because of co- financing requirements. The evaluation was carried out by the Economic and Social Research Institute (ESRI) in Dublin. The evaluation team estimated the micro-economic impacts of support by drawing on research work and the evaluations of similar support mechanisms. These primary impacts were introduced into a macro-economic model, making it possible to simulate the functioning of the national economy. The ESRI had previously created this model for the European programme HERMES. The model first had to be adapted to the specific evaluation situation, particularly by breaking down its initial sectoral structure to more effectively take into account the effects of European support. The model made it possible to estimate impacts on the scale of the national economy, by taking into account numerous macro-economic mechanisms. The risk of an excessive increase in the public debt was confirmed. These three examples illustrate the use of several tools in situations in which evaluation covered an entire multi-sectoral programme. The following section will show that the evaluation toolbox is sufficiently well stocked for most of the functions of this type of evaluation to be tool(s)- based. Tools for structuring an evaluation When a socio-economic programme is complex, the structuring of the evaluation is a particularly important phase because it conditions the comprehensive and comprehensible nature of the results. Moreover, if the evaluation is being undertaken in a partnership context, this phase is essential for clarifying the evaluation criteria suited to the different partners' needs. It strongly determines the acceptability of the conclusions. SWOT analysis is generally used ex ante for defining a programme strategy. This tool is helpful in validating objectives and defining more precisely the effects to be achieved and the most appropriate interventions. It may also be used to evaluate a strategy that has already been established or even implemented. Concept mapping of impacts is used to define the effects that are to be evaluated and, in cases where there are multiple objectives and where these have not yet been firmly established, the associated indicators. This tool may also be very useful if the set of objectives lack precision. It is suited to evaluation in a partnership context because it is based on the aggregation of individual points of view for the purpose of reaching consensus between the partners. The colour vote is particularly well suited to the clarification and grading of evaluative questions in an evaluation involving several decision-makers. It also provides a fast and robust means for choosing evaluation criteria, validating conclusions and formulating recommendations. This tool is useful when there are many stakeholders and when their differing points of view have to be taken into account. together It basically consists of grouping The impact matrix is a tool that provides a simplified overview of a complex programme. the different components of the programme into between ten and twenty sub-sets (the lines of the matrix) and of clarifying the main expected impacts (the columns of the matrix). Each cell in the matrix is given a score representing the probable im­ portance of a given impact for a given component. The tool helps to ensure that all the essential effects of the programme have been considered before selecting those that are to be evaluated more thoroughly. This tool is not described in the following pages and for further information the reader is referred to Volume 4 of the Collection devoted to innovative techniques. Principal evaluation techniques and tools Some classical structuring tools are not suited to the overall evaluation of a complex programme. For example, the objectives tree and the logical framework do not allow the multiplicity of actions and their expected impacts to be adequately taken into account. Tools for structuring an evaluation SWOT Analysis (Strengths-Weaknesses- Opportunities-Threats) in a fluctuating SWOT analysis was developed 50 years ago to help firms define their strategy and restrictive competitive environment. This decision-making tool owes its name to the fact that it examines the strengths and weaknesses the within the the opportunities and firm, as well as threats of O Overall evaluation Φ Low technical complexity Φ Low cost G Θ Prospective use Use stabilised market. It is one of the classical tools of strategic analysis, like the BCG (Sos- fon Consulting Group) matrix that is similar in many ways. In the public sector, cities were the first to use SWOT analysis in the 1980s as a framework for reflection on different development scenarios. The use of this tool has recently spread to territorial diagnoses and an ex ante evaluation of regional programmes. Example: Prior appraisal of the Swedish Objective 6 plan SWOT analysis was used in drawing up the Objective 6 regional development plan in Sweden. The results produced the main environmental, structural and demographic characteristics of the context of the chosen region. The identification of strategies and priorities, as well as the formulation of the action plan, were subsequently made on the basis of major factors identified through SWOT analysis. These can be summarised as follows: this method highlighted through • Strengths: Large nature reserves, considerable natural resources. • Weaknesses: Isolated region with a low population density; vulnerability to economic fluctuations and to fluctuations of public budgets; structural weaknesses of firms. • Opportunities: Tourism, quality of forest resources, fresh-water piscicultural production; mineral prospecting. • Constraints: Population loss; lack of jobs. The recommendations from this analysis based on regional socio-economic data were judged by the managers as being relevant for their strategic planning. Their usefulness for justifying the choices made in the framework of the Regional Development Plan seemed to be recognised. Since this first attempt at the concrete use of SWOT analysis in the framework of programmes financed by the European Commission, this tool has been used in many other regions. Source: Prior appraisal of the regional development plan for the Objective 6 region of Sweden, Nordland Research Institute, Bodo, June 95 Principal evaluation techniques and tools What is the purpose of the tool? The purpose of SWOT analysis is to incorporate into the reflections on a programme, both the intrinsic characteristics of the territory concerned and the determining factors in the environment in which it takes place. The tool is intended to reduce the areas of uncertainty related to the implementation of a project or programme applicable to the relevant territory. It thus enables the definition of a more relevant strategy in the context in which the action is to take place. The aim of the tool is therefore: • to highlight the dominant and determining factors, both within and outside of the territory, likely to influence the success of the project; • to produce relevant strategic guidelines by linking the project to its environment. SWOT analysis may be expanded by means of tools similar to those known as "portfolio management", such as the BCG matrix, to examine the validity of a strategy that has been proposed or is being applied, and to recommend changes where relevant. The classification of the different possibilities takes into account their feasibility (assets available "in house" = strengths and weaknesses), as well as their potential (attractive features in relation to the outside environment = opportunities and threats). The usefulness of such tools in evaluative processes lies in their capacity for drawing a systematic ¡mage of the relationship between the evaluated programme and its direct environment. • In which cases should it be used? Developed about twenty years ago by specialists of private management, strategic decision-making tools are, once again, being used increasingly in public policy strategic reflection. Originally constructed in terms of products, customers, markets and competitive advantages, their use has now spread to public collectivities, in so far as the policies of these collectivities are aimed at creating competitive advantages on the scale of the territory for which they are responsible. The notions of strengths, weaknesses, opportunities and threats may apply to a regional economy in the framework of European competition or to a firm competing in the market. SWOT analysis serves to identify the most relevant strategic guidelines in relation to a global objective of economic development. Use of the tool is Tools for structuring an evaluation therefore more particularly recommended in the planning of a programme and during its ex ante evaluation, helping to improve the integration of the programme into its context. How should it be implemented? The implementation of a strategic approach such as SWOT analysis involves six distinct steps: Step 1. "Scan" of the environment of the programme: This operation enables the detection of the major trends and problems likely to affect the future of the territory under consideration, by means of socio-demographic, economic, political and physical indicators. Indicators of regional disparities are particularly useful for revealing opportunities and threats. This phase should not be exhaustive, for the aim is limited to obtaining an overall picture to illustrate the key issues the community in question will have to face. Step 2. Inventory of possible actions: Possible actions are formulated in general terms, and constructed in relation to the main problems identified. Step 3. External analysis of opportunities and threats: This consists of a list of the parameters of the environment which are not under the direct control of the public authorities and which, it is assumed, will strongly influence socio-economic development. Step 4. Internal analysis of strengths and weaknesses: This step completes the external analysis by making an inventory of the factors which are at least partly under the control of the public authority, and which may either promote or hinder development. Step 5. Classification of possible actions: This is aimed at highlighting those actions (strategic guidelines) most likely to reduce development problems by focusing on the strengths and reducing or even eliminating the weaknesses, with a view to maximising opportunities and minimising threats. Step 6. Evaluation of a strategy: This optional stage may be included if it is appropriate forjudging the relevance of a strategy already implemented or being planned. It is designed on the basis of a "portfolio of activities" analysis. Like a firm with its products and markets, a socio-economic programme contains a set of interventions some of which combine strengths and opportunities while others try to compensate for weaknesses Principal evaluation techniques and tools or to warn of threats. The evaluation team places interventions on a plane with two axes: (1) internal feasibility, strengths and weaknesses, and (2) external environment, opportunities and threats. The discussion of the map thus produced can be used to judge the relevance of the evaluated strategy. Application to the Structural Funds Strategic analysis is suited to the clarification of issues relating to the socio­ economic development of those regions covered by the Structural Funds. It is recommended for the planning of regional development strategies that have to be centred on strengths and weaknesses, while trying to overcome weaknesses and threats. A tool such as SWOT analysis proposes a starting point for defining the strategic guidelines to be applied in a specific environment. SWOT analysis may therefore serve as a management tool for assessing the relevance of a strategy in the planning or implementation stage. In the framework of ongoing evaluation, this tool may be used to check whether a strategy remains relevant to the needs of the regional economy. Strengths and limits of use • SWOT analysis applied to public action is oriented towards the search for an effective strategy. • The matrix tool is derived from management theories that are already old and are often criticised for being simplistic. This limitation should be kept in mind in its transposition to public management, so as to avoid restricting the analysis to a simplistic framework. • SWOT analysis, as an evaluation tool, serves to make a simple classification of activities in terms of their relevance. Its main weakness derives from the often-subjective procedure used by the evaluation team to classify the activities. The association of partners in this classification is a way of enhancing the credibility and usefulness of the analysis. • SWOT analysis requires a deliberate intention, on the part of the different actors participating in its application, to reach a stable consensus. The process of formulating strategic guidelines is only of value under this condition. Otherwise, this model may tend to produce an erroneous and/or inapplicable diagnosis. Tools for structuring an evaluation Further reading Bryson Ü. M. and Roaring W. D. (1987) "Applying Private Sector Strategic Planning in the Public Sector", Journal of the American Planning Association, n°53, pp. 9-22. Examination of the application of corporate strategic planning methods to contingencies in the public sector. Quinn, J. B. & al (1988) The Strategy Process, New Jersey: Prentice hall. Critical study of strategic planning. Tools for structuring an evaluation C o n c e pt m a p p i ng of impacts The concept mapping of impacts is an adaptation of the technique known as "concept mapping", used over the past ten years for programme evaluation. It was first used to help to define strategies or the reorganisation of firms, before being applied to public policy making, in particular, as regards social programmes. O Overall evaluation O 0 Medium technical complexity O Φ Medium cost © Retrospective use Prospective use Innovative use Data processing software is used to obtain a graphic ¡mage of the actors' representations of a socio­economic reality, in general, and of expected impacts, in particular. The MEANS programme developed an application specially designed for the clarification of the impacts of a regional development programme and to find associated indicators within a partnership framework (see Volume 4 in this Collection, devoted to innovative techniques). Example: Clarification of expected impacts of the ERDF 1994­99 Operational Programme of the Generalität de Valencia. This example corresponds with that of Box 11. The concept mapping of impacts used in the intermediate evaluation of the ERDF 1994­99 O. P. of the Generalität de Valencia had two objectives: • clarification of specific and global impacts of the programme, in order to determine criteria for judging the effectiveness of interventions for intermediate evaluation; • proposal of intermediate and global impact indicators in order to complete the battery of programme indicators. The evaluation took place in two phases focusing first on intermediate impacts and then on global impacts, with the assistance of the managers of programme measures and of the overall programme managers, respectively. Officials of the Generalität in charge of the implementation of measures within the programme were divided into three groups corresponding to the fields of intervention (direct support to businesses, indirect support to businesses, infrastructure). At the end of a series of three meetings, each of the groups determined specific evaluation criteria for actions in the field of intervention concerned. The same groups then determined the intermediate impact indicators most likely to provide the information Principal evaluation techniques and tools required to make a judgement in terms of the specific evaluation criteria selected. In a second phase and by means of the same process, the overall programme managers (representatives of the European Commission, of the central Spanish administration and of the regional authorities) constructed global evaluation criteria for the programme. They drew upon the results of the groups' work. The process ended with the selection of the corresponding global impact indicators. Source: Programme MEANS 1996 - Instituto Valenciano de Económicas, "Projet pilote de cartographie conceptuelle des impacts" Investigationes What is the purpose of the tool? The concept mapping of impacts is used to assist in the clarification of impacts expected from a public intervention. The clarification/grading is obtained by taking into account the representations of all the stakeholders. The group's work is facilitated by the systematic nature of the approach that avoids fruitless digressions, and by the visualisation of the similarity or difference between impacts. It is thus possible to conduct collective reflections in large heterogeneous groups (up to 200 persons) and to structure and grade the main impacts expected from an intervention. The tool makes it possible, in an ex ante context, to determine the main objec­ tives and priorities. In an ex post context, the impact mapping technique is used to draw up the evaluation reference system (effectiveness criteria and impact indicators). In which cases can it be used? This tool is valuable for structuring the content of a complex programme. It was used, for example, in a neighbourhood in Northern Ireland to define a social programme in partnership with the Protestant and Catholic communities concerned. It is useful in evaluation when the evaluability of a programme has to be enhanced, that is to say, when it is necessary to make the objectives of a programme more explicit. This technique is particularly relevant when the evaluability of a programme has to be enhanced in the partnership framework involving several decision makers/funding bodies. It is thus possible to produce a common evaluation reference system, despite differences in objectives that may exist between these partners. Two characteristics frequently found in programmes can thus motivate the use of impact mapping: the lack of clarity and precision of the programme objectives and of its expected impacts, and the lack of impact indicators Tools for structuring an evaluation in the programme. Since planned is relatively cumbersome, it is advisable to limit the use of impact mapping to complex programmes with many expected impacts. implementation its How should it be implemented? Step 1. Constitution of a group of stakeholders in the programme. This step is essential because it will determine the framework in which the subsequent steps in the process are to take place. The group must be balanced (in terms of the different actors in the programme), knowledgeable (in-depth knowledge of the intervention by the members of the group) and stable (full participation continues throughout the process as a whole). Step 2. Inventory of expected programme impacts in the form of descriptors. This step consists of producing an exhaustive list of all the impacts expected from the programme, in the form of short qualitative statements called "impact descriptors" (e. g. "actors in tourism became more professional"). This list must include any unfavourable impacts of the programme which were not expected but could have been foreseen. This list draws upon two sources: • identification by the evaluators of all the impact descriptors featured in the programming documents; • impact descriptors formulated by members of the group in relation to their knowledge of interventions. These concern the impacts of the measures or actions of programmes, which each member of the group expects to observe in the short, medium or long term. Formulation of these descriptors may take place during interviews or in the context of a first group meeting. A group meeting is essential for validating the descriptors and determining the final list together. Step 3. Weighting of descriptors and constitution of piles. In the context of a second group meeting, each member is given a recapitulative list of the descriptors, as well as a set of cards (with a descriptor marked on each one). Each member of the group is asked, individually: • to weight the different descriptors, that is to say, to give a score to the descriptors in terms of their strategic importance for the programme. The more important the role of the described impact in the success of the programme, the higher the score. Principal evaluation techniques and tools • to constitute piles of descriptors in terms of their conceptual proximity; each member of the group will thus group together conceptually similar descriptors, that is to say, those in the same "family" of impacts. Step 4. Calculation of the impact map. The data produced by each member of the group (weightings and piles) will be analysed statistically by an appropriate software package. At the end of a non-metric positioning, multidimensional analysis the descriptors are represented in the form of a cloud of points in a two- dimensional space. In this cloud of points, the descriptors that were often placed in the same pile by the participants will be positioned close together. Those descriptors which were never put in the same pile by the participants will be furthest apart. At the end of a second analysis consisting of a hierarchical classification in clusters, the software will identify groups of descriptors. Those descriptors that were most often grouped together by the different members of the group will be put into the same cluster. Groups of descriptors that are similar from a conceptual point of view are thus constituted. Tools for structuring an evaluation Finally, the average weighting of descriptors included in each of the groups is calculated. The final result is a map showing weighted groups of descriptors (in the diagram in Box 14 the weight is represented by a thicker line). Step 5. Definition of effectiveness criteria. The impact map is examined in a third meeting when the composition of the different groups of descriptors is discussed. Some descriptors can be moved to make its composition more coherent. Thus, the map calculated by the software is not a final result but rather a basis for discussion. Once the composition of the families has been validated, they are named. The aim is for the members of the group to agree on a name for each family, which is representative of all the descriptors in the family. This name corresponds to an evaluation criterion. Thus, there will be as many evaluation criteria as families of impact descriptors. Since the families are weighted, the criteria obtained are also weighted. This allows these criteria to be used directly for a multicriteria judgement. Group reflection may be continued with the aim of specifying a time-scale for the impact families (impacts which are expected in the short, medium or long term), and the presumed causal links between these impact families (e. g. a particular impact family constitutes an intermediate impact which will help to achieve a more global objective). This process may finally be taken further with the aim of proposing impact indicators directly related to the different evaluation criteria. The descriptors included in each of the families are helpful in defining these indicators. Thus, some of these descriptors, and particularly those to which the most substantial weightings were given, may directly inspire the definition of an indicator. Discussion by the group on each of the proposed indicators will relate both to the relevance of the indicator in terms of its relation to the evaluation criterion, and the availability of sources of information. Application to the Structural Funds The concept mapping of impacts has proved to be particularly well adapted to the context of structural programmes. The aims of this type of programme are defined by the European Union and adapted to suit to the characteristics of each Member State or beneficiary region. Since the Single Programming Docu­ ments are the product of negotiation, the programme objectives are frequently not specific enough for constructing a reliable evaluation. Impact mapping is Principal evaluation techniques and tools consequently useful in the first stage of clarification. It helps to clarify evaluation questions, grade effects and define evaluation criteria in a partnership context, which facilitates the appropriation of the evaluation results. The impact map is useful in the preparation of intermediate programme evaluation. It makes it possible to systematically establish impact families cutting across several measures, and thus to construct relevant criteria for the evaluation of the programme as a whole. The procedure can be divided into two phases, as in the case of Valencia, which are summarised in the following diagram. Box 15- Partnership process of impact mapping Operators Group A Operators Group Β Operators Group C PHASE 1 ■ ' 1 Map of specific impacts A ' Map of specific impacts Β Map of specific impacts C PHASE 2 ■ ' Group of managers 1 Map of global impacts Strengths and limits in practice • Using a computer programme to mediate is a valuable support for reaching agreement, despite the heterogeneity of initial points of view. Tools for structuring an evaluation • The group dynamics generated by the technique has proved to be useful and positive for increasing mutual knowledge of different management actions, and may thus favour adjustments to the implementation of pro­ grammes. By mobilising the stakeholders in the preparatory phase of evaluation work, it facilitates the subsequent stages of the evaluation and makes the results far more readily acceptable. • Impact mapping is a technique that is used to enhance the evaluability of programmes. It makes it possible to produce a common evaluation reference system, taking into account the points of view of the different partners. The successive group discussions, which are part of the implementation of this tool, guarantee the legitimacy of the reference system obtained. • The results may be produced in a few days. The technique is thus compatible with the evaluation cycle. • The resources required (even if they are limited) and the mobilisation of numerous stakeholders during successive meetings can be justified only when the evaluation stakes warrant such an investment. • The constitution of the group of partners can be difficult and care has to be taken to ensure that there is a sound balance between the stakeholders. Considerable efforts may be required in cases of conflict. • Although the technique of drawing up impact maps limits risks related to the leadership of a few members of the group, this risk remains during the phases of production of descriptors and exploitation of the impact map. • This tool can be used for all the functions of the structuring stage. Further reading Knox, C. (1995) 'Concept Mapping in Policy Evaluation: A Research Review of Community Relations in Northern Ireland', in: Evaluation, Vol. 1, Number 1. An interesting example of use of the tool in a European context. Trochim, W. M. K. (1989) An introduction to concept mapping for planning and evaluation', in: Evaluation and Program Planning, 12 (1): 1-16. Presentation of the tool and its advantages for evaluation. In the MEANS Collection: Volume 4 devoted to innovative techniques. Tools for structuring an evaluation Colour vote In the field of evaluation, group work is often essential, particularly when several partners are involved in commissioning and steering an evaluation. The different voting techniques can be used to structure collective reflection, when the aim is to improve effectiveness, stimulate creativeness or define a common standpoint. While there is a large number of these techniques, this information sheet O Overall evaluation A Θ φ Low Low technical cost complexity O o In­depth evaluation Retrospective Prospective use use © Innovative use focuses more particularly on the "Abaque de Régnier", or colour vote. Example: Evaluation of an economic development programme in Switzerland As part of a study of the measurement of the effects of public actions in Switzerland, the CEAT (research community for national development) evaluated a canton policy for economic development. Two work groups were formed with the actors involved in the implementation of the policy. The first group was to focus on the "objectives and strategies" of the policy and the second group on its "implementation and co­ ordination". The evaluation team used the Abaque de Régnier as a technique for conducting group sessions. The groups based their work on assertions formulated by the evaluation team. These were often provocative, the intention being to encourage the participants to take a stand. By asking the participants to vote on these assertions, the technique created a starting point from which to express their positions and to become aware of the other opinions present in the group. The debate helped to define a common position on the objectives of the policy of economic development. Source: Alberton S. , Rey M. et L. Vodoz (1993) La politique valaisanne de promotion économique: une évaluation privilégiant le point de vue des acteurs. Berne: Programme National de Recherche 27. W h at is the purpose of t he tool? Voting techniques facilitate the expression of opinions on a given subject by all the participants. It is the way in which these opinions are obtained that varies. The colour vote is based on the idea of making participants aware Principal evaluation techniques and tools of all opinions in the group, by means of a coloured visual presentation of the results of individual votes. In this way the strong personality of some does not conceal the opinions of other, more reserved, participants. With the colour vote, issues on which there is consensus are soon distinguished from conflictual issues. In this way time is saved by allowing the group's reflections to be concentrated on most problematical issues. Voting techniques also help to avoid certain drawbacks of group work such as pressure to conform, convergence of opinions around core values, predominance of certain actors, etc. In which cases should it be used? A work group may use these techniques for any type of evaluation: ex ante to define challenges or needs to be met, ongoing to clarify objectives as in the example above and, where relevant, ex post to validate evaluation conclusions. Very often the objectives and judgement criteria of a programme cannot be legitimately clarified by a single person. In this case the voting techniques facilitate the work of a group composed of people with very different skills and levels of responsibility. In other words, meetings for managing the performance of an evaluation are an ideal opportunity to apply voting techniques. From this point of view, the colour vote is particularly useful for grading programme objectives, defining evaluation criteria and selecting the most important criteria. How should it be implemented? Step 1. Formulation of statements. The statements are assertions expressed concisely, generally in a single sentence. They are formulated beforehand by the evaluation team, on the basis of personal interviews and of an initial group discussion, during breaks in the meeting. In the above example the items were formulated as follows: • "For industrial enterprises, financial support is derisory" • "For industry, economic promotion should be aimed at 'high-tech' firms" • "Economic promotion should benefit all areas of the canton equally" • etc. At this stage, it is useful to collectively check the understanding of each statement. Tools for structuring an evaluation Step 2. Individual vote: the participants give a colour to each statement. The signification of the colours is simple: Red Pink Yellow Total disagree­ ment General disagree­ ment No clear- cut opinion on the subject Light green Dark green General agreement Total agreement White Black No opinion on the subject Does not want to participate ¡n the vote Step 3. Transcription of votes on a magnetic board or on a computer screen. The result is thus immediately presented to the participants in the form of a table filled with coloured disks. Tendencies are clearly visible and this enables each person to know whether consensus exists or not in the group. Step 4. Group discussion. The group leader selects statements for discussion. Most often, he or she will choose those on which there was disagreement. Step 5. Reformulation of statements with the group. This final step is used to define a collective position on the subject. Application to the Structural Funds The Structural Funds are managed within a partnership framework, which means that numerous working meetings are organised. Apart from those of an administrative nature, all meetings with a "creative" aim can be based on voting techniques to structure and motivate the work (meetings devoted to the selection of projects, the evaluation, reorganisation or renewal of programmes, etc. ). The colour vote is particularly well suited to situations in which a common standpoint or consensus are required. It is also used to grade proposals made in a group, which can make it easier to reach conclusions in a meeting. Strengths and limits in practice • Groups appreciate the colour vote when they want to reach consensus in a relatively conflictual domain. The transparency and simplicity of the technique ensure a high level of confidence in the process. Principal evaluation techniques and tools • The results of this vote are immediately visible and, thanks to the technique, tendencies are easily perceived. The possibility of voting "whi­ te" or "black" helps to highlight a lack of information or points of disagreement on the subject under discussion. • The facilitator has a high level of responsibility; he or she therefore requires extensive experience. • The fact that the tool is not of a highly technical nature enhances the participants' confidence (NB: the most recent version of the Abaque de Régnier is computerised. ) The game-like nature of the method also has the advantage of relaxing the atmosphere during meetings. There is, however, a risk of rejection if the participants find the technique too scholastic. • Potential predominance of one actor in the debates is limited by the non­ verbal nature of the tool. Further reading Anzieu D. et Martin J. -Y. (1994) La dynamique des groupes restreints. Paris: PUF. 395 p. Reference manual for the functioning of small groups. Annex containing various methods of "discussion-meetings". Mucchielli R. (1992) La conduite des réunions. Paris: Librairies techniques. 120 p. Popularised presentation containing practical exercises. Viveret P. (1992) 'Les outils d'intelligence collective et l'évaluation', in: L'évaluation en développement. Paris: La Documentation Française, pp. 179-183. Summarising group techniques. Tools for observing changes in the field When a programme is complex and has to be evaluated in its entirety, the field of observation to be covered is very broad. Exhaustive observation is therefore extremely costly. Tools do, however, exist for collecting the data needed for evaluation. Observation in the field can be carried out by organising a series of individual interviews with operators (indirect observation) or with the addressees of the programme (direct observation). This technique is particularly appropriate when the information to be collected is factual. Focus groups , with operators or addressees as participants, are used to collect both opinions and data. A well-conducted focus group allows reliable, high quality data to be collected. Systematic field observation can be carried out by means of carefully selected case studies. In the evaluation of a complex programme, the case study seems to be the only appropriate tool for observing effects on direct or indirect addressees, while respecting the complexity of the different programme components. Some classical observation techniques are difficult to apply in the overall evaluation of complex programmes. For example, the questionnaire survey of direct addressees of the programme is too expensive if, as is often the case, there are up to ten different categories of addressees. Tools for observing changes in the field Individual interview The interview technique is used to gather qualitative information and the opinions of those persons affected by a particular programme or project, its context, implemen­ tation, results and impact. Several f o r ms of interview can be distinguished, each of which fulfils a different purpose: the informal conversation interview; the semi- structured, guide-based interview; and the structured interview (the most rigid approach). O Overall evaluation A £3 <D Medium technical complexity Medium cost O G In-depth evaluation Retrospective Prospective use use Θ Use stabilised The following presentation is based on a semi-structured interview, the technique that is used most frequently in the evaluation of Structural Funds. Example: Evaluation of the North Jutland Technology (NordTek) programme The goal of the NordTek programme was the development of North Jutland through the introduction of information technologies into the regional industrial fabric. The programme had two aims: job creation and the improvement of a negative image as a peripheral region. To this end, the development strategy implemented by this programme was based on four objectives that were to be achieved through three types of action. lastly, developing attractiveness for foreign The objectives were: reinforcing local networks of actors; improving the qualification of workers in new technologies; orienting regional production towards local demand; and, investments. The actions implemented for that purpose were measures favouring the knowledge and utilisation of new technologies, measures for training people in the use of new technologies, and measures providing investment support for SMEs. The programme was evaluated by means of semi-structured interviews among 70 beneficiary firms. The goal of these surveys was to understand how firms used the services received and what they got out of them. Those firms in which interviews were held were selected from 250 beneficiary firms in five of the most important projects in the NordTek programme (the different projects comprising the programme concerned specific subjects such as consultancy or training and networking, for example). The firms selected had to have received a relatively large amount of assistance during at least six months, and to be representative of the different forms of assistance that had been granted. Principal evaluation techniques and tools The approach chosen was a method using an interview guide, so as to gather all the information concerning the firms. The interview guide also made the approach more rigorous, thus facilitating the comparison of the interview results obtained by several different interviewers. After the interview, the survey continued with a more informal conversation during a visit to the firm. It lasted a total of about two hours. The interviewee then received the summary of the interview in the form of a 4-8-page document, for validation. The themes addressed during the interview were, for example: the reasons for joining the project, the expected benefits from the assistance and the benefits actually received, and the possibilities of obtaining assistance from a different source. The results of the interviews made it possible to identify the reasons behind the firms' decision to participate in the NordTek projects. On the whole, the interviews revealed that the assistance provided by NordTek was unique in certain cases and could not be found elsewhere, whereas for certain aspects it competed with local companies and even other NordTek projects. The NordTek networking and consultancy services were developed over a period of several months, and even several years; personal knowledge and geographical proximity were then seen as important factors for the firms. It nevertheless remained difficult to quantify the benefits that the NordTek projects were able to provide to firms, due to the numerous external factors capable of influencing the process in which the services were provided. Source: Olsen, Leif and Olaf Rieper. NordTek-evaluieringen, AKF Forlaget, 1991 W h at is t he p u r p o se of t he tool? This type of survey is a way of learning about and examining the views of the actors (addressees and other stakeholders) on a project or programme, e. g. how far the project or programme meets needs, or its results as c o m p a r ed to expectations. Interviews are also used in social science as a tool for investigating users' reasoning. The individual interview is an exploratory technique that serves to obtain relevant information on the reasoning, conceptions and representations of the persons questioned on a project or programme. Apart from subjective representations, it also serves to gather information on individual practices. It is particularly valuable in exploring the ways in which an intervention has been implemented and for identifying g o od practice. It is therefore often used as part of a formative evaluation that is designed to test a theory of action and/or to provide guidance about fine-tuning a policy and program­ me. It may also be helpful in providing a summative evaluation of a Tools for observing changes in the field programme which is intended to achieve changes in actors' behaviour or perceptions (such as technology transfer or training programmes) as opposed to the more tangible, 'harder' outcomes (such as creating new jobs or constructing new facilities). In which cases can it be used? The interview is used in an exploratory context, in other words, when one does not have a priori hypotheses or adequate knowledge on a project or a target public to make a questionnaire survey possible. It is also a relevant technique when the public concerned by the evaluated programme or project is too small to be the subject of a statistically representative survey. Semi-structured, guide-based relatively standardised information is required about the programme being studied. A certain amount of prior knowledge about the subject is needed when using this technique. interviews are used when The individual interview technique may be used at any stage of the evaluation process: to establish a theory on the evaluated programme or identify the problems and needs of a region and the improvements needed during the implementation of the programme. It may be particularly powerful when used to gain the views and understandings of key actors such as senior policy makers or programme managers. • How should it be implemented? In order to conduct an interview well, the following steps have to be adhered to: Step 1. Selection of interviewees: The samples needed for carrying out interviews are smaller than for questionnaire surveys. The information obtained is validated by the context and not by the probability of occurrence peculiar to questionnaires. The number of interviews depends on the subject of the study, on the variety of reactions to the subject, and on the resources available (in general 20 to 60 interviews). The selected sample is most often based on the selection of components said to be characteristic of the population (diversified sample). The sample may be selected either by direct access (e. g. via administrative lists) or through the intervention of a third party, which allows for more targeted selection but presents risks of distortion. Principal evaluation techniques and tools Step 2. Planning the interview: Planning the interview includes the drawing up of an interview guide. This consists of specifying the topics that the interviewer wants to address. It is not essential to follow the interview guide in any precise order. This guide is more a sort of checklist enabling the interviewer to check that he or she is dealing with the essential questions. The interviewer may modulate her/his intervention in relation to the interviewee, and formulate new questions. The first interviews may give rise to adjustments or amendments in the definition of the questions if the interviewees have a problem with them. In many cases it is also helpful to gather basic information from interviewees in advance to save time during the interview itself. It may also be useful to provide the interviewee with a guide to the issues that are to be covered so that he or she can gather together any necessary factual information prior to the meeting. Certainly it is usually worth establishing a 'contract' with the interviewee explaining the purpose of the interview, how long it will last, the level of confidentiality, the use made of findings etc. Step 3. Selection and training of interviewers: If they are to be conducted properly, interviews require a high degree of professionalism in the interviewer. He or she must have skills in communicating, listening and note taking. To facilitate the smooth running of the interview and ensure that the interviewee feels at ease, it may be useful to ensure that there is social proximity between interviewer and interviewee. The least structured inter­ views also require the interviewer to have substantial knowledge of the field. Step 4. Course of the interview: An interviewer must have a "respectful" attitude vis-à-vis the interviewee and the information gathered, but also be able to convey a good understanding of the subject matter and context. The initial contact is very important, for it is the basis of communication. The interviewer must be careful not to influence the interviewee by approving or orienting her/his answers. The interview may be recorded, to ensure that the interviewee's assertions are not distorted and that the most important remarks are not omitted from the report. However, in some cases interviewees may feel less able to give full and frank answers on tape, and the costs of transcribing interviews should be taken in to account in designing the evaluation budget and timetable. The interview transcript or summary should be checked with the interviewees and it may be useful at this stage to follow up any unresolved questions. It may also be useful to thank interviewees by letter for their contributions (their co-operation may be needed in future studies and it is therefore worthwhile ensuring their goodwill). Tools for observing changes in the field Step 5. Analysis of results: This final phase consists of analysing the conversations, interpreting and comparing the information given by the interviewees, and finding common and divergent viewpoints so as to draw up a review of the evaluation. A summary focused on the coherence of each interview, or a transversal thematic analysis more suited to the search for models capable of explaining individual practices, may be drawn up. In order to produce results and then discuss them, the interviewer establishes an analysis grid based on the reading of the interviews and of descriptive hypotheses. This grid is an explanatory tool applied to each interview report. Application to the Structural Funds Interview techniques are used extensively in the evaluation of structural interventions and, in particular, for the evaluation of programmes. Interviews with programme managers and addressees remain one of the most commonly used tools in the intermediate evaluation of structural programmes. The principle consists of selecting several addressees or managers, depending on the characteristics of the action implemented and the public concerned. Used as a tool for analysis, the interview is a technique which makes it possible to make a brief overview of the programmes. Very loosely structured interviews - informal conversations with the managers and other persons concerned by the programme - may be used to identify those parts of the programme which need to be considered in more depth. Interviews with policy makers may also form an important part of the early stage of focusing an evaluation and determining the key outputs which policy makers require. The interviews may also prove relevant and provide useful information in the framework of programmes of a social nature where the addressees often lack motivation for filling in the questionnaires, such as measures aimed at giving jobs to the long-term unemployed who have been on a training course. Strengths and limits in practice • The advantage of this type of survey is that it provides in-depth information on the values, facts and behaviour of the interviewees; it Principal evaluation techniques and tools makes it possible to link up a group of elements, thus producing a relatively exhaustive study on a given subject. • An individual interview takes into account situational and individual factors. It is therefore difficult to draw general conclusions. Individual interviews may allow for an exhaustive identification of effects and possible causes, but cannot be used to measure impacts or grade causes. • A well conducted interview may provide insight into the mechanisms of implementation and the causal links peculiar to a programme, and help to identify success stories or obvious shortcomings. In that case, the technique produces information that can easily be communicated in the form of boxed examples in a text. • If it is to be effective, this practice requires a lot of time and the contribution of professionals. Specific skills are needed to plan, conduct and interpret an interview; inadequate skills of this nature will produce information of no value. Interviews are not always conducted with the professionalism required to produce effective results. • The benefit of these surveys depends on the knowledge of the interviewees and on their co-operation in answering the questions. Information may also be distorted due to the choice of interviewees. Further reading Blanchet Α. , Gotman Α. (1992) L'enquête et ses méthodes: l'entretien. Nathan Université This book is entirely devoted to interview surveys, from an educative point of view. Casley D. , Kumar K. (1991) Collecte, analyse et emploi des données de suivi et d'évaluation. Banque Mondiale Clear and accessible presentation of interview techniques - Ch. 2 and 3, p p. 1 2 - 47 Patton M. Q. (1987) How to use qualitative methods in Evaluation, Sage Publications This book devoted to all the qualitative techniques gives a glimpse of their application in the field of evaluation. Tools for observing changes in the field F o c us G r o up A focus group involves several people with the same characte­ ristics, and provides qualitative information during a targeted discussion. The focus group technique was devised by marketing specialists to analyse the potential impact of advertising strategies and advertisements. The method solicits the participants' creativity carrying out simulations. It in is O Overall evaluation e m High technical complexity Medium cost A O o In-depth evaluation Retrospective Prospective use use © Innovative use particularly valuable for analysing themes or fields which give rise to divergent opinions or which involve complex issues that need to be explored in depth. Example: Thematic evaluation of the Objective 3 SPD, 1994-99, UK In the framework of an evaluation of the Objective 3 SPD in the UK, a focus group technique was used to complement other techniques. The general objectives of this evaluation were to check: a) whether the actions implemented within the SPD had effectively reached the targeted addressees and b) whether the structure of the SPD favoured the emergence of effective projects in terms of the integration of addressees into the job market. The method devised for this evaluation had a quantitative section based on the exploitation of monitoring data, and a qualitative section based, in particular, on four focus groups formed in four regions representative of the diversity of situations in the UK: Scotland, the "West Midlands", London and the Southwest. These focus groups consisted of representatives of project promoters (notably training centres, regional councils, higher education institutions, non-governmental organisations, and Training and Enterprise Councils). The subjects and questions addressed were structured as follows: • examination of the sources of information available for planning projects corresponding to the needs of targeted publics as well as for choosing the means to reach these publics; • highlighting the strengths and weaknesses of the modes of selection, approval and management - especially financial - of projects, and their effectiveness in terms of targeting; • discussion on the means to measure and assess the results of these actions in relation to the specificity of the publics targeted and employers' requests which have been debated; Principal evaluation techniques and tools • finally, debate on the effectiveness of the SPD as an instrument for promoting projects suited to the addressees' needs. These discussions served to reveal new ideas and considerations, often covering themes closely related to the questions asked, such as the existence and advantages of "integrated projects", the modes of functioning in networks of project promoters, the "additionality" of projects, etc. The results drawn from these focus groups, although not representative in the statistical sense of the term, served to highlight "ideas to work on" in various fields such as: access to problem publics, conditions facilitating the emergence of relevant projects in terms of targeting, and so on. Source: Targeting and the ESF Objective 3 1994-1996 Single Programming Document (SPD), final report to the Department for Education and Employment (DfEE) ECOTEC, June 1996 What is the purpose of the tool? Structured discussions in focus groups are starting to be used extensively today, especially in the social sciences and particularly for evaluation. By playing on the interaction and confrontation of points of view, this techni­ que serves to reveal the participants' perceptions on a given theme, whether its implementation, results or impacts. it concerns the relevance of the evaluated programme, The focus group technique is a particular application of techniques for chairing meetings. The tool is standardised, with a view to creating optimum interaction for the collection of information. Other group techniques are used in the social sciences, such as the Delphi technique or the nominal group (recording of all individual points of view which are then submitted to the group and discussed), that do not feature truly interactive discussion. The focus group makes it possible to bring together, simultaneously or sequentially, several stakeholders in a programme (managers, addressees and/or operators) and to collect a large amount of qualitative information in a relatively short space of time, by means of the "shared intelligence" technique. The comparison of the experiences and representations of the participants in the group make it easier to understand the phenomenon under observation. By involving the actors or addressees of a programme in the evaluation work, the conclusions of the study will be more credible and more easily accepted. The focus group technique may also be used for the validation of data collection, or for completing quantitative data. Tools for observing changes in the field In which cases can it be used? Focus groups are adapted to those cases where the topics to be addressed provoke divergent opinions but where discussion remains possible. The subjects dealt with will be in the "public interest", so as to be debatable in a group. How should it be implemented? For the same reasons as those applying to individual interviews, several steps are required to prepare and conduct focus groups properly: Step 1. Selection of participants: The choice of participants in the group depends on the characteristics of the population and the subject addressed. It is preferable to form several types of groups (between 3 and 5) in the selected categories. Limiting the work to a single group may undermine the validity of the study. The group dynamic could, for example, be distorted due to the heterogeneous composition of the sample. The participants will be selected so as to ensure that there is a degree of homogeneity in the group. The number of participants generally ranges from 6 to 10 per group, so that each person has a turn to speak, and so that sub-groups are not formed. Usually the participants do not know one another. The participants are sometimes remunerated. However, rather than offering an incentive remuneration, it is preferable to offer to refund their travel expenses, where relevant, and to serve refreshments at the end of the session. Step 2. Designing the interview guide: Given the time limits and the number of persons, it is important to carefully define and limit the topics addressed. The guides for each group will be different. The group interview guide must be short because more time is necessary when several persons participate in the discussion and answer the same questions. Most of the questions asked are open; they must also be carefully defined and arranged in a series with the most general ones first. Step 3. Choice and training of facilitators: The facilitator must prove her/his creativeness and motivation, and have a good relationship with the participants. In their training facilitators must learn which attitudes to have vis-à-vis the group (e. g. respect for people's ideas) and what knowledge to the objectives of the study and data analysis absorb concerning techniques. Facilitators must also have communication skills, note-taking Principal evaluation techniques and tools skills, etc. It is useful to make provision for a second person per group, particularly if the session is not tape recorded, so that one person can take notes while the other leads and facilitates the discussion. Co-facilitators may observe the discussion and then make recommendations to the facilitator on the way the meeting was conducted. When this technique is being used to test advertising or products, the co-facilitators usually observe the group from another room by means of a two-way mirror or video link. In the case of scientific study, the use of this approach without the knowledge of the participants poses a number of ethical problems and is not therefore recommended. Step 4. Course of the discussion: The discussion may be launched fairly openly, by introducing the subject of the session and asking a simple question of general Interest. This will enable each participant to give an initial opinion or a remark on the subject. The interview guide serves to remind the facilitator of all the essential points that need to be addressed. If all the advantages of the interview are to be maintained, there must be a written report. It is strongly recommended to record or even to film the interview, although the latter alternative is often seen as an "intrusion" by the participant. Some rooms are designed in such a way that the participants do not notice they are being recorded or filmed. Step 5. Analysis and report on results: This final phase consists of interpreting and comparing the information given by the participants, and looking for shared and divergent opinions within each group. The informa­ tion collected is codified so as to organise the results in relation to the objectives of the survey. The interpretation of data must take into account and distinguish two major aspects of the discussion: what the participants consider as interesting, and what they judge as being important. The analysis will depend on the number of focus groups questioned, and on the nature of the interviews (for example, did the focus group discussion have a structured approach or not?). The results from the different groups are compared so as to identify any convergence there may be. The report may quote the most noteworthy statements made by the participants, together with a summary of the discussions. Application to the Structural Funds The focus group technique may be used to test an innovative measure (ex ante evaluation), clarify the objectives of a project, establish a theory of Tools for observing changes in the field region and action for the programme being evaluated, and identify the problems and the needs of a implementation of the programme. This technique may also be used at the end of the programme in the framework of an ex post evaluation, to collect information for identifying and/or interpreting the effects and impacts of the programme concerned, and to fix new priorities and orientations. required during improvements the Strengths and limits in practice • This type of survey provides in-depth information on the values and opinions of the interviewees; • The fact of bringing several persons together provides a certain balance in the answers given and makes it easier for the evaluation team to define the general opinion on a particular programme. Owing to the participation of several persons, the focus groups provide a certain "quality control" over data collection, by judging the "pros and cons" of each person's arguments and thus avoiding extreme opinions. • In a short space of time (from one and a half to two hours), it is possible to collect a large amount of qualitative information. • Specific skills are required for managing the group dynamic and obtaining a balanced discussion while avoiding the dominant influence of opinion "facilitators", for example and the technique of time and resource intensive. There is also a risk of "polarisation", that is to say, that at the end of each group discussion the participants may be influenced by the majority opinion expressed at the beginning of the discussion. • The discussions may sometimes be biased, due to the fact that the participants (addressees) of public policies are subject to an effect of dependency and will produce a positive judgement a priori. • The number of people does, however, limit the number of questions due to the time required to answer them. Moreover, conflicts may emerge between the different persons, which may undermine the quality and effectiveness of the interview. • It is possible that participation in a focus group changes peoples' perceptions - either because of the 'Hawthorne effect' (the fact that the behaviour of persons who know themselves to be under observation changes) or because their interaction with other participants gives them new insights and perspectives. Thus for example programme managers Principal evaluation techniques and tools may actually improve their performance as a result of participating in a focus group. In this way the focus group methodology may therefore have an impact on the programme being evaluated or on successor programmes. Further reading Anzieu D, Martin J. Y. , (1994) "La dynamique des groupes restreints", PUF Le psychologue Complete presentation, by psychosociologists, of the concept of a limited group, the main group phenomena and some fields of application. Aubel J. , (1992) "Guide pour des études utilisant les discussions de groupe", BIT Practical guide to conducting group interviews. Greenbaum, TL. , (1998) 'The Handbook for Focus Group Research' (Second edition). London : Sage Publications ; Krueger, R. A. , (1988), 'Focus Group : A Practical Guide for Applied Research'. London : Sage Publications. Moore M. C, (1994) "Group techniques for idea building", Sage A more detailed approach on group techniques, presenting specific techniques (Nominal Group Technique, Interpretive Structural Modeling, etc. ) Ideawriting, Morgan, D. L. , (1997) 'Focus Groups as Qualitative Research' in Qualitative Research Methods Series, 16. London : Sage Publications. Tools for observing changes in the field Case study This tool is based on an in-depth study of the data collected on specific cases. It may c o n c e rn individuals, programmes, organi­ sations, projects, groups of people or decision-making processes. The a p p r o a ch uses a range of qualitative and quantitative data collection techniques. The case study is r e c o m m e n d ed for O Overall evaluation A In-depth evaluation 0 Medium technical complexity O φ Medium cost Θ Retrospective use Use stabilised evaluations aimed at a detailed analysis of c o m p l ex processes of implementation of programmes and the representation of their impacts. The case study generally applies to situations that require an exploratory field study due to the lack of available information. Example : Evaluation of the NordTek programme, Denmark This programme, intended for small and medium-sized firms, consisted of about 100 projects offering services to firms in the areas of training, consultancy, access to modern computer equipment, etc. It was therefore a programme that could be qualified as "soft", as opposed to "hard" programmes aimed at improving infrastruc­ tures. The results of such programmes are difficult to evaluate precisely because they are not tangible. Six large projects were selected as cases to be studied. In terms of the selection criteria, the projects had to be representative of the diversity existing in intervention strategies and locality. The selected projects also had to have benefited from significant funding in the framework of the NordTek Programme and be oriented exclusively towards firms (rather than research institutions or administrations). The six projects evaluated represented such a large portion of the total Programme that the analysis of the impacts of these interventions could be used to cover all the important phenomena. It was recognised without ambiguity that the selected projects represented the "best cases" of the programme. The data were collected at two operational levels: at the level of the project as such, and at that of the firms. As far as the project was concerned, the evaluators studied each of the cases in the field four or five times over a period of two years. Each time the managers were questioned and each project had to keep a record, in a "register" designed especially for the purpose, of the services proposed and the firms concerned. The analysis of the documents thus obtained was handed to the project leaders and discussed during visits. As for the firms, the data were collected during Principal evaluation techniques and tools interviews with a sample of 20 SMEs that had used the services offered. Finally, 40 SMEs who had not used the proposed services were questioned in the final stage of the evaluation. Source: Olsen, Leif and Olaf Rieper. Nordtek-evaluieringen, AKF Forlaget, 1991. What is the purpose of the tool? The case study may have the following aims: • illustration: the case study is a tool that may be used to add realism to an evaluation if it is presented in a narrative form. The case must, however, be chosen carefully because it must be representative of the programme as a whole or illustrate a specific point - for example a particularly effective action or an approach which was found to have serious deficiencies and which should therefore be avoided in future. • exploration: putting forward hypotheses for future investigations, identifying the various points of view of the stakeholders. • critical analysis: verify and validate a statement concerning a programme, project or strategy. • analysis of implementation: examine the diffusion of services and its mechanisms, often in different places. • analysis of the impacts of programmes: understanding the nature of the processes producing impacts. The results of a case study are always presented in a narrative form, as a story, thus giving the reader an "inside view" of the case studied and an impression of authenticity. The case study therefore has an analytical and communicative aim. Readers are more likely to relate to cases where the programme(s) and personnel involved are identified. However, particularly when a case study is being used to illustrate and learn from failures it may be necessary to anonymise some or all of the material in order to secure access to data and personnel. Case studies can often be designed in a cumulative way to help answer evaluation questions. The same case programme may also be studied over time to provide an analysis that is updated iteratively. Cases may be descriptive, normative or designed to show causality. They may be particularly useful in pedagogic/training situations for example being used to enable officials to evaluate alternative evaluation methodologies. Tools for observing changes in the field In which cases should they be used? Case studies have been used extensively in evaluation during the past decade. Today this approach is known to provide valid information for both the evaluation of programmes and the diffusion of new knowledge. Case studies which use sophisticated selection procedures (e. g. "multiple case studies with replication design") tend to replace large-scale quantitative surveys carried out in diverse cultural contexts. its use is appropriate for the The case study is a method of holistic analysis applied to complex in-depth situations. This means that understanding of behaviours and social phenomena, by using the persons and organisations analysed as a frame of reference. Case studies are valuable inductively, by developing assumptions concerning the phenomena linking cause and effect. These assumptions must then be supported by information drawn from the different case studies and tested through the search for alternative explanations. the effects of programmes identifying for This may prove useful for observing expected results and impacts, but also for revealing unexpected ones. The approach is less suited to the identification of causal links, although it may be used to demonstrate that they are likely to exist. The case study is intended to be the most complete illustration possible of a given situation, so as to give a precise image of current phenomena and to understand their causes. This is obtained by means of the description and then the analysis of examples situated in their context. It follows that this type of analysis must be based on multiple data sources, such as interviews, observations over time, statistics, physical information, etc. The data must also be cross-checked in order to ensure its coherence. The notion of "context" encompasses all the factors that could affect the case studied. Thus, for example, the impacts of a specific project on the addressees are influenced by a large number of external factors. How should it be implemented? The quantity of work required by a case study may vary widely. One must bear in mind that the case study must be sufficiently rich to give the reader an impression of what actually occurred. However, the case study is part of the least standardised methods and may encompass a range of different Principal evaluation techniques and tools approaches in different situations. Carrying out a case study involves the following steps: Step 1. Selection of cases to study: There are at least three criteria for selecting cases: convenience/access, the purpose to which they are to be put and the extent to which they can be considered to provide wider insights beyond the particular case in question. The selection of cases is a critical step for generalising and answering evaluation questions. It is difficult to justify a selection based only on convenience (easy access to data) and probabilistic surveys are sometimes difficult to carry out. Thus, a choice based on purpose is appropriate in most cases, as shown in the following table: Which questions can it answer? Basis of selection What happens at the extremes? What explains these differences? contrasting cases What explains the effectiveness of a project? the best cases Why does a project not function? How can the different types of project be compared? the worst cases by sub-sets Among the examples chosen to represent significant variations, what happens and why? representative cases On a typical site, what happens and why? typical cases In these specific circumstances, what happens and why? particular cases (Adapted from GAO, 1990, p. 23) Step 2. Data collection and process: Theoretically, data collection covers all available information about a case including that derived from project documents, project meeting reports, and collected at the various operational levels: interviews with project leaders and staff; observation of the site of the project; surveys among the addressees of the services provided by the project. These data must be collected, recorded (compilation of a "register") and pieced together so that they can be used in the final report. Step 3. Case report: Drawing up the report on the case involves the organisation of all the raw data on the case into a body of exploitable information. This is then edited, redundant information is eliminated, and the different parts are combined. The report is organised in such a way as Tools for observing changes in the field to be easy to consult, either chronologically orthematically. The report must include all the information required for subsequent analysis, that is to say, for constructing an account of the case study. Step 4. Account: The case monograph should give the reader immediate access to relevant information and to the particular situation of the case - the situation of a project - and provide an understanding of the project as a whole. Each case study, in an evaluation report, must be isolated (the size may vary between one and five pages). Nevertheless, in the last steps of the analysis the cases may be used as contrasts or comparisons, depending on the evaluation objectives. Application to Structural Funds The multiple case study method is particularly well suited to analyses of the various member States and regions, but also to thematic evaluations. The flexibility of each case study makes it possible to draw up an adequate description of the peculiarities of a given place or a project. The formulation of a common set of questions, relative to the evaluation, facilitates the analysis of the results obtained from multiple case studies. In fact the results prove to be more sound when they are produced in relation to a variety of places (through re-using case studies). Similarly, the specificity of success stories or failures will then seem more obvious. It should be recalled that the transversal analysis of cases consists of cross-referenced qualitative examinations and a description based on the frame of reference established by the evaluation questions. A cumulative process may be sought when the evaluation is focused, for example, on operational programmes in Objective 1 regions and when the conclusions have to be synthesised on the scale of several member States. The presentation of the results of several case studies could be a barrier to more generalised use. This difficulty may be solved by means of a graphic summary providing a brief report of the case history and a graphic presentation of the results, in relation to each of the questions. In this form, the answer to each of the evaluation questions, for each case, is set out on a single page: a graphic presentation at the top, a short but rich summary of the case history, with the main results, and a concise conclusion. The summary of the transversal case could be sketched in the same way, followed by conclusions and recommendations. Principal evaluation techniques and tools Another type of case study that may be applied to the Structural Funds is the "integrated approach". This approach takes into account, for example, the study of results and impacts in the context of a specific programme. In the NordTek example mentioned above, for each of the projects, a number of addressees (SMEs) were selected for a mini case study. In that way the report by the SMEs on their use of the results obtained could be used in the context of the project. Thus, variations in the results among the different SMEs can be explained by differences in the situation of the SMEs concerned. Strengths and limits in practice • The case study is relevant for giving a view of processes and complexities that are impossible to see in any other way. It may even make outside persons, such as European managers who are hardly involved in this field, aware of the reality of daily actions. It provides them with a clearer view of the way in which the programme is put into practice once the decision has been taken with the national authorities. • Depending on the selection strategy opted for, if several case studies are carried out, it may prove impossible to extrapolate beyond the results of those case studies. It is however impossible to generalise, in the statistical sense, the results to a global population; to go from a few projects to a programme, or from a few addressees to all the addressees of a certain category. • This approach is less appropriate for measuring the amplitude of impacts or for inferring the causality. • Due to the cost of setting up a good case study (requiring sources of multiple data and competent evaluators), it is necessary to limit the number of observations. The case study may, however, be re-used and applied to other contexts, thus providing economies of scale. • The credibility of the results of the case study is likely to be undermined if the method is not implemented correctly: incompleteness, arbitrary selection of information, comments cut short, distortion of results, etc. To enhance the reliability of a case study, several precautions are recommended, e. g. : rereading of the case studies by the persons concerned in order to verify the precision and the veracity of the data and their interpretation; or having two different evaluators write down their Tools for observing changes in the field comments on the same case; involving outside professionals (such as journalists) in the writing of the comments. Further reading Dufour S. , Fortin D. , Hamel J. (1994), 'L'enquête de terrain en sciences sociales: l'approche monographique et les méthodes qualitatives', Montréal: Saint-Martin GAO (1990), 'Case Study Evaluations. ' Washington DC: General Accounting Office, 133 pp Sound general approach to the method. Yin R. K. (1994), 'Case Study Research - Design and Methods', 2nd ed. Newbury Park, Sage Publications Complete guide to the case study technique. Tools for analysing data An important function of evaluation is the comparison of primary and secondary data. In the situation concerned, these data are generally numerous and heterogeneous, making the use of sophisticated tools difficult. Most evaluations use classical tools such as tables or graphs (see, for example: Henry G. T. ed. , 1997, "Creating Effective Graphs: Solutions for a Variety of Evaluation Data", New Directions for Evaluation, no. 77). An innovative and particularly useful technique in this situation is the geographical information system which can be used to compare data by considering their location. The maps produced by means of this tool are used to make visual comparisons that are often highly informative. Owing to its flexibility, the case study technique, mentioned above, is one of the rare techniques allowing for an exhaustive, although qualitative and non-graded, analysis of all the effects of a complex programme. the Most quantitative analysis tools are heterogeneity of data and the multiplicity of publics and processes of production of impacts. For example, it seems very difficult, in this situation, to use tools such as factor analysis, regression analysis or comparison groups. inappropriate because of On the other hand, macro-economic tools are appropriate in cases of vast programmes covering an entire territory, provided that the situation meets the following criteria. The funds spent in the framework of the programme must be significant compared to the economy of the territory concerned. This territory must be large enough (a country or a large region), or closed enough (an island) for the global functioning of its economy to be considered in isolation. When these conditions are met, the evaluation team can use several tools borrowed from regional economics or macroeconomics. Shift-share analysis consists of projecting national economic trends onto the economy of a region. This tool is used to estimate a policy-off situation. By comparing this with the policy-on situation, the global impact of the programme can be evaluated. The input-output model and the macro-economic model are used to simulate the economic development of the region. These tools are generally applied ex ante to two types of assumption (with / without the programme) and provide an overall estimation of probable macro-economic impacts. Tools for analysing data Geographical Information Systems O Φ © High cost Overall evaluation High technical complexity Geographical information systems (GIS) were originally developed in the 1970s to stock and process information related to the physical environment (natural and man- made). More recently, the interest of GIS users has spread to socio­ economic information. GIS have thus become a way of assembling any data that can be referred directly or indirectly to a geographical location. They are now used in the field of spatial planning to collect, gather, accumulate, analyse, exploit, display and update all spatially referenced data and information. © G © Retrospective Prospective Innovative use use use in computer technology, particularly computer-assisted Advances cartography, have led to the development of high-performance information systems. The management of public interventions is thus able to cope with a considerable number of interdependent factors, so as to better deal with the complexities of the real world. Thanks to GIS, these complexities can be described and analysed by combining information from the thematic, spatial and temporal dimensions. Example: Evaluation of the Languedoc-Roussillon IMP in France This example corresponds with that of Box 12. The GIS technique was used in an evaluation of the Languedoc-Roussillon Integrated Mediterranean Programme (IMP), primarily to analyse its spatial effects. This particular tool was chosen because the evaluation was required to account for the territorial synergy of a programme composed of multiple actions consisting of subsidies granted in selected areas. A wide range of fields was affected by this programme. These concerned problems of regional development such as the maintenance, diversification and adaptation of agriculture, the creation of new jobs, the development of tourism, and the protection of the environment, within spatial (in defined geographical zones), thematic (by economic activity) and temporal (reference to the initial state) perspectives. To assess the impact of the IMP measures and the effects of synergy, it was necessary to observe the changes brought about by the programme and then to explain them. Principal evaluation techniques and tools For this purpose an "initial state" was constituted to describe the reference situation. This step required a substantial effort gathering all the data and indicators of the socio-economic structure in the territory under consideration - an effort that was essential in order to be able to make a comparison with the situation observed after the allocation of the subsidies. Once the data base had been compiled, the evaluators, using mainly geographical representations, attempted to verify if the actions undertaken respected the logic underpinning geographical location, spatial integration and spatio-temporal integration. The use of cartographic representations associating material and relational data (social, political and economic) is relevant when the programme itself is based on the identification of sensitive areas within a given territory. Thus, for example, actions to prevent forest fires (reforestation, protection) were correlated cartographically with the areas prone to fires (frequency) in order to assess the logic behind their spatial integration. While representation on a map is relevant in this case, it should be recalled that very often the understanding of a map is not immediate and the results require interpretation. Initially, only the actions of the programme in question and the current year were registered. Actions of the same type undertaken by other organisations between the time of the "reference situation" and the "observed situation" were not recorded. This led to a biased interpretation of the real situation. Source: Commission européenne DG XVI (1993) L'évaluation du PIM Languedoc- Roussillon, MEANS pilote. W h at is t he p u r p o se of t he tool? In general, GIS are used to gather all sorts of information of a spatial nature and to present it in a form that makes it easy to read. GIS is able to incorporate s o c i o - e c o n o m ic data, data from censuses, surveys and inquiries, and monitoring data. The spatial representation may be direct (cartographic co-ordinates, precise postal address) or indirect (postal code, area of census). In terms of evaluation, the main a d d ed value of a GIS is that it allows the cross-referencing of data that other techniques do not allow. In particular, the tool can c o m p a re information on which the geographical references differ, e. g. address of firms assisted and urban areas in difficulty, or areas eligible for assistance and environmentally sensitive areas. A GIS also makes it possible to estimate the value of an indicator for a given area, when its vasue is registered on different statistical scales. (This point is discussed in detail in Volume 2 of this Collection, devoted to indicators. ) Tools for analysing data In which cases can it be used? Constructing a GIS is a heavy investment, especially in terms of data input. It is therefore difficult to make such an investment in the framework of an evaluation with a limited budget and duration. Such an investment is, however, justified in the context of monitoring. If a GIS already exists, an evaluation would be well advised to use it. The collection of evaluation data has to include systematic geographical references, so that cross-referencing with other information already contained in the system can be performed. How should it be implemented? GIS use a basic structure of special data. All the information is linked to a system of geographical co-ordinates characterised by nodes, lines and areas (and therefore also of points, arcs, polygons, etc. ). Box 16 - Principle of a Geographic Information System Topography Properties Protected areas, forests, etc. Density of population, Constructed areas, etc. Fire protection projects Resulting map (selecting and combining variables) The different steps in a GIS are as follows: Step 1. Collection of basic data: The first phase is the collection of information. The basic data are taken from a wide variety of sources: Principal evaluation techniques and tools géomorphologie maps, aerial photographs, existing socio-demographic data, etc. In some cases, when the raw data cannot be saved directly on computer (e. g. a sample bank), it is recorded by means of indicator cards. This step includes the digitisation and inputting of geographical data and associated alphanumerical data. Step 2. Storage and management of geographical data base: This function includes the storage of data in an alphanumerical form and the programming of recall mechanisms to meet the needs of the subsequent steps, such as users' requests. The idea is to organise the data (creation of an index) to facilitate thematic, topological, etc. criteria. its access based on geographical, Step 3. Exploitation and analysis of data: A wide range of techniques is needed for the transformation and use of data in the framework of a specific problem. The user proceeds either by "questioning", that is to say, he or she asks the GIS a question directly, or by "extraction" when he or she wants to compile a partial file for a specific purpose. Moreover, the GIS allows the combination of data and the identification of correlation between certain variables with a view to highlighting the characteristics of a geographical area. Step 4. Rendering results: A GIS comprises different graphical systems allowing for the organisation of data in models, such as maps, graphs, pictures, projections of sequences, etc. These facilitate communication and allow for user-friendly exploitation of the data. Application to the Structural Funds This tool has considerable potential for integrating and synthesising information. Its ability to integrate the territorial dimension is relevant for the Structural Funds. It is used to centralise results from statistics offices, monitoring, and evaluation surveys carried out in the eligible territory. information relating to the A GIS also allows easy visualisation of the programme. This characteristic can be extremely useful during presentation of the results to the various committees and work groups concerned. This potential for synthesis facilitates the illustration of the coherence of the measures implemented in the framework of the programmes, something that is an asset for the publication of synthesis documents. Tools for analysing data Strengths and limits in practice • Above all, GIS have a didactic faculty: results can be rendered in a user- friendly way, which helps to enhance the awareness of actors. However, cartographic language is not univocal; signs and keys are codes that vary, depending on the designers, and need to be explained. • Apart from the fact that the creation of a complete geographical database often proves to be costly in both time and money, the use of a GIS may be complex. Existing data are often collected on different spatial entities (administrative partitioning into communes, cantons, etc. ) which requires time consuming transformations or sometimes even a new collection of data. Existing data may also be stored on information systems that, due to their incompatibility, make it impossible to use them. • Before being integrated in a GIS, the data produced by monitoring must that may be techniques be processed by means of statistical cumbersome. • In practice, GIS facilitate the processing of a substantial number of data. It also has the advantage of allowing the transmission of data from one user to another. The centralising nature of the tool facilitates the updating of data and allows for the manipulation of geographical information on varying scales (e. g. from a small area of land to an entire region). • GIS are also useful when they are combined with other analysis techniques such as factor analysis. • Experience tends to show that the tool may generate conflict between territorial officials due to their biased reading of the maps produced. Further reading Collet C. (1992), Système d'information géographique en mode image, coll. "Gérer l'environnement", Lausanne: Presses polytechniques et universitaires romandes. 186 p. Simple but very effective for an introduction to GISs. Fisher M. and Nijkamp P. (1993), Geographic Information Systems, Spatial Modelling and Policy Evaluation, Berlin: Springer-Verlag. 280 p. Volume comprising background articles with a specific focus on applications to evaluation. Principal evaluation techniques and tools Martin D. (1996), Geographic Information Systems, socioeconomic applications, London: Routledge. Introduction to GISs with a specific focus on socio-economic applications. Tools for analysing data O Overall evaluation S h i f t - S h a re a n a l y s is Shift-share analysis has been used since the 1960s for evaluating regional policies. It is a t o p - d o wn evaluation technique in so far as it projects observable trends at the level of a country, o n to the economy of a region. The starting point of a shift-share analysis is not the study of the consequences of a p r o g r a m m e, but the o b s e r v ed Θ Φ Low technical complexity Low cost Ο Θ Retrospective Stabilised use use trends at the national and regional levels. Shift-share analysis focuses only on macro-economic dimensions, to construct a basis for comparison of what would have happened without the programme. Example: Employment impact of regional development policies in Scotland 100 -200 This graph represents a difference in thousands of jobs between the policy- on situation and the policy-off situation A positive value corresponds to a regional creation of jobs greater than forecasts based on national trends. The difference (residue) is attributed to the effects of regional policy. The case of Scotland is a classical example of shift-share analysis. It is particularly meaningful because it presents two clearly distinct phases in regional policy: a passive phase during the 1950s, during which there was no regional policy, followed Principal evaluation techniques and tools by a more active phase during the 1960s when the British government provided abundant support for the development of Scotland. The application of shift-share analysis to the evaluation of British regional policy in Scotland was used to draw the graph above. It can be seen that, in the 1970s, jobs in Scotland tended to increase faster than forecasts based on national trends suggested. This leads to the conclusion that regional policies had a positive impact on employment in this region. Source: Diamond D. , Spence N. (1983), Regional Policy Evaluation. A Methodological Review and the Scottish Example, Gower House: Ashcroft What is the purpose of the tool? Shift-share analysis is used to give an indication of the effects of a policy in a given region. The effect of the policy is determined by comparing the real situation in the region, i. e. the policy-on situation, with a reconstruction of what the regional situation would have been like had it followed national trends, i. e. a policy-off situation. In which cases can it be used? Shift-share analysis is designed to evaluate the effects of regional policy at a macro-economic level. Used for retrospective analysis, it is based on statistical indicators and requires no survey on the target groups of the programme. How should it be implemented? Shift-share analysis consists of two steps: the construction of a policy-off situation, and the comparison. Both steps are illustrated by means of an analysis in terms of employment (the most frequent use of shift-share analysis). Step 1. Construction of the policy-off situation: this is based on simple modelling. The national employment trends are applied to the structure of regional employment. The formulation of this procedure is as follows: IM, = IM0 + NS IM characterises the sectoral distribution of employment in the region (i. e. the industry mix) and NS is the variation of the employment that would be produced if each regional sector developed at the pace observed for this sector nation-wide (i. e. the national share component). Tools for analysing data Step 2. Comparison for determining the impact of the action: This step consists of the comparison between the policy-off situation, obtained during the preceding step (IMt) and the policy-on situation. The difference between these two situations (the residue) is used to highlight the impacts of the political action. RS = E, - IM, Here, the programme impact is equivalent to the residue (i. e. the regional shift component). Box 17 - Example of shift-share analysis of regional employment The number of jobs in a region in which the public authorities provide e m p l o y m e nt incentives is 300,000 ( I MJ in which 100,000 are industrial jobs, 150,000 are in the service sector and 50,000 are in agriculture and the f o od sector, during the period t = 0. Nationally, employment trends ¡n the industrial, tertiary and agricultural sectors are - 1 %, 3% and - 2% respectively between t = 0 and t = 1. Assuming that regional employment developed at the same pace as national employment, the increase in regional employment w o u ld increase by 2,500 (NS). The level of employment in the policy-off situation w o u ld then be 302,500 (IM,). By contrast, during the period t = 0 to t = 1, employment in the assisted region progressed by 5 %, i. e. 15,000 jobs. The number of jobs w as then 315,000 (E,). According to this analysis, the number of jobs created as a result of the intervention is 12,500 (RS). Application of national trends to the region I M0 = 300 000 jobs IM-, = 302 500 jobs 315 000 jobs RS = 12 500 Residue Impact Distribution per sector of employment in the region Service sector + 3% Industrial sector - 1% Agricultural sector - 2% Before the programme Policy-off situation Policy-on situation After the programme Principal evaluation techniques and tools Application to the Structural Funds Because shift-share analysis was designed to evaluate the effects of regional policies it is, theoretically, directly applicable to the Structural Funds. It can therefore be used in ex post evaluations of regional development programmes, whenever the emphasis is on the study of a particular impact such as employment or value added. Shift-share analysis is appropriate for use in programmes implemented in the framework of the ERDF, due to their regionalised nature. Because this technique has few requirements, it has the advantage of being easy to use. The statistical data it requires are available for most European regions, particularly in the form of employment data and value added for NUTS 1 and 2, provided by EUROSTAT. Strengths and limits in practice • Shift-share analysis is used to obtain quantified conclusions on the effects of policies. • The implementation of a shift-share analysis is not standardised (dates of start and conclusion, sectoral breakdown, national reference); as a result it is difficult to compare the results of analyses of different programmes. • It is an inexpensive tool because it can use widely available statistical data. All that is needed is an impact indicator (employment or value added), with a sectoral breakdown at the regional and national levels. • The logic of the implementation of a shift-share analysis is often simplified to the extreme. Usually, the variations observed in regional employment can be explained by factors other than the simple projection of national trends. Among such factors are the growth of the region's exports, productivity, quantification of manpower, population-growth pressure, immigration, and the intensity of research and development. It is not credible to attribute the effect of all these factors to regional policy only, unless this policy had a very strong hold over all the regional economic agents. • In shift-share analysis the impact consists of the residue, which is the difference between the policy-on and the policy-off situations. However, the residue may be attributed both to the programme under evaluation and to all other national and European political actions, as well as to the former trajectory of regional policy. In theory, shift-share analysis can be Tools for analysing data applied more satisfactorily if a very far-reaching regional policy is evaluated at the end of a period characterised by the absence of regional policy. Further reading Bartels C. P. A. (1982), Measuring Effects of Regional Policy. An introduction, Regional Science and Urban Economics 12, 41 p. A review of all techniques used to analyse impact of employment effects. Shift-share analysis is addressed from a critical point of view. Highly accessible. the Steevens B. H. , Moore C. L. (1980), A Critical Review of the Literature on Shift-Share as a Forecasting Technique, Journal of Regional Science, Vol. 20, N°4, pp. 420-435 A highly formalised article presenting the main shift-share models and their possible uses in forecasting. Tervo H. , Okko P. (1993), A Note on Shift-Share Analysis as a method of estimating the employment effects of regional economic policy, Journal of Regional Science, Vol. 23, N°1, pp. 115-121 A critical article on the use of the shift-share technique for evaluating regional employment effects. Based on formalised demonstrations. Tools for analysing data Input-Output Model I n p u t - o u t p ut matrices were originally d e v e l o p ed to analyse interaction between the different sectors within relatively c l o s ed economies (low level of exports and imports). They describe the way in w h i ch the productive system satisfies final d e m a nd (consisting of c o n s u m p t i o n, investments and exports). An O Overall evaluation 0 High technical complexity O φ) High cost Θ Prospective use Use stabilised input-output matrix represents the links between an economy's resources and their consumption. The matrix may vary from the simple (three sectors: industry, services and agriculture) to the complex (over 500 branches). It is one of the only techniques applicable to the evaluation of the sectoral impacts of structural interventions, because it makes it possible to divide up the productive structure of an economy in great detail. An input-output matrix can be compared to a macro-economic model that is highly simplified as regards the economic mechanisms represented, but extremely detailed from the sectoral point of view. Example: The impact of Structural Funds on Objective 1 countries and regions A team of German academics adapted input-output analysis to the study of the impact of Structural Funds on Objective 1 countries and regions. It used matrices provided by EUROSTAT for the reference year 1985 and forecasts for the years 1991, 1994 and 1999. In this example, the team tried to estimate the impact on overall economic growth and on sectoral distribution. The following results are the most representative ones from this study: Without the Structural Funds, average economic growth in Objective 1 areas would have been less than 0. 7% in absolute terms. In 1999, the Structural Funds are expected to induce a 1. 9% growth in employment in the Objective 1 areas. The analysis was differentiated for the different countries and effects on growth and employment were shown to be greater in Greece and Portugal. For the period between 1994 and 1999, changes in the sectoral structure of the production (in terms of GDP) of these regions can be represented as follows: Principal evaluation techniques and tools 19. 0% 119. 9% 19. 0% 19. 1% Public sector Services Building and civil engineering l9. 6% _J9. 1 % Manufactured products Energy Agriculture I5. 594 |5. 7% u. 2% 5. 2% ZJ42. 8 _ | 4 1. 0% D 1999 D 1994 0% 5% 10% 15% 20% 25% 30% 35% 40% 4 5% Source: J. Beutel, Konstanz University What is the purpose of the tool? Input-output matrices are used primarily in economic forecasting, where they serve to verify policy scenarios, based on the technological structure of the economy of the country and on the state of final demand. In an evaluation they can be used with or without policy interventions, in the same way that a macro-economic model is used (see corresponding sheet). The tool is used to estimate impacts differentiated according to the different branches of an economy. Among the numerous applications of input-output matrices to the evaluation of development programmes, the above- mentioned example will be considered because it has the advantage of presenting harmonised findings on a European scale. The model was produced by Konstanz University and EUROSTAT. It was used for estimating the net effects of the main programmes financed by the Structural Funds. In which cases can it be used? In general, this tool is used on a national scale, because this is where the statistical data needed for the construction of matrices are collected. Tools for analysing data Regional matrices do exist and also lend themselves to input-output analysis. However, their statistical value is not as great because of the difficulty of monitoring the movement of goods that are not subject to customs duty. How should it be implemented? The following lines describe the implementation of the tool from its construction to effective simulations in the context of the evaluation. In reality, an input-output matrix is rarely constructed for an evaluation. The implementation of the tool starts, in practice, with step 4. Step 1. Construction of a transaction matrix: the connections between the different branches of an economy are described. The lines of this matrix consist of the outputs, which concern resources supplied by a given sector to each of the sectors of activity and to the final demand. The columns consist of the inputs from the different sectors, i. e. the consumption per sector required for production. Outputs Inputs Agri­ culture Industry Services Final demand Inter mediate consump­ tion 250 400 400 100 320 600 Total utilisa­ tion 350 720 1000 1050 1020 2070 50 100 200 350 650 1020 1000 2070 Agriculture Industry Services Intermediate products Value added 50 150 100 300 50 Production 350 150 150 100 400 320 720 The values taken in this example are monetary. Thus, for example, the agricultural branch provides a value of 150 to the industrial branch for its production, whereas a value of 100 is intended for the final demand. The industrial branch consumes a value of 150 in intermediate products from the agricultural branch for its production, leading to the creation of value added of 320. By adding the intermediate consumption of agricultural products by the different sectors, to the final demand for the agricultural Principal evaluation techniques and tools branch, one obtains the total utilisation of agricultural production. The final demand may itself be broken down into final consumption, investments and exports. The sum of the intermediate inputs consumed by the agricultural branch for its production, and the agricultural value added, equals the agricultural production. The totals of the lines and the columns are therefore equal. Step 2. Construction of a matrix of technical coefficients: This is based on the matrix of transactions. The technical coefficients are nothing other than values taken from the matrix of transactions divided by total production. The relative part of the value created by each branch for all the other branches can thus be seen. The technical coefficients serve to provide indications on the technical structure of the modelled economy; they show which sectors have a high level of value added, productivity and exports. These coefficients are artificially fixed for the following steps. Step 3. Construction of an inverted matrix: This is the result of a matrix transformation through which multiplier coefficients can be calculated. These coefficients summarise all indirect effects. Step 4. Hypotheses of primary impacts: A quantitative estimation is made of the main impact of the programme, in the form of an increase in final demand (increase in public spending, increase in consumption, increase in investment, etc. ). The assumption of increased demand infrastructure includes a breakdown per sector, so that it can be introduced into the matrix. Step 5. Estimation of total impact: The calculation of total impact is based on the assumptions of primary impact and of the inverted matrix. Based on the primary impact on a sector, the tool is used to estimate all indirect effects on all sectors, particularly in terms of consumption, imports, production and employment. Application to the Structural Funds Among the different applications of the input-output models, the focus here will be on the model mentioned above, known as the "Beutel model". This example is particularly interesting because the model was constructed especially for evaluation and has the advantage of providing harmonised estimations on a European scale. The Beutel model makes use, above all, of a series of harmonised input- output tables for the year 1985, drawn up by EUROSTAT for the economies Tools for analysing data of the European Union. These input-output tables are completed by regional tables, particularly in the case of Spain and Italy. The countries covered are Portugal, Greece, Ireland, Spain, Italy and the UK (Northern Ireland). Germany is, moreover, included in the planning period 1994-1999. Since the 1985 database is obviously outdated for any practical purposes, it has to be updated. For this purpose Beutel developed a forecasting technique adopted by EUROSTAT (see more detailed information at the end of this section). The main impact of Community funding is estimated by dividing total expenditure into three categories: property investments, investments in capital goods and salaries. A comparison is then drawn between the development of the economy with and without Community funding. As far as the evaluation of interventions in the framework of Objective 1 is concerned, the Beutel approach provides estimations focused on the following key macro-economic variables: • Employment: comparison between cases with and without intervention is established not only in terms of total employment, but with a distinction being also made between employees on the one hand and independent workers on the other. • The GDP and its growth rate: It is possible to draw a comparison between growth interesting comparison is made between the annual growth rate with intervention and cases where interventions are interrupted during a specific year. rates with and without intervention. Another • Gross capital formation: This variable captures changes induced in investments by the interventions and its annual growth rate. • Value added by sector: The sectoral distribution of changes induced in 25 economic sectors by the interventions is calculated each year. A stan­ dard presentation exists of the evolution of the 6 sectors undergoing structural change: agriculture, forestry and fishing, fuel and energy, building and civil engineering, private services, and public services. • Exports and imports: The impact of interventions on exports and imports is calculated each year and sub-divided into exports and imports to and from countries belonging to the European Union, and in exports and imports to and from non-EU countries. Strengths and limits in use • The most interesting contribution of input-output matrices concerns impacts on sectoral distribution and trade. Leakages due to imports by Principal evaluation techniques and tools the beneficiary country from Member States of the European Union and third countries may prove to be important for EU policy-making. Similarly, knowledge of the impact of sectoral demand may prove to be important, particularly if it is considered that some sectors must be stimulated to accelerate the growth rate. • The value of this approach lies above all in the broadness of its scope and in its coherent treatment of the main economies of the European Union benefiting from Objective 1 support. • The advantage of the tool is limited to the estimation of effects on demand. Therefore, it does not take into account one of the most im­ portant effects aimed at by Objective 1 interventions, i. e. lasting effects on productive potential. Most effects on supply, which are likely to lead to a sustainable increase in the growth rate of assisted regions and to enable them to catch up with more developed areas, are totally overlooked (creation of new productive capacity, improvement of the training and education of the workforce, construction of infrastructure, productivity gains throughout the economy, spread of technological progress and intensity of high-tech activities in the productive sector). It is all these effects on supply which transform productive capacity in a lasting and irreversible manner. • A tendency to over-estimate the impact of Structural Funds on demand has been observed, because is not the moderated by feedback effects, particularly as regards price changes originating in product markets, financial markets and the labour market. interventions impact of Further reading W. K. Brauers (1995), Prévisions économiques à l'aide de la méthode Entrées - Sorties, Paris: Economica, 111p An introduction to input-output analysis, for the general public. Highlights the main techniques, recounts the debates that took place around this subject and provides examples of application. Concentrates on the most important aspects. R. E. Miller and P. D. Blair (1985), Input-Output Analysis, Foundations and Extensions, New-Jersey:Prentice-Hall, 464p Intended for the initiated, this book describes the basis of input- outpu: analysis and provides a wide view of the different types of Tools for analysing data analyses that may be carried out by means of this technique. Readers can easily select those articles of particular interest to them. H. D. Kurz, E. Dietzenbacher and C. Lager (1998), Input-Output Analysis, Cheltenham: Edward Elgar, 3 volumes, 1504p In this work the evolution of input-output analysis is studied through a collection of articles published since the origins of this type of analysis. More in-depth: the construction of Beutel model matrices In the Beutel model, the EUROSTAT input-output matrices are updated by means of economic forecasts by the European Commission Directorate General for Economic and Financial Affairs. These forecasts are derived from the QUEST model (see sheet on macro-economic models). Use is also made of the forecasts of 25 productive activities, prepared by ERECO (European Economic Research and Advisory Consortium, composed of Ifo- NEI/Rotterdam, Institut/Munich, WIFO/Vienna, and Cambridge Econometrics/Cambridge). Prometeia/Bologna, BIPE/Paris, The data used for updating the model consist of the input-output table for the reference year, including the import matrix, growth rate forecasts of value added per sector (agriculture, industry, services) and the forecasts of components of final demand (consumption, investments, exports and imports). The updated input-output table is estimated by a procedure of iteration which provides estimations on the intermediate consumption of imported goods and services, the structural composition of final demand by product, domestic production by product and imports by product. This technique tries to avoid random changes of input coefficients by treating all activities equally. Apart from its limited need for data and the fact that it uses official sources, this technique has the further advantage of producing sound updated versions at a low cost. Tools for analysing data Macro-economic model The p u r p o se of the m a c r o- economic model is to represent the functioning of an entire economy a nd of the different markets comprising it (e. g. product market, job market, financial market, etc. ). A model schematises the functioning of an e c o n o my at equilibrium. It also simulates the evolution of this e c o n o my O Overall evaluation Β High technical complexity φ) High cost O Θ Prospective use Use stabilised following an exogenous shock (e. g. variation in oil prices, variation in exchange rates, tax imposition), the adjustment of modelled economic variables and the attainment of a new market equilibrium. Macro-economic models are used by administrations for the preparation of their budgets and for the study of political choices when the economic stakes are high. Some public and private forecasting institutes also use models for short- and medium-range economic forecasts. The use of this type of model for evaluation is far more recent and still involves methodological innovation. This does not, however, preclude the existence of success stories such as the following case. Example: Evaluation of the probable impact of European economic support to the Republic of Ireland The evaluation of the probable impact of European economic support to the Republic of Ireland (CSF 89-93) was carried out by the Economic and Social Research Institute in Dublin (ESRI), using a macro-economic model constructed by this institute in the framework of the HERMES programme. The evaluation was to estimate the probable impact of European support on the Irish economy as a whole and, in particular, on variables such as the per capita GDP or the public debt. The HERMES model was constructed to simulate the functioning of the Irish economy in the context of an energy crisis. It was modified slightly during the evaluation in order to take into account the impact mechanisms of European programmes more satisfactorily. The model used for the evaluation contains over 600 equations representing assumptions on the behaviour of the economic agents concerned directly or indirectly by European assistance. The evaluation conclusions were as follows: In the year 2000, the impact of the Community intervention is estimated at 2. 7% of Irish GNP. Principal evaluation techniques and tools However, this impact alone will not be sufficient to allow the Irish economy to catch up with the European average. It is interesting to note that this evaluation was the subject of political debates during the Irish referendum on the Maastricht Treaty, and that it served as an argument during the renegotiations of European assistance for the period 1994-99. It can therefore be seen that despite the inherent fragility of any estimation, macro- economic evaluation served as a basis for constructive debate. (Source: Bradley J. and Fitzgerald J. , Medium-Term Review 1989-1994, Economic and Social Research Institute, Dublin, 1989) What is the purpose of the tool? A macro-economic model provides a simplified representation of the mechanisms of an economic system. Used in evaluation, the tool makes it possible to simulate one scenario with public intervention and one without it, and therefore to estimate the consequences of the intervention by observing the differences between the two scenarios. The models are entirely quantified and therefore provide an estimation of the extent of the impacts on all the macro-economic variables that have been modelled. Obviously a model cannot predict everything, nor even guarantee the full reliability of these simulations. However, the tool remains relevant if an estimation of macro-economic effects is required, for no technique is more reliable. It provides reference values on the impact of specific measures that are difficult to obtain by other methods due to the complexity of the mechanisms in action (e. g. impact on employment and private investment). These reference values are used to stimulate discussions and to provide solid information about the real extent of the issues. Finally, the value of the tool is often educational. Use of the model and the examination of the results obtained make stakeholders in the evaluation pay more attention to understanding macro-economic mechanisms. In which cases can it be used? Effects can be estimated ex ante. In this case, two forecasts (with and without) are compared. This is the most frequent use in evaluation. Although it has very seldom been done, a model can also be used in an ex post situation. In this case, the comparison concerns past economic developments as recalculated by the model, and a scenario without public intervention, reconstructed ex post by the model (simulation of the policy- Tools for analysing data off situation). In the framework of intermediate evaluation, the tool may be used, where relevant, to guide the re-orientation of the programme (e. g. distribution of infrastructure development and support for R&D). investment support, funds between How should it be implemented? The implementation of an evaluation using a macro-economic model must be considered only if the evaluators have adequate knowledge, skills and time. This is because numerous highly technical steps are involved. Step 1. Adapting the model to evaluation: The construction of a model is a heavy investment that may take up to a few years. A macro-economic model is therefore never constructed for the specific needs of an evaluation. It may happen that several models are available and that the first step consists of choosing one, but this is exceptional. Generally, the evaluation team's choice is dictated by the existence and availability of a single model. The problem is then to decide whether the model can be applied in its present form or whether it must be adapted to better simulate the impacts of the evaluated intervention. In the first case, the adjustments can generally only be minor ones because the equilibrium conditions of the model cannot be challenged without the risk of having to reconstruct it entirely. In general, adaptation of the model is limited to a few adjustments that make it possible to take into account the content of the programme better, and to integrate the primary impacts in the form of exogenous shocks. Step 2. Adjusting the model: Before being used, and particularly if it has been modified, the model must be adjusted recent developments observed and to the most recent statistics. A model is ready for use when it is capable of explaining correctly what happened in the five to fifteen preceding years. In practice, the model must regularly be re- estimated to take into account revisions in basic information and thus to allow for it to be "adjusted" to the most recent data. the most to Step 3. Estimation of primary impacts: The diagram (Box 18) shows the the model. taken macro-economic mechanisms Employment depends on the production of firms, on productivity and on the level of salaries. These three variables are endogenous, i. e. internal to the model. To evaluate the effect of a public intervention, for example its effect on employment, this public intervention must be considered to introduce exogenous changes into the model (the term exogenous shock is used). into account in Principal evaluation techniques and tools The evaluation team must then select a few variables or coefficients that are part of the model and which will be directly affected by the public intervention. These values constitute "bridges" between the evaluated intervention and the model. In the following example they are public spending and productivity. The primary impacts of the Structural Fund interventions may also take the form of investments in productive structures or in human capital. The evaluation team must then estimate the primary impacts of the intervention (micro-economic impact) and introduce them into the model. Box 18- Primary impact and macro-economic mechanisms Primary impact Supply side CSF _^ Primary impact Demand side Salaries «-» Employment Public debt A Taxes Costs and productivity Import. Export. Production ·* Income Public spending I * Demand International prices Step 4. Simulations: In an evaluation context, the model will always be used at least twice. The first time it will be used to adjust the intervention to the situation (case of an ex post evaluation) or to simulate this situation (case of ex ante evaluation). The second time it will be used to simulate the situation without intervention. The impacts of the intervention are estimated by establishing the difference between the two simulations. Step 5. Presentation of results: The last phase is that of the formulation and presentation of results. Given the complexity of the tool and the numerous variables that can be quantified, it is necessary to take specific steps to present results in a simple and easily accessible way. This phase in the must therefore be considered as one of the most implementation of the tool. important Tools for analysing data Application to the Structural Funds Macro-economic evaluation of a Structural Funds programme depends on the existence of a model for the eligible territory. Available macro-economic models are generally limited to national models for reasons of availability of statistical data and the cost of modelling. The models constructed on a regional or local scale are rare, particularly because economic flows are diluted as soon as they are not in the territory concerned. However, modelling remains meaningful for small economies, provided that they are relatively closed. That is why insular economies are often equipped with macro-economic models. Thus, the use of models is generally possible for big programmes for which the eligible zone corresponds to national borders (Portugal, Ireland, Greece) or to vast regional blocs (South of Spain or of Italy). Almost all the models are ill suited to programmes for which the eligible zone is a small, open region. Furthermore, the use of a model has meaning mainly if the intervention of the programme attains sufficient mass in relation to the modelled economy. It is considered that in order to attain this critical mass, the economic importance of the programme must be at least around 1% of the GDP of the economy of the territory. In the framework of European structural interventions, macro-economic evaluation is limited to a few experiments of which the two most well known used the QUEST and HERMIN models presented below. In the case of the QUEST II model, Structural Fund spending is considered to be an increase in both public spending and the national stock of capital. From this point of view, the expenses incurred are considered as an exogenous increase of overall public consumption (demand side effect), as well as an exogenous increase in public investment, leading to an overall increase in production (supply side effect). This type of approach makes a very broad estimate of supply side effects since all kinds of investments (infrastructure, R&D and training, both private and public) are assumed to have similar impacts on productive potential. Further refinements of the model might consider differentiated impacts stemming from various publicly funded interventions. This would have the advantage of fully taking into account the supply and demand effect of Structural Fund interventions in a clear and fairly direct way. The supply effects would be simulated separately and directly via the impact on Principal evaluation techniques and tools potential production. The combination of supply and demand effects would indicate the extent to which the structure of the economy is likely to benefit from the increase in productive potential The HERMIN model was used in evaluations in Spain, Ireland and Portugal. As in the experiment with the QUEST model, the demand effects of spending in the framework of Structural Fund interventions were introduced in the form of exogenous increases in public spending. Yet evaluations carried out with HERMIN introduced an interesting innovation by explicitly simulating the supply effects of public expenses, trying to make separate estimates of the effects of three categories of interventions: basic infrastructures, human capital and investments in the productive sector. The primary impacts of Structural Funds in terms of supply are introduced into the model in several ways. They first take the form of an increase in productivity that affects one or more parameters of the production function. In the Spanish and Irish versions, productivity is "embodied" in labour (a single parameter). In the Portuguese model, it is total factor productivity which is increased (all the parameters). The increase in productivity is a function of investment generated by European support in the form of private capital, human capital and public infrastructure capital. It is assumed that there is a constant relationship (called elasticity) between the increase in investments and the increase in productivity. A real difficulty in evaluation relates to the choice of the value of this relationship. Empirical observations which might allow its estimation are barely reliable and, to avoid totally random values, similar studies based on American or international data are often used. The credibility of the basis of this form of estimation of supply effects is somewhat limited. A second hypothesis on primary impacts is that the action of Structural Funds helps to draw more foreign investments and therefore to increase the their competitiveness of exports. To estimate this chain of effects, equivalent in terms of increase in international demand is estimated. Here again, it is assumed that there is a constant relationship between the increase in investments generated by the European funds and the increase in exports. The choice of a value for this relationship is even more problematical here. First, arbitrariness is inevitable, given the total absence of applicable empirical studies not only for the countries concerned but also at an international level. Worse still, the final result seems to depend heavily on the value chosen for this relationship. This is particularly true for the long-term impact on GDP. Tools for analysing data To assess the potential of HERMIN for the evaluation of macro-economic effects, it must be kept in mind that this work is still underway and that many current difficulties are likely to be solved as the project progresses. Various teams' estimations of the impact of the CSF should help to refine the assumptions of impact, the parameters of the models, the quality of the data, and the comparability of results. Strengths and limits in practice • The aim of the Structural Funds is economic and social cohesion. This objective is essentially defined at the macro-economic level. The macro- economic model is practically the only tool that can be used to formally ascertain whether European policy has achieved its aim. • In the evaluation of socio-economic programmes, one of the major limits of models is that they do not give a credible explanation for "supply-side effects". For example, if a description of the effect of new infrastructure on the productivity of firms is required, it will be difficult to find an equation which reflects this effect and which is convincingly adjusted to past statistics. • One of the main limits of the tool relates to the construction of models. This represents a substantial investment and always takes more than a year. Moreover, the parameters of a model have to be continuously updated, under the supervision of specialists with considerable know- how. Experience shows that statistics and forecasting institutes are obliged to "readjust" their models several times a year in accordance with recent statistics. • Only exceptionally is a model constructed specifically for an evaluation. Use of the tool therefore depends on the existence of a model. The existing model must often be adapted to the questions posed by the evaluation. • Each macro-economic model is based on its own specific set of assumptions. When two models are constructed for the same territory, they never produce exactly the same simulations. • The weakness of many macro-economic evaluations relates to the way in which the primary micro-economic effects are evaluated before being introduced into the model. In this sense, the macro-economic evaluation encounters the same difficulty as the other evaluations of multi-sectoral programmes: the effects of a multitude of heterogeneous interventions have to be summarised in the form of a few standardised and quantified micro-economic impacts. This task, which is very difficult to accomplish fi Principal evaluation techniques and tools for a normal evaluation, is only the first step in a macro-economic evaluation. It is the step that was defined above as the estimation of primary impacts. Irrespective of the accuracy of its internal assumptions, the model only transforms estimations of primary impacts into macro- economic estimations. Thus, it accumulates the difficulties of the two levels of estimation. Further reading In the interests of clarity, the preceding paragraphs deliberately omitted the technical aspects of the construction of models. For readers who would like more detailed information, a list of publications on macroeconomics and modelling is provided below. Since the use of models is still at its experimental stage for the evaluation of socio-economic programmes, it is difficult to find manuals with a detailed explanation of techniques for estimating macro-economic impacts. This information sheet has therefore been extended to cover the QUEST and HERMIN models in greater depth. Andersen, K. V. (1995) Information systems the political world: Implications of Economic Modeling in Implementation, Use and Amsterdam: IOS Press. , 283 p. This book investigates the use of econometric models for policy analysis and evalution in Europe and the USA, at central and regional level. It focuses on the social and political dimensions of the processes by which models are built and used. Daloz J. P. and Goux C. (1985), Macro-économie appliquée. Du simple au complexe, Paris: Cujas, 156p. Very clear, basic approach which introduces econometrics in a very simple way. Gray A. W. (1995), EU Structural Funds and other public sector investments. A guide to evaluation methods, Gill & Macmillan, 106 pp Considers a set of models adapted to the evaluation of Structural Funds, and explains problems in this field. Highly accessible. Kebabdjian G. (1994), Les modèles Paris: Dunod, 194 pp A macroeconomics manual which addresses all current trends without any bias. A demanding, synthetic and useful tool. théoriques de la macro-économie, Tools for analysing data More in-depth: the QUEST and HERMIN models QUEST: a multinational model The QUEST model (Quarterly European Simulation Tool) was developed by DG II of the European Commission. Its main purpose is to make medium- term and long-term forecasts. The model is used to quantify scenarios in various fields: monetary policy, budgetary policy, and the economic incidence of changes on the international scene, such as fluctuations in exchange rates, interest rates, oil prices and other world prices. The way in which the multinational dimension is modelled is an essential characteristic of the model. The QUEST model uses national sub-models to cover all the Member States of the European Union (with Luxembourg associated with Belgium), the United States and Japan. For the rest of the world, only international trade is modelled. It is divided up into seven main OECD countries and six other regions. The equations of bilateral exports for the commodity market are calculated, on the basis of import volumes and export prices, for the 26 countries/regions. On this basis, the models of international commodity trade can be represented in a series of equations on commercial relations. This bloc of commercial relations can be used to monitor an economic shock arising from international trade. It is separated from national sub­ models. It is thus possible to study an impact on a country, either taken alone or embedded in international trade as a whole. The national sub-models all have the same basic structure. They have a relatively low number of behavioural equations, for which the parameters have been adjusted for each country. The basic QUEST structure combines three types of relationship: the supply bloc, the demand components, and the wage/price ratio. (a) Supply The main purpose of the supply bloc is to determine productive capacity. The basic assumption is that productive capacity is roughly limited by the stock of national capital. It should be noted that QUEST is a single sector model that establishes no differences between the products of the various sectors. Productive capacity serves, together with actual production, to define the degree of capacity utilisation. The latter variable plays an important part in the model and partly determines demand in terms of investments and imports, and therefore global demand and price levels. Principal evaluation techniques and tools Productive capacity depends on private and public capital, employment and energy, plus an exogenous increasing productivity trend applying to all factors. Full effects on potential output take about six years to materialise. (b) Demand Private consumption is determined by the present discounted income of current and expected income and wealth of households. Also current disposable income plays a role since a proportion of households is subject to liquidity constraints. Investment is positively related to the present value of profits and is a negative function of the cost of capital. Imports depend on domestic final demand and their relative price vis-à-vis domestic prices. (c) Salaries and prices It is assumed that firms fix their prices on the basis of a mark-up over marginal cost. Exports depend both on domestic production costs and prices of international competitors. There is also short run price rigidity, modelled using an adjustment costs constraint. Nominal wages principally depend on four factors: (1) the disincentives that arise from the provision of social protection and the value of leisure time, (2) labour productivity, (3) labour market pressures and (4) the general price level. Wages do not immediately adjust to changes in economic conditions since wage agreements are signed for a period of several years. Interest rates and exchange rates are also treated as endogenous variables in the QUEST model (contrary to many other models), that allows crowding out effects to be estimated (the fact that the funding of public expenditures generates an increase in real interest rate, and a consequent decrease of private investment). HERMIN: a four-sector model The HERMIN models (HER from the macro-sectoral model HERMES from which it was inspired and MIN to indicate the limited dimensions of this model) cover Ireland, where the prototype was originally developed, Portugal, Spain and Greece. The HERMIN prototype was taken from HERMES, a system of complex models (eleven sectors, over 600 equations), spawned by trans-European collaboration conducted by DG XII Tools for analysing data and covering, with various degrees of success, all the Member States of the European Union. The HERMES models were developed to analyse energy policies and therefore focused on the role of energy as a factor of production. They consequently included a careful simulation of production- related processes. The HERMIN models are small- and medium-scale (4 sectors with 100 to 150 equations each). They were developed by national laboratories in each of the four peripheral countries in the European Union. One of the main objectives of these models is the comparative study of the impact of Structural Funds on the periphery of the Union. The basic structure of HERMIN combines three types of relations: the components of supply, the components of demand and the wage/price ratio. (a) Supply The HERMIN model comprises four sectors: the exchangeable goods sector, the non-exchangeable goods sector, the agricultural sector and the public sector. These sectors have the following distinct characteristics: The exchangeable goods sector is subject to international competition and its production is determined primarily by international demand (expressed in terms of world production) and competitiveness (measured by the ratio of the unit cost of labour to world unit costs). Prices are determined externally in international markets. The non-exchangeable goods sector covers services, building and civil engineering, and community facilities. It is subject to domestic demand and prices are determined by the profit margin on labour costs. The production of the exchangeable goods and non-exchangeable goods sectors is modelled in the form of a function where the value added depends on capital and labour. As a result, the levels of employment and investments depend on the level of production. The demand of the two factors of production also takes into account their respective cost (under the assumption of the elasticity of constant substitution). Technological progress is considered to be an exogenous, time-related trend that differs from one sector to the next. The agricultural sector covers agriculture, forestry and fishing. It is considered to be essentially exogenous, with prices and production quotas defined in the framework of the European Union Common Agricultural Policy. Principal evaluation techniques and tools In the public sector, which includes the administration, education and health, the level of activity depends on government decisions. Spending is limited by the rules of budget equilibrium and debt repayment. (b) Demand Consumption is determined by real available personal income and by real financial wealth. Private non-residential investments are derived from the production function as a demand for fixed capital. Residential investments are a function of available income per capita. Finally, public investment is an exogenous variable. The financial sector is not modelled and the exchange rate, as well as the domestic interest rate, are exogenous variables. the Agricultural exports are estimated exogenously. Exports exchangeable goods sector are a function of the production of this sector and of the GNP growth rate (a reflection of the economic cycle) in the case of Spain. Tourism and service exports are a function of world production. Imports are determined in a residual fashion to differentiate between the total production of the four sectors and final demand (private consumption, public consumption, investments and exports). from (c) Wages and prices Wages are fixed by an equation that simulates the process of wage negotiations between employers and unions. Negotiations on nominal wage rates are influenced by the situation of the labour market, as reflected in the current unemployment rate or its rate of variation. They are also subject to the influence of "fiscal leakages", i. e. the difference between gross salary before tax, expressed in terms of production costs, and net salary after all deductions, expressed in terms of consumer prices. Finally, the level of production costs and that of labour productivity also affect the process of wage negotiations. In the case of Ireland, the job market is assumed to be homogeneous and wages in the exchangeable goods sector dominate all the other sectors. In Spain, salaries in the non- exchangeable goods sector are evaluated separately, but the public and agricultural sectors are also dominated by the exchangeable goods sector. The supply of manpower is determined differently in the various countries. Migration plays an important part in the case of Ireland, particular into and from the UK. In the case of Spain, a distinction is made between the employment rate of men and of women. Tools for making a judgement The overall evaluation of a complex programme is characterised by the fact that there are numerous expected effects and that these do not all have the same degree of importance for the different partners. Some tools for making judgements are particularly well suited to this situation. For example, the expert panel constructs a synthetic judgement of the programme being evaluated. This tool, when implemented with optimum efficiency, enhances the credibility and acceptability of the evaluation conclusions because differences between points of view are respected and consensus is reached. the Multicriteria analysis has significant advantages acceptance of a multiplicity of points of view. The tool includes a mechanism for the identification and explanation of points on which there is consensus. in terms of The most well known tools for making a judgement are cost-benefit analysis and cost-effectiveness analysis. For various reasons, these tools do not have the flexibility required for taking into account the complexity of a judgement on an entire socio-economic programme. Benchmarking is a more flexible tool for making judgements, which could theoretically be applied to the overall evaluation of a complex programme. However, to date this tool has been used only for simple evaluations. Consequently, it is presented further on in this document. Some tools presented above, such as those for structuring an evaluation, may also be used to formulate a judgement. For example, SWOT analysis may be used to judge the relevance of a strategy, just as the colour vote may help to validate key points in the final evaluation synthesis in a context of partnership. Tools for making a judgement E x p e rt p a n e ls Expert panels bring together several independent specialists recognised in the fields covered by the evaluated p r o g r a m m e. The experts collectively p r o d u ce a value judgement on the programme and its effects. This low cost tool rapidly provides an overall judgement incorporating the main information available on the programme, as well as numerous previous and external experiences. O Overall evaluation A In-depth evaluation Θ © Medium technical complexity Medium cost O o Retrospective Prospective use use Θ Use stabilised The panel may be considered as an evaluation tool in so far as there is a standard and reproducible procedure for forming it, bringing it together and leading it to produce its conclusions. Inspiration for the tool was based on university juries - which explains why it appeared in the early 1970s - in the field of Research and Development programme evaluation. An "expert panel" is a specially constituted work group that meets for evaluation. A similar tool, the Delphi survey, is described elsewhere in this Volume. It also relies on experts but differs in several other respects. Example: Evaluation of the quality and relevance of development programmes in Sweden For several years the Swedish National Council for Technical Development has be financing programmes for the development of basic skills and knowledge. Since 1981 the quality of these programmes has been regularly evaluated. The first step consists of selecting five international experts who are recognised and representative of the diverse points of view concerning the programme. These experts make an undertaking to participate in all the panel meetings. One of them is elected as the chairperson. In the second stage the panel examines the programme documents concerning the resources mobilised (staff, equipment, finances), the research themes, results, publications and co-operation. The panel interviews several addressees. The following stage consists of visits in the field by each expert, either individually or by groups of two. In the final stage the panel draws up a joint report. If the points of view diverge, the panel discusses them until a consensus is reached. Experience has shown that the Principal evaluation techniques and tools search for compromise inevitably softens the conclusions. However, the value of the evaluation is enhanced by the legitimacy of the panel's conclusions. Programme managers are invited to express themselves and the final report is then published. This tool is considered to be effective and inexpensive. It is applied extensively in all the Scandinavian countries and more generally in all countries that evaluate their science policies. The OECD also applies this tool to numerous other political fields. Source: Christensen, H. C (1987) "Evaluation of research Programmes", in E. Ormala (ed. ) Evaluation of Technical Research and Development , pp. 88-108, Espoo, Nordforsk What is the purpose of the tool? The expert panel is used mainly to assess a programme, but it is a generic tool. The terms of reference given to the panel may include a wide range of questions, from the relevance of programme objectives to an estimation of real or probable effects. In which cases can it be used? Expert panels are used to reach consensus on complex and ill-structured questions for which other tools do not provide univocal or credible answers. The use of groups of experts makes it possible, within a few months, to gather the main points of view and knowledge relevant to the evaluation. The tool is therefore recommended when sufficient expertise exists in the field and when the evaluation is complex. It is also well suited to small, simple programmes, the evaluation of which does not warrant the mobilisation of many resources. How should it be implemented? Step 1. Identification of a list of potential experts: The members of the panel must be specialists recognised in at least one of the fields concerned by the programme. They must have extensive experience in the field and be independent vis-à-vis the commissioners of the evaluation. They must also have the required availability and wish to become involved in the evaluation. Step 2. Selection and mandating of the experts: The panel is generally composed of between six and twelve members belonging to different "fields of expertise". The current tendency is to broaden the range of interests and Tools for making a judgement to seek the greatest possible diversity of points of view in the panel. The experts are nominated intuitu personae and do not represent their institution. Each expert signs a contract which, depending on the case, does or does not provide for remuneration. The president of the panel is chosen by the commissioner or elected by her or his peers. It is essential that the secretariat of the panel be entrusted to a person with sufficient availability, which is generally not the case with the experts themselves. Step 3. Investigations: The experts meet between three and six times, at one-month intervals. All the dates of their meetings must be planned at the outset. The panel's internal debates are under the seal of secrecy. The members of the panel consult the programme or project documents (reports, preliminary studies, inquiries) and interview the programme leaders and several typical addressees. They may also make field visits, generally in groups of two to limit the risks of bias. Step 4. Synthesis: The panel produces a report and formulates conclusions and recommendations that are collectively accepted. In case of disagreement, It may be useful to express the majority conclusion and to attach a comment by the minority expert. The panel must hold a final meeting after reception of the commissioner's comments on its draft report. Part of the panel's work, particularly in its final phase, may make use of distance communication techniques (Internet or videoconferences). Application to the Structural Funds The tool may be used to carry out an entire ex ante evaluation, provided that at least three months are available. For example, an expert panel could be asked to estimate the probable impact of a programme in terms of employment, or to assess the merits of the programme in terms of potential synergy. The tool may also be useful every time the structuring or judgement stage needs to be reinforced. In that case, the expert panel may intervene at the beginning and end of the evaluation, in combination with other tools used for the collection or analysis of the data (see example in Box 39). The expert panel may be used to formulate an independent, authoritative judgement, which is particularly useful in a partnership context, especially if there are differences in the partners' views. In that context it is possible to ask each partner to choose some of the experts, based on the similarity of their points of view. A list of experts exceeding the needs of the panel Principal evaluation techniques and tools may also be proposed, and each partner be given the right to delete one or two names from the list. Strengths and limits in practice • The expert panel is a relatively inexpensive, rapid and flexible tool. Its conclusions enjoy a high degree of credibility when recognised experts are used. • Expert panels may fulfil numerous functions, it is preferable however to limit its work to only a part of the evaluation: the structuring of objectives and estimations of effects or judgements. The more clearly the panels' work is defined, the more its significance will be recognised. The reliability of the tool may be undermined if the questions put to the experts are too broad. • The risk of bias from empathy is significant in so far as panels are too often limited to the specialists in the fields covered by the programmes (i. e. peers) who are hardly inclined to criticise the relevance of the objectives or to focus on any undesirable effects. • The comparison of opinions often leads to the under-evaluation of minority points of view. The consensual mode of functioning on which the dynamics of the panel is based, produces a convergence of opinions around majority values which are not necessarily the most relevant. • The tool is more recommendations if it is combined with suitable leadership such as those presented in this Volume (METAPLAN®, colour vote). likely to produce creative conclusions and rich techniques Further reading Callón M. , Laredo P. and P. Mustar (1995), La gestion stratégique de la recherche et de la technologie. Paris: Economica, pp. 31-88. Presentation in the first two chapters of this book of the expert panel model used by the Commission of European Communities. Cozzens S. E. (1987), 'Expert Review in Evaluating Programs', Science and Public Policy, 14(2), 71-81 Inventory and analysis of the American experience of the use of expert panels for evaluation. The interest of the conclusions transcends the context of science policy. Tools for making a judgement Nadeau M-A. (1988), L'évaluation de programme. Laval, Québec: Presses de l'université de Laval, pp. 349-352 This manual presents the methods and tools of programme evaluation; a few pages are devoted to expert panels. Witkin B. R and J. W Altschuld (1995). Planning Conducting Needs Assessments, Thousand Oaks: Sage. pp. 193-203. This work presents a series of tools applicable to the evaluation of needs; a section is devoted to expert panels. Tools for making a judgement Multicriteria analysis Multicriteria analysis is used to make a comparative assessment of alternative projects or heterogeneous measures. With this technique several criteria can be taken into account simultaneously in a complex situation. This helps "decision-makers" to integrate the different options, reflecting the opinions of the actors concerned O Overall evaluation 0 φ Medium technical complexity Medium cost © O G Innovative use Retrospective Prospective use use by a decision-making process, into a prospective or retrospective framework. It implies the participation of the decision-makers and results in operational advice or recommendations. Example: Evaluation of the regional development programme in the Hainaut region Multicriteria analysis was used in the evaluation of a regional development programme co-financed by the three European Structural Funds and the government of the Walloon region. The programme applied to the Hainaut province. It covered the period 1994 to 1996 and its budget amounted to over one billion ECU, almost 50% of which was financed by the European Union. The multicriteria analysis tool was chosen for its "multi-judge" feature that makes it possible to use different weighting systems particular to each partner. Eight evaluation criteria were defined in agreement with the representatives of both levels of government: (1) viability of the trained; (3) sectoral assisted businesses; (2) employability of the persons diversification of the economy; (4) openness of the economy; (5) environmental integration of activities; (6) territorial distribution of activities; (7) services to firms; (8) synergy between the components of the programme. The effectiveness of the measures within the programme was assessed by the managers of the measures. During the interviews they gave scores for effectiveness to each measure, in terms of these eight criteria. The results showed vast differences in the scores given for effectiveness, depending on the measures and the criteria. Then, by means of a formalised interview procedure, six "judges" (2 members of the European Commission, 2 from the Walloon government and 2 from the Hainaut Region) established a weighting of the criteria used, based on the importance they attributed to each of these criteria. Once again, the results varied widely. One of the judges gave a score of "3" to the viability criterion of businesses, while another one gave the same criterion a score of "26". This demonstrates the differences in centres of interest. Principal evaluation techniques and tools After being processed by computer, the measures were sorted by order of effectiveness, taking into account the fact that there was a category for each judge (since each one had her or his own weighting system for the criteria). Despite the differences in opinion - among managers, on the effectiveness scores, and among the judges, on the weighting of the criteria - the various classifications converged towards a virtually identical result on the relative value of the measures in relation to each other. This new light shed on the subject was of unquestionable use to decision-makers, particularly in a context where the intention was to reallocate a part of the budget during the course of the programme. Source: MEANS Handbook n°4 (1995) Applying the Multicriteria Method to the Evaluation of Structural Programmes. Brussels: European Union. What is the purpose of the tool? The purpose of the tool is to structure and combine assessments taken into account in decision-making. Decision-making is understood here to be a process made up of multiple choices. Decisions relative to these choices condition the final decision to a large extent. Multicriteria analysis is intended above all to highlight the reasoning and subjective convictions of the actors on a particular question. It also serves to synthesise the opinions expressed by these same actors, in order to determine the priority structures, to analyse conflictual situations, or to formulate recommendations or operational advice. In which cases can it be used? Multicriteria analysis is similar to the techniques adopted in the field of organisational development or information systems management, such as the Critical Factors of Success method developed in the 1970s. It also resembles cost-benefit analysis although it does not reduce highly disparate phenomena to a common unitary (monetary) base. This improvement is essential because it allows criteria to be taken into account even when their cost is not known in advance (scenery, noise, satisfaction, etc. ). Multicriteria analysis is particularly useful during the formulation of a judgement on complex problems. That is the case with contradictory judgement criteria (e. g. jobs as opposed to the environment) or when a choice between the criteria is difficult. In general, this technique is used in ex ante evaluations of public projects and of their variations (the layout of a highway, the construction of a new Tools for making a judgement infrastructure, etc. ). More rarely, multicriteria analysis is applied to the intermediate or ex post evaluations of programmes. How should it be implemented? Multicriteria analysis is a tool for comparison in which several points of view are taken into account. It can be broken down into several phases described chronologically below. It is possible to repeat the phases and thus to make corrections. Step 1. Definition of the projects or actions to be judged: Inventory of the planned or implemented actions, or the elements on which the comparative judgement will be made. Step 2. Definition of a battery of judgement criteria: The evaluator or the commissioners of the evaluation will specify the criteria. Particular attention must be given to the definition of criteria, in order to be as exhaustive as possible and to define the question properly. The criteria must reflect the preferences of the decision-makers or the different points of view, so as to summarise and group together diverse characteristic dimensions used to evaluate an action. Step 3. Analysis of the impacts of the actions: When the projects and criteria have been defined, a quantitative estimation or a qualitative description of the impact of each project, in terms of these criteria, must be made. For this purpose short statements describing the different levels of impact may be used: "impact descriptors". Step 4. Judgement of the effects of the actions in terms of each of the selected criteria: Evaluating impacts by giving them scores may be done by the evaluator, based on quantitative data, or, more subjectively, by experts or the addressees of the evaluation themselves. Step 5. Aggregation of judgements: This combination may be based on three principles: • Personal judgements: the different judgement criteria are not synthesised in any way. Each of the addressees of the evaluation constructs her or his own personal judgement based on the analysis and uses it to argue her or his point of view. • Assisting coalition: a computerised technique sorts the actions in relation to each other. An action will be classified above another one if it has a better score for the majority of criteria (maximum number of allies) and if Principal evaluation techniques and tools it has less "eliminatory scores" compared to the other criteria (minimum number of opponents). • Assisting compromise: a weighting of the criteria is proposed by the evaluator or negotiated by the addressees of the evaluation. The result is a classification of actions in terms of their weighted score. The main option in multicriteria analysis is that of the involvement or not of the evaluator in the definition of criteria, in their evaluation and in their weighting. If the evaluator is actively involved in the analysis, the credibility of the results is undermined. On the other hand, when the addressees of the evaluation participate in the definition of the criteria, each partner prolongs the discussion until at least one judgement criterion is found that places her or his favourite action in first position. • Application to the Structural Funds As seen in the above example, multicriteria analysis is well suited to managing structural programmes in partnership and to their evaluation. The opinions of national and supranational members may be expressed jointly without losing any of their specificity or having to make too many concessions regarding their value scales. The Hainaut experiment was seen as a possible use of the technique by the Commission, in the evaluation of the Structural Funds. It produced encouraging conclusions as to its possible implementation. In particular, it served to develop a variation of the method called "multicriteria-multijudge" analysis, which enables each partner to construct her or his judgement based on the criteria and weights of her or his own choice. Strengths and limits in practice • Multicriteria analysis provides a framework in which the actors take part in decision-making and in problem solving. Its negotiated character and explicit to an treatment of judgement criteria help to give form unstructured reality. It thus allows the values and individual opinions of several actors to be taken into consideration, and the processing of functional relations within a complex network, in a quantitative way. • The intervention of an expert, the leeway left to the decision-makers and similarities with vote based methods make this technique a suitable tool for a partnership approach. Tools for making a judgement • The technique may help to define a coalition or a compromise, but it does not dictate the individual or collective judgement of the partners. • Multicriteria analyses are often preferred by decision-makers because they are readily involved in the process through a relatively simple technical framework, unlike cost-benefit analyses that artificially measure all the effects in monetary terms. • Specific problems of implementation may limit the use of multicriteria analysis, or require the presence of experts. • This technique is not always used in an interactive way, as it should be, and tends to fix criteria that are, in reality, fluid. Further reading Roy B. and Buyssou D. (1993), Aide Multicritère à la décision: Méthodes et Cas, Paris: Economica, 695 p. Very complete work. The most essential elements are marked, facilitating a reading at several levels. Saaty T. L. (1984), Décider face à la complexité, Paris: Entreprise Moderne d'Edition, 231 p. Book describing the technique, based on varied examples, with reflection on the decision-making process. Schärlig A. (1990, 2ème éd. ), Décider sur plusieurs critères, panorama de l'aide à la décision multicritère, Lausanne: Presses polytechniques et universitaires romandes, 303 p. Complete and highly accessible manual. Toulemonde, J. , Fontaine, C, Laudren, E. , Vincke, P. (1998), Evaluation in in partnership. Practical suggestions Evaluation, 4(2). London : Sage publication. The article describes the evalution of an urban development programme using a multicriteria-multijudge analysis. for improving their quality Vincke, P. (1992), Multicriteria Decision-aid, New-York : John Willey & Sons. Comprehensive presentation Tools for in-depth treatment of an evaluative question An in-depth treatment of an evaluative question is possible when this question concerns a single intervention or a small number of similar interventions, when the public concerned is homogeneous and when there are few effects to evaluate. Thus, if an evaluative question is sufficiently simple, it is possible to consider it in depth, that is to say, to collect primary data from direct addressees, to make an in-depth analysis of impacts and to seek comparable interventions to serve as a reference for judgement. In this situation, described in most evaluation manuals, almost all evaluation tools are applicable without any need for innovation. This section presents eleven tools, emphasising the conditions in which they can be applied to the evaluation of socio-economic programmes. Examples of evaluations with in-depth treatment of a question The examples presented below describe situations which are simple, either because the evaluation focuses on a single action or an homogeneous group of actions, or because it focuses on a single theme such as obtaining a result or a particular impact (for more information on thematic evaluations, the reader is referred to Volume 5 of the Collection devoted to that subject). When the object of the evaluation is simple enough, the collection of primary data, the analysis of impacts and the formulation of judgements can be carried out in greater detail. Almost all evaluation tools are applicable to this situation. The following three examples provide an illustration. In the first example, officials in a French ministry wanted a comparative estimation of the employment effects of several interventions. The evaluative question was as follows: "Are some employment measures more effective than others?" Box 19 - Evaluation of the impact of recruitment assistance in France In France, the Employment Ministry frequently conducts telephone questionnaire surveys on samples of addressees of the different employment assistance measures. Each of these surveys is used to estimate a deadweight coefficient and, where relevant, substitution or attraction coefficients. These coefficients are used to extrapolate monitoring data (gross effects) in order to estimate net effects. An expert panel met on several occasions to compare all the coefficients resulting from the various surveys, and to propose general rules for estimating deadweight effects, in relation to the nature of the assistance. In this example, the questionnaire survey technique was used for observation. The expert panel technique helped to consolidate the estimation of effects. The following example is drawn from an experiment carried out by the MEANS programme in Catalonia. An evaluation of assistance to co­ operatives was requested after an audit of the implementation of this assistance, which expressed doubts as to its effectiveness in terms of the creation and safeguarding of jobs. In this example, the observation tool was the questionnaire survey. Factor analysis made it possible to fulfil the function of comparing observations very successfully. Principal evaluation techniques and tools Box 20 - Ex post evaluation of assistance to Catalonian co-operatives In 1992 the Catalonian authorities developed a programme for assisting co­ operatives. They considered that they could have an important impact on employment, due to the fact that the demand for profitability was less than in privately owned enterprises. However, doubts remained as to the effectiveness of this assistance and the Department of Labour commissioned an evaluation of the programme. Because of the number of co-operatives, their diversity and the lack of knowledge about the addressees, it was decided to carry out a questionnaire survey on the entire targeted population, i. e. about 3,000 co-operatives. The answers obtained were treated by factor analysis in order to constitute a typology of co-operatives. This analysis helped to clarify the characteristics of those co-operatives that had created jobs, and of those that had not. The analysis showed that the co­ operatives which were most likely to transform the assistance received into the creation and maintenance of jobs, were those which were already established, were members of a federation of co-operatives, were not situated in the coastal area, were medium-sized, and so on. On the other hand, it also appeared that these co-operatives' investment was unrelated to the assistance. The third example describes work carried out to evaluate the effectiveness of a consultancy programme for SMEs in the UK. This evaluation used a large number of tools in the framework of a very complete procedure of observation in the field. The collection of data was carried out by means of two questionnaire surveys, a series of individual interviews and case studies. The function of estimating effects was treated by means of the comparison groups technique (by matching pairs). Examples of evaluations Box 21 - Evaluation of consultancy to businesses in the UK A programme financed by the British ministry of commerce and industry offered SMEs management consultancy, training, and information services. The impact of this programme could not be expressed in jobs created, since it was aimed at enhancing productivity which, in the short term, led to the destruction of jobs. It was therefore necessary to evaluate the programme in terms of other criteria such as productivity, profitability or investments induced. The method developed for the evaluation was as follows: 1. An initial questionnaire was sent to the 75 programme operators (service providers). This questionnaire concerned the consultancy services provided to the addressees, and the categories of businesses targeted. 2. Individual interviews with 20 operators were used to describe in greater detail the strategies developed by the operators. 3. Case studies were carried out for six operators. In these case studies, the particulars of the addressees were noted. 4. A sample of 300 beneficiary firms was constituted. Another sample of 200 non- beneficiary firms was also constituted, in keeping with the comparison group technique (matching pairs). 5. A second questionnaire was addressed to the beneficiary firms and to those in the matching group. A statistical comparison between the firms in the two groups provided quantitative indications on the effects of the programme, in terms of turnover, productivity and investments. In order to avoid "biased" statements, the questionnaires comprised numerous and partially redundant questions (triangulated questionnaire). The survey incorporated a search for deadweight effects. To recap, these three examples show that the evaluation tools were used mainly for collecting and analysing data. The following pages show that tools can also be chosen for structuring and judgement functions. Tools for structuring an evaluation In a situation of in-depth evaluation, the following tools can be used effectively for clarifying the effects to be evaluated and selecting evaluation criteria and indicators. The logical framework is a tool designed for the ex ante evaluation of development aid projects. It is used to rapidly clarify the logic underpinning an action, define expected impacts and construct impact indicators. METAPLAN® is a tool which is used extensively for group work. In evaluation, this tool is particularly well suited to clarifying and grading expected effects. The colour vote (described above) can also be applied, particularly for choosing evaluative questions. By contrast, tools such as the impact matrix or concept mapping are too cumbersome for situations characterised by the simplicity of the object to be evaluated. Tools for structuring an evaluation Logical framework The logical framework technique is both an exercise in the structuring of c o m p o n e nt elements of a project (or single programme) and a t e c h n i q ue for analysing t he internal and external coherence of the project. The product of this technique, the logical framework, is a formal matrix presentation of the internal f u n c t i o n i ng of t he A In-depth evaluation Θ Low technical complexity G Φ Low cost Θ Prospective use Use stabilised project, of the means for verifying the achievement of the goal, and of the endogenous and exogenous factors conditioning its success. Example: Logical framework of a local fishing project This example concerns a project financed by the European Union as part of its development aid policy. The global objectives (or aims) of the project are as follows: raising the standard of living, guaranteeing a more stable food supply, increasing export earnings, increasing employment, and using resources in a sustainable way. The specific objective of the project (or goal) is a higher income for the economic actors in the local fishing sector. To evaluate the achievement of this goal, a series of objectively verifiable indicators can be used: evolution of income from fishing or distribution of income by economic actor (fishermen, industries, transport firms, wholesalers, fishmongers), gender and fishing zone. Among the factors conditioning the achievement of these goals are, in particular, the need for sufficient quantities of consumer goods at affordable prices, as well as sufficient accessibility of other services (health, education, advisory services, etc. ). The expected results are: optimum production, optimum processing of fish, satisfactory commercialisation of fish production, and adequately defined local fishing zones. An evaluation of the achievement of these results may rely on a series of objectively verifiable indicators: number of catches, number of products sold, distance between the fishing zone and the shore, types of fish caught, duration of the fishing, balance in the distribution of catches between fishermen, duration of the sale, time between delivery and sale, depth of resources, and agreement between fishermen on fishing zones. Among the factors conditioning the achievement of the results is, in particular, the fact that demand has to be great enough, as does the level of prices for producers. Principal evaluation techniques and tools The following activities are part of the project: creation of fishing and fish processing co-operatives - particularly by women - making fishing equipment available, drying sheds and warehouses, organisation of transport, information on demand and the market, training - particularly for women - in new techniques, training in new packaging techniques, adequate information on potential fishing zones, negotiation of agreements between fishermen, organisation of a system of control over fishing activities, etc. Source: European Commission - DG VIII What is the purpose of the tool? In so far as it is a technique supporting the objective definition of a project, as well as its formulation in operational terms and its implementation, monitoring and evaluation, the logical framework facilitates: • the systematic, synthetic and comprehensible description of a project; • the articulation and clear grading of objectives; • the clarification of the conditions for the success of the project; • the clarification of the assumptions peculiar to the project; • the identification of the causal links between resources, results and objectives; • and, finally, the identification, during the planning of the project, of success criteria and means for verifying the achievement of objectives. is most effective when The technique team interaction and applied as soon as the problem that the project is supposed to solve has been identified. It will continually evolve and expand in relation to the environment of the project and thus, by extension will allow for the complete organisation of all the phases of the "project life cycle". it is prepared through The logical framework is a tool which facilitates the formulation of the key stages of a project: it is used to draw up a detailed budget, to allocate responsibilities, to plan activities, and to draw up the monitoring schedule and terms of reference of on-going and ex post evaluations. In which cases can it be used? Conceived about thirty years ago in the United States, the logical framework technique has progressively become the preferred methodological tool of development project planners. The approach is currently used in most Tools for structuring an evaluation national and supranational cooperation bodies, including the European Union. Its success owes much to its capacity to describe, in an ordered, causal and logical way, the internal functioning of a project in a given environment. It is this faculty which has enabled it to impose itself so rapidly in the field of development co-operation. The logical framework technique has proved to be ¡deal in the context of simple programmes where the objectives and main actors (from the sponsors to the operators) are clearly identified. How should it be implemented? The application of a logical framework to a project necessitates a complete analysis of its context, objectives and strategies. Step 1. Analysis of the context: The context is the general framework in which the project is situated. By analysing this context, the problems that the project must help to solve are identified. The following are required: • a definition of the political, institutional, social and economic environment (government policy, characteristics and prospects of the sector, etc. ); • an analysis of the motives and expectations of the beneficiaries and main actors; • an identification and grading of the problems to be solved and their probable causes. Step 2. Analysis of objectives: The problems that the project is aimed at solving lead to the definition of the project's objectives. Formulating or reformulating them consists of focusing on: • the prospects and the description of the future situation when the problems have been solved; is not merely discursive. • the grading of objectives in terms of goals and aims. This distinction between goals and aims It meets a methodological need in so far as the construction of a logical framework, in order to be coherent, requires the use of a limited number of main objectives (or aims) and specific objectives (or goals). The use of specific tools for grading objectives, of the "objectives tree" type, is necessary here. • the visualisation of the set of relations between resources mobilised and aims. Principal evaluation techniques and tools Step 3. Analysis of the logic of the project The logic of a project generally consists of four different "value levels": resources or inputs (e. g. human, financial, material or regulatory resources), products (outputs), main goals (specific objectives) and aims (global objectives). These levels are related to each other by assumptions of causality (of the "if. then. " kind). Whereas the first three levels are internal to the project, the fourth, that of aims, links the project to a broader programme or policy which is frequently defined outside of the sphere of management. Thus, the vertical logic, or intervention logic, is intended to clarify the intentions of the project while specifying the uncertainties related to their fulfilment. It serves to describe the logical articulation of the different "value levels" as well as the critical conditions influencing their realisation. These critical conditions, whether internal or external to the project, are the expression of the constraints needed to move from one "value level" to another in the logical framework. These moves, of which the causal links are expressed in the form of assumptions, have to be formulated. The horizontal logic is aimed at identifying and describing precisely that which is produced at each of the four "value levels", by means of two concepts: Indicators (OVI) and Sources of Verification (SoV) of these indicators. the Objectively Verifiable The OVI correspond to all the criteria likely to show that what was expected was also produced. They are, in a sense, the "operationalisation" of the declaration of intent of the intervention logic, thus providing tangible proof of success at each of the four levels: inputs, outputs, goal and aim. The SoV of these indicators are described by detailing the means and sources of information needed to obtain the required data. The inter-related nature of the elements of the logical framework can be presented schematically as follows: Tools for structuring an evaluation Box 22 - Basic elements of t he logical f r a m e w o rk Intervention logic AIM Global impact Raison d'être of the programme to which the project belongs Objectively verifiable indicators Sources of verification Critical conditions Measurement of the decrease in needs that have to be met Data from outside the project GOAL Specific impact Raison d'être of the project Elements of the success of the project from the recipients' point of view Data from outside the project OUTPUTS All that has to be produced in order to achieve the goal Indicators used to quantifiy and qualify the outputs and correlation with planned production dates Data from within the project INPUTS All the resources needed to produce the outputs and to achieve the goal Monitored inputs Data from within the project Causality assumptions "if. goal, then. aim" Causality assumptions "if. outputs, then. goal" Causality assumptions "if. inputs, then. outputs" Preliminary conditions before project launching Source: adapted from ACDI (1981) Guide d'utilisation de la Méthode du Cadre Logique dans la gestion et l'évaluation des projets de coopération de l'ACDI, Toronto: ACDI. Application to the Structural Funds The advantage of this tool for the evaluation of Structural Funds is its capacity for clarifying and structuring public action and its intentions. Its Principal evaluation techniques and tools contribution is further enhanced when the public action can be organised around a single main objective, taken here to be the aim. The logical framework technique is used in this case as a procedure for analysing overall coherence. Within a complex programme the tool helps to structure the objectives in the form of a tree. The trunk is the aim that branches out into sub- programmes, priorities, measures and projects. The advantage of this type of complete interpretation in steps is its simplicity, but it often proves to be ineffective for grasping complex realities. Strengths and limits in practice • As an analytical process for the planning and monitoring of projects, the logical framework technique has proved, in practice, to be a structuring frame that is rated highly by practitioners. It is a synthetic tool that clarifies links between means and ends, as well as exogenous constraints and the solutions that can be used to guard against their influence. The tool also facilitates the establishment of a monitoring and control system from the planning stage of the project. It is, moreover, used to delimit precisely the responsibilities of each actor involved in the execution of the project. • This technique provides a framework for examining the logic of a project, but those persons responsible for the project have to fill in the framework. This technique should therefore be implemented with the participation of the partners involved in the project. • The logical framework technique is more difficult to apply to cases of complex projects and programmes with multiple aims. • Although this tool clearly lends itself more readily to use in the context of ex ante analyses or monitoring. For ex post evaluations its contribution is one of complementing other, less systematic approaches that focus more directly on the main evaluation questions. • The presence of several decision-makers (sometimes with differing interests) may cause them to design a programme with a "vague" outline and mechanisms. In such cases only the presence of intermediate phases of collective reflection and the regular emergence of strong consensus can ensure the tool's potential is fully realised. Tools for structuring an evaluation • If the application of the logical framework technique is seen merely as a compulsory technocratic exercise in the planning of the project, its ability to reflect the evolution of the state of the project during its application will be strongly undermined. This could be harmful to all the documents obtained by extension, such as the terms of reference of an evaluation. Further reading Commission of 'Project cycle the European Communities Framework'. management : 'Integrated approach and Logical Brussels : Commission of the European Communities, Evaluation Unit Methods abnd Instruments for Project Cycle Management. Brochure presenting the project cycle management. (1993), Agence canadienne de développement international (1981) Guide d'utilisation de la Méthode du Cadre Logique dans la gestion et l'évaluation des projets de coopération de l'Agence canadienne de développement international, Toronto: ACDI. Reference manual of the Evaluation Division of the Canadian International Development Agency. Cracknell, B. (1989), 'The Logical Framework : A project Management technique' in The Crown Agents Review, N°3. Cracknell, B. (1989), 'Evaluating the Effectiveness of the Logical Framework System in Practice' in Project Appraisal, 4(3). Guildford : Beech Tree Publishing. Evaluation of two American and Canadian experiences dealing with the development of the Logical Framework technique. Swedish International Development Cooperation Agency, Department for Policy and Legal Services (1996) Guidelines for the Application of LFA in Project Cycle Management. Guidance manual on how to use the Logical Framework technique. Tools for structuring an evaluation M E T A P L A N® The METAPLAN® technique (a brand name registered by a German firm) is used for structuring group discussions. Originally designed to facilitate collective decision making in urban planning, it has spawned numerous variants. All are based on the same principles: asking participants in a meeting for their opinions, structuring and synthesising the information gathered, and grouping it together O Overall evaluation A In­depth evaluation Θ Medium technical complexity O <D Low cost O Retrospective Prospective use use Θ Use stabilised in a form on which consensus is reached by the participants, by weighting the ideas produced in the meeting. Example: In the context of the launching of intermediate evaluation of the Objective 2 programme in Franche­Comté (France), the first meeting of the evaluation steering group was primarily aimed at producing a list of the main impacts expected from the said programme. The METAPLAN® tool was used for this purpose. Outside consultants, who had been trained to use the tool, were responsible for planning the session. The participants (13 persons) were mainly administrative officials from local authorities, including a representative of the commissioner of the evaluation. The session, which lasted about two hours, was intended to reveal the main impacts expected from the programme, bearing in mind that these impacts had to serve as a basis for the evaluation and, in particular, the formulation of specific evaluative questions. The three questions that served as a basis for discussions in the METAPLAN® session were formulated as follows: "What do you think of the usefulness of the programme?" "On which impact would you like information?" and, finally, "Which two impacts are to be evaluated?" The latter question was used to grade the priorities. Owing to the way the process was structured, the steering group soon reached consensus on a list of five impacts which appeared as particularly important to evaluate: "Reskilling", "Adaptation of training", "Attractiveness of the region", "Reconversion / Diversification" and "Local initiative jobs". Source : Rapport C3E (1998) L'évaluation intermédiaire des interventions structurelles dans la zone d'objectif 2 de Franche­Comté Principal evaluation techniques and tools What is the purpose of the tool? METAPLAN® is used to group together and structure information based on multiple points of view, and to produce a synthesis acceptable to most of the participants in the meeting. This type of tool meets several types of need: extracting information and aggregating it by means of questions asked in an appropriate way, while respecting the viewpoints of all the members of the group; increasing the acceptance of decisions taken within the group because all points of view are taken into account, including those which are not dominant within the group. In which cases can it be used? This is a tool used for discussion rather than decision-making. It helps to identify the obstacles and resistance to the successful accomplishment of an action or strategy, and highlights the likely solutions to these problems. It is used to structure an action composed of multiple priorities. With this tool, it is possible to reduce tension within a group, particularly through the participation of all the stakeholders and through discussion and weighting techniques. Finally, this type of tool may be used in the construction of a common representation of the problems and solutions to be implemented, which can help stimulate a common will to act. How should it be implemented? The sequence of steps must necessarily be entrusted to an outside moderator who is neutral and familiar with the tool. It will be up to him or her to prepare and chair the meeting, and to present the results. Step 1. Preparation of the agenda: The agenda consists of three questions on which the session will be based. The formulation and articulation of these questions is important in so far as they must be sufficiently open to leave room for a wide range of answers, and must be expressed in such a way that the participants can and want to answer. The order in which the three questions are asked is also important. The first question or "introductory question" is used to illustrate the functioning of the technique and to establish a common system of reference. The second question or "card question" is the core of the approach: it is a more in- depth consideration of the first question. Its treatment will be more elaborate in so far as the answers will be discussed in more detail. The third Tools for structuring an evaluation question, known as a "weighting question", will be used to choose between the answers given, in order to establish priorities for action. These three questions must be prepared in advance by the moderator who ensures their appropriate articulation and the progressive nature of the procedure. Step 2. The work session: The meeting as such consists of a single work group. Depending on the size of the public, several sub-groups may be created during work on the second question (card question). It is, however, important that the work organised around the first and third questions (introductory question and weighting question) involve all the participants. The moderator poses the "introductory question" and asks for answers in the form of simple assertions that he or she writes down on Post-its (maximum 30). These are stuck onto a paperboard, forming "blocks" by grouping together answers that are similar or underpinned by the same logic. This initial question enables the participants to create a common representation of the problems, to give the same names to the same concepts, and to understand the method of discussion. The "card question" is the most important because it deals with the true object of the meeting and will serve as a basis for the group discussion. To answer this question, the participants are divided into groups of two or three persons and are asked to provide answers in the form of short sentences which one person in the group writes on the Post-its (3 to 5 per group). The moderator collects these answers and then, reading each one out aloud, displays them on the board, trying to group them into "blocks". The participants may contest the answers or ask for further details. In this case the moderator puts a "flash" sign on the Post-it concerned (the flash rule). Further details on the question may be provided and may be noted on a Post-it of a different colour stuck next to the flash. Gradually, with the participants' help, the moderator completes the "blocks" of answers (6 post-its per "block" at the most) which should be underpinned by a similar logic. The participants must then consult each other to find titles describing the content of the different "blocks". The " weighting question" will be used to show the importance that the different participants attribute to the different "blocks". The participants answer the question by pasting stickers onto those blocks that, in their view, provide the best answer. Principal evaluation techniques and tools The following is an example for these three types of question: 1. Introductory question 2. Card question 3. Weighting question "What do you expect from a training policy?" "On which specific points would you like the evaluation of a training policy to focus?" "Which elements should be treated as a priority by the evaluation?" The first question serves to launch the debate on the impacts of a training policy and to initiate reflection on possible typologies of effects. The answers given will also be used to guide the answers subsequently given to the framework q u e s t i o n. The s e c o nd question serves to consider in more d e p th the points addressed in the first question, but also to a dd information. The third question is used to select elements of answers that seem to be priorities. The following diagram summarises the last t wo phases of a METAPLAN®: Box 23 ­ Structuring an evaluation using the M E T A P L A N® technique Answers to the "card question" are noted on Post­it notes and placed on the paper board Answers are grouped together into "clusters" with a common logic. A title is given to each cluster The participants grade the clusters by weighting them with stickers Cluster Α Cluster Π π π EP π u □ π Tools for structuring an evaluation • Application to the Structural Funds METAPLAN® is a tool with several possibilities for use in the framework of a Structural Funds evaluation: It can serve to clarify the logic underlying actions and to identify measures that need to be considered in more depth. It is a structuring tool that may be used by programme managers during preparatory meetings before the launching of evaluations. METAPLAN® can also serve as a basis for the definition of evaluation plans and terms of reference. It facilitates the definition of evaluation questions (e. g. based on the questions proposed above) and, from these questions, the selection of evaluation criteria. The advantage of this tool is well suited to a pluralistic approach to evaluation in which the stakeholders are involved in the selection of evaluation criteria. This helps to increase the use of evaluation results since the criteria are accepted by a majority of stakeholders. Strengths and limits in practice • METAPLAN® is a structuring and formalising tool that promotes the appropriation of conclusions by those who have participated in the meeting, irrespective of the number of people. Thus, it facilitates consensus building. • This tool allows for the identification of points of conflict and makes it possible to overcome situations of deadlock which tend to appear in meetings. Since each into account and discussed, it is generally possible to avoid situations in which a few individuals dominate the entire group. idea proposed is taken • Whereas METAPLAN® is a tool that facilitates the formalisation of ideas, it is not a tool for creating new ideas. It expresses nothing more than what is in the minds of the participants during the meeting. If implicit agreement exists on solutions to implement (e. g. same idea), false solutions on which there is consensus may emerge. In other words, this type of tool can also promote deadlock. • METAPLAN® requires the presence of a qualified person if it is to produce sound results. The formulation and sequence of the three types of question are important and have to be prepared in respect of both Principal evaluation techniques and tools content and form. The same applies to the group leadership. In view of its apparent simplicity, it may be tempting to organise work sessions with this tool, despite a lack of information. In such cases the results of the work are likely to be disappointing and to have repercussions on the entire group. Further reading Schnelle, E. (1979), The METAPLAN®-Method : Communication Tools for Planning and Learning Groups. Hamburg: Quickborn Tools for observing changes in the field The questionnaire survey is the simplest observation tool. Administered through the post, by telephone or, more recently, by e-mail, it is used to collect numerous homogeneous data that can serve for statistical purposes. This tool is not suitable for ex ante evaluation. Several tools presented above are also suited to observation in the field, in the case of in-depth evaluation: individual interviews, focus group interviews and case studies. Finally, ethnographic observation is used to collect complete and detailed information in cases where the populations are difficult to reach or reluctant to participate in a survey. Tools for observing changes Questionnaire survey The questionnaire survey consists of putting a series of standard questions in a structured format to a sample of individuals who are usually selected as being representative of the population under observation. It may sometimes be exhaustive, covering the whole of this population. A In-depth evaluation S Medium technical complexity O Φ} High cost Θ Retrospective use Use stabilised The development of questionnaire surveys and sampling was based on the first precise forecasts of election results obtained during the inter-war period by opinion poll institutions. Considered as one of the basic tools of the social sciences, questionnaire surveys are today used by most public and private research, survey, statistics and evaluation institutions. Example: A survey carried out among employers on the effects of employment policies This example corresponds to that of Box 19. The French Ministry of Labour, Employment and Occupational Training frequently uses questionnaire surveys (generally telephone surveys) to evaluate interventions aimed at promoting employment. A survey was carried out in 1990 to evaluate a measure aimed at promoting the employment of the long-term unemployed, in the form of an exemption of social charges for employers. A questionnaire was submitted telephonically to the employers of a representative sample of about 1,000 jobless persons who had benefited from the measure. The sample covered only employers and unemployed persons affected directly by the measure. In each firm the person questioned had enough knowledge about the decision that had been taken to recruit the jobless persons and to give information about the role played by the offer of exemption from social charges. To avoid biased answers the employers were advised that the questionnaire would be treated anonymously. The questionnaire included several cross-related questions designed to reveal what would have happened in the counterfactual situation i. e. in the absence of the exemption. These included no additional recruitment, recruitment of a person with different characteristics, recruitment of the same person but in different conditions, etc. The survey results were used to estimate, for each firm, the level of deadweight (i. e. the number of unemployed people who would have been employed even without the exemption) and substitution effects (i. e. the extent to which the exemption Principal evaluation techniques and tools encouraged the employer to recruit an eligible jobless person rather than another non-eligible person). Observation of the entire sample provided an estimation of the average coefficients of deadweight« and substitution effects. Since the sample was representative, these coefficients were generalised to all the firms benefiting from the exemption, in order to estimate the jobs created by the measure. Source: "Aides publiques à l'embauche: quels effets sur l'emploi" - Premières synthèses, N°6, Mai 1990, Ministère du Travail de l'Emploi et de la Formation Professionnelle What is the purpose of the tool? In the evaluation context, questionnaire surveys are used mainly to collect addressees' opinions. The questions asked may be descriptive (e. g. How did you learn about the existence of the support?), normative (e. g. Does the service provided correspond to a priority need in your organisation?) or causal the evaluated intervention?). (e. g. What would you have done without In which cases can it be used? The advantage of questionnaire surveys is that they produce results applicable to the whole of the observed "population", either directly if the survey is exhaustive or by extrapolation from a sample. However, the minimum size of a sample that can be generalised in this way is around 1,000 units. A questionnaire survey must therefore be used in cases where the population to be observed is large and homogeneous (e. g. trainees, firms, travellers, etc. ). Questionnaires are most useful when the investigator has a precise and clear idea of what he or she wants to observe. In this case the simplest survey consists of closed questions to which a series of replies are given from a number of predetermined responses. There are also cases where it is useful to use questionnaires that allow responses to be directed, particularly when the investigator, prior to the survey, does not have clear ideas about the opinions of respondents or the way the intervention works. In this case a combination of open and closed questions can be useful. It must however be noted that the analysis of open questions is more complex and expensive since, once the primary data has been collected, the investigator has to determine, group and codify similar in which Tools for observing changes is particularly well suited to the production of responses. This tool descriptive information and to make classifications in different categories (e. g. a breakdown of assisted firms in terms of their size, sector of activity, age, nature of their market, etc. ). It can also be used to gather both qualitative and quantitative information. However, subjective data must be carefully interpreted when the evaluation tries to establish causality or provide normative information. A single sample (panel) may be questioned in two or more successive periods in order to provide longitudinal data allowing the monitoring of beneficiaries and the estimating of the sustainability of effects. For example, panel surveys are used to evaluate the sustainability of results obtained in the field of employment (e. g. trainees are questioned before and after their training. How should it be implemented? If a questionnaire survey is to be effective, it should consist of the following four steps: Step 1. Designing the questionnaire: The first step is to rigorously define the objectives of the survey and devise appropriate questions. Questions need to be stated clearly and formulated in an interesting way. It is important not to multiply secondary questions which could confuse respondents and/or interviewers thus undermining the quality of the survey. Where comparisons with other data sets are to be used, the terms used in a questionnaire must be carefully chosen so as to ensure their coherence. Most questionnaires are designed to ask primarily closed questions (choice between a limited number of answers), though many combine these with a small number of open (spaces to be filled in by the respondent themselves) or semi-open questions (a list of options with the answer "other" included to allow respondents to give other opinions/answers to those envisaged by the investigator). In the latter two cases however the need to codify answers after the data is collected may pose problems of interpretation and comparability if respondents raise very different issues from each other. However it is also often a useful ways of ensuring that issues overlooked by the investigator at the outset are not excluded from the eventual analysis if they emerge as being important for the addressees of a programme. It is essential to ensure that the formulation of the questions does not influence the answers. It should also be noted that questions requiring respondents to recall past events, situations and decisions are usually Principal evaluation techniques and tools 'double negative' questions. The considered problematic. As are importance of phrasing questions depends on the method that will be used. Thus in the case of postal surveys or other self-completion surveys it is absolutely vital to ensure that questions can not be misunderstood by respondents. This may be less important where surveys are administered face to face and the interviewer has an opportunity to clarify questions or prompt respondents where they fail to understand the question. The capacity to do this though will depend on the level of experience and understanding of the interviewers and, unless they are experienced and senior evaluators, it is unwise to rely on them to resolve ambiguities in question design. The questions may be nominal (e. g. Activity of your organisation: agriculture, industry, trade, other), ordinal (e. g. What did you like most in this training: its quality, price, relevance?) or numeric (e. g. How many days of training did you receive?). If the questions are normative or causal, it will be necessary to use triangulation. Triangulation consists of asking several similar questions that make it possible to define the limits of an answer that presents a risk of bias. For example, the following three questions allow one to estimate deadweight: Without the support, would you have made an investment of the same size, a little smaller, much smaller?. At the same date, a little later, much later?. With the same technology, with a different technology? Step 2. Sampling: The population to survey must be defined precisely (e. g. all firms with between 1 and 50 employees, which received assistance over the past 5 years) and available information on the structure of this population must be gathered. One must also check whether the population is known (names, addresses and telephone numbers) or unknown. When a sample is used, it constitutes a model of the entire population. In order to be representative, it must be large enough and must comply with precise statistical rules. The following indications are highly simplified rules that must be verified on a case by case basis by a professional. The standard size of a representative sample is about 1,000 individuals. The size of the sample does not depend on that of the population; a sample of 1,000 persons may represent a population of 2,000,000 persons or one of 6,000 persons. The size of the sample does, however, depend strongly on the required precision. One has to multiply the number of questionnaires by four to double the precision. Finally, the size of the sample must also take Tools for observing changes into account the heterogeneity of the answers. For the most important questions, an early estimation of the proportion of respondents who are expected to give a particular answer must be made. If 20 to 80% of the respondents are expected to choose one of the answers, a sample of 1,000 individuals will be enough to make the answer valid. If 5% or fewer respondents choose one of the answers, the sample will have to be bigger. The survey provides a valid figure for an answer if at least 100 respondents have chosen this answer. Several techniques are proposed in the following table for constructing a representative sample. Only the last two are applicable if the population is unknown or hardly known. Box 24 - Sampling tecl iniques Technique Principle Simple random selection Systematic selection Stratified random selection Selection by clusters By quota "Snowball" Each observation unit is selected randomly. It has an equal chance of being part of the sample and can be selected only once 1 individual out of 10 or 100 is chosen; the first individual is chosen randomly the population When particularly homogeneous, it is divided into homogeneous sub­ groups and a sample is then taken in each stratum thus constituted not is The population is divided into sub-categories or clusters (e. g. Firms assisted by the municipality); some clusters are chosen randomly and then all the individuals in a cluster are questioned A reduced model is constructed, consisting of the known characteristics of the population under study; the researchers then have to find the individuals corresponding to these characteristics Based on a sample with a limited number of persons, more units, related to the first ones, are added Step 3. Pre-test: A pre-test or pilot of the questionnaire, on a limited number of respondents, is essential. It provides a critical view of the form and content of the questionnaire, thus enabling one to assess its relevance. The final version of the questionnaire will then be drawn up. This is Principal evaluation techniques and tools particularly important in the case of self-completion surveys and those which involve complex answers or 'filters' (ways of steering respondents through a survey to those questions which are relevant to their experiences - thus for example ensuring that only the users of service give opinions about it and that those who have no basis for making judgements do not). Step 4. Administration of the questionnaire: The questionnaires may be sent by mail or Internet or distributed by hand - for example at a particular event which the researcher is seeking feedback about. In these cases surveys are self-administration or self-completion questionnaires. In others they may be completed with the help of an interviewer, for example being carried out over the telephone or face-to-face. In the case of self- administration a low rate of response (sometimes 20% for answers to a first batch) must be expected. The rate of response may be improved by reminders and, where relevant, by information in the specialised press. Postal sun/ey responses may be boosted by providing respondents with pre-addressed and postage paid envelopes. Depending on the difficulty of the context, a high rate of response ranges from 60 to 90%. In case of repeated surveys on the same panel, a loss of around 10% per year is considered inevitable. Potential respondents are more likely to participate if the purpose of the survey is explained to them in a way that demonstrates the value of their contribution. Where feasible respondents should be given a guarantee of the confidential treatment of her/his answers. The interviewers must be informed of the "rules of conduct" and be able to make this clear to interviewees. Respondents may also be offered incentives to participate and to return forms promptly. Many survey organisations offer the chance to enter a prize draw or a similar incentive. This is less common in the public sector, but has been used in some contexts. In some cases it may be useful to offer respondents the opportunity to receive a summary of the evaluation report. It is good practice to follow up respondents with a letter thanking them for their assistance. A random check is generally carried out after the survey to check that the interviews actually took place. Step 5. Codifying the data: The next stage is to codify and record the answers - almost always in computerised form for subsequent analysis. At this stage adjustments are generally made into account imperfections in the sample, and especially to correct the bias introduced by non-answers. take to Tools for observing changes Step 6. Interpreting and disseminating the results: The final stage is to conduct an analysis of the meaning of the results of the survey. This involves understanding the purpose of the survey and it limitations - for example constraints related to sample size, the reliability of the responses and so forth. It is therefore important that the analysis phase is led by an experienced researcher who understands these problems and was involved in the survey design at the outset. For the survey to be of any practical use, its results have to be presented in a way that is accessible and meaningful to policy makers. Numerous specialised computer programmes can help to undertake this step. It is usually important therefore to ensure that the final report provides sufficient details of the way in which the survey was performed (provided in a technical annex for those who wish to check its validity) but restricts reporting in the main text to a straightforward and clear explanation of the findings. Application to Structural Funds The questionnaire survey is suited to the observation of the results and impacts of an intervention affecting a large and sufficiently homogeneous population. It is therefore likely to be a tool reserved for the retrospective evaluation of a simple intervention, for example, the evaluation of training for the unemployed. However, it may also play an important role in 'before after' analyses where it is used to establish a baseline at the start of a programme against which progress is measured during or after the implementation of an initiative. Strengths and limits in practice • The questionnaire survey has the advantage of producing structured, quantified and generalisable information, providing the sample has been properly selected. It is therefore particularly useful for quantifying indicators, including monitoring indicators. • This technique is costly since a sample has to contain more than 1000 units to ensure that the information provided is statistically significant. On the other hand, it allows a wide population to be reached at a relatively low cost. • It is sometimes difficult to know the population to be questioned, especially if the population is large and not particularly well identified by the monitoring system. In this case the tool loses one of its main advantages: the generalisability of the observations. Principal evaluation techniques and tools • The need to standardise responses through closed questions may lead to simplification. For this reason the questionnaire is usually reserved for situations where the effects of the intervention have been identified beforehand and are clearly understood. By contrast the impacts of a highly innovative action would be best evaluated through open questions and a variety of other non-survey based techniques. • The quality of a questionnaire survey may be undermined if some answers are wittingly biased by respondents who are usually in favour of the intervention from which they benefit. • Research has shown also that many people answer questions in an attempt to be polite or avoid appearing ignorant when they in fact do not know the answer or feel unable to give a judgement. To be reliable a questionnaire should therefore consist of questions that all respondents interpret in the same way and can answer accurately. Further reading Dussaix A. -M. , Grosbras J. -M. (1996), 'Les sondages: principes et méthodes. ' collection Que sais-je? Paris: Presses Universitaires de France Practical and accessible, this book contains the most essential information on polling techniques and practice. Fink A. (1995), 'The Survey Handbook' Sage A complete and practical guide. Grange D. , Lebart L. , ed. (1994), 'Traitements statistiques des enquêtes' Dunod A large body of technical and statistical information, provided in an accessible form. Henerson M. E. , Morris L. L. , Fitz-Gibbon C. T. (1987) 'How to measure Attitudes?' Sage Covers the construction and use of scales of measurement. Javeau C. (1992) 'L'enquête par questionnaire, manuel à l'usage du praticien', Edition de l'Université de Bruxelles A practical guide in fifteen steps. Tools for observing changes Ethnographic observation The e t h n o g r a p h ic o b s e r v a t i on technique consists of observing the daily activity of the actors w ho implement the programme, or the beneficiaries w ho enter into contact with the programme. Thus, the observer is present during the selection of projects or participates in a training course, for example. He or she thus tries to observe and A In-depth evaluation 0 Medium technical complexity O Φ Medium cost © Retrospective use Innovative use understand the situation "from the inside". This technique is particularly useful when one has to observe a situation about which there is little knowledge, or when it is suspected that the same situation is understood very differently depending on whether the point of view is "external" or "internal". Ethnographic observation is the only technique that allows observations to be made in fields where overt research activity is a serious problem. The technique was born over a century ago, with the science of ethnography. Example: Evaluation of a public housing service Ethnographic observation was applied to the evaluation of social housing policy in France. The aim was to analyse the discrepancy between the national objectives of this policy and its implementation in the field by civil servants in contact with the public. For several months the evaluator accompanied teams of civil servants employed by the agencies managing housing for these authorities, he visited these homes when the tenants arrived and left. These visits were intended to judge the state of the housing and to determine the amount of reparation work to be paid either by the tenant or the managing agency. low-income groups. Together with The position as an observer enabled him to "melt into the background" and to observe about fifty situations in which the national policy was applied in concrete situations by the civil servants. The situations observed were recorded and about twenty typical sequences were selected. These sequences were written up, with due respect to confidentiality, and then presented to policy actors in meetings organised separately at different levels (civil servants in the field, heads of the agencies, national authorities). The evaluation helped to show how local authorities apply the rules, how the tenants interpret them, and how the latter influence their application. Principal evaluation techniques and tools The evaluation highlighted the fact that tenants had difficulty understanding the reasons for the rules. In view of these results, several amendments were made: the heads of the agencies revised some rules and initiated a review of the recruitment and training of their agents. Source: Philippe Warin, " L'évaluation des usagers au cours des relations de service ", Les annales de la recherche urbaine n°47 (1990). What is the purpose of the tool? Ethnographic observation serves to collect in-depth information on a few typical situations in the implementation of an intervention. It provides detailed insights into the effects of the intervention and the influence of the context, and is sensitive to the viewpoints of the key actors and the beneficiaries. In which cases can it be used? Ethnographic observation has been used to understand the functioning of policies in education, health (e. g. diagnosis, care), justice, police, scientific research, urban transport, etc. In the evaluation framework, this technique is recommended particularly for the observation of processes of interaction between the administrations and their public. In the example used above to present ethnographic observation (the evaluation of a public housing service) the researcher also carried out an ethnomethodological analysis. Ethnomethodology appeared in the 1960s and developed in the early 1980s in Europe. It studies the practical daily reasoning used by individuals (civil servants, beneficiaries, users, etc. ) to judge the situations they encounter and the collective actions in which they are involved. If carried out properly this technique does not disturb anyone or influence his or her behaviour. It can therefore be used to observe the spontaneous behaviour of populations who are reluctant to accept the formalism of questionnaire surveys. It is the only technique applicable when there are serious difficulties of gaining access to the field, for example in the case of conflict within the organisations responsible for implementation or when the beneficiaries' behaviour is partly illegal or irregular. The technique is then particularly interesting if the evaluation is launched with a view to amending the regulations suspected of being ineffective. Tools for observing changes Due to its inductive character, ethnographic observation serves to observe the results of an intervention of which the functioning is not well known. It is particularly useful when it is suspected that the reality is not perceived in the same way by those who implement the intervention and by the public who receives it or benefits from it. On the other hand, the technique is time-consuming and since the observations focus on a few restricted situations they cannot easily be generalised. The tool is therefore best suited to the evaluation of an requires extensive and relatively homogeneous considerable skill on the part of the researcher to absorb and reflect accurately the behaviour of the key actors, and it may take time for the researcher to 'melt into the background' and therefore for participants to behave in a normal way. intervention. It also How should it be implemented? Observation has generally been carried out in the following steps: for Step 1. Choice of situations for observation: The situations observation are defined beforehand in relation to the preoccupations of the evaluation commissioners. They consist of situations of interaction or of negotiation between the public actors and the beneficiaries of the evaluated policy. Step 2. Observation: The observer follows and registers the course of the interaction, taking care to disturb the behaviour of the actors as little as possible. This work consists of note-taking and audio-visual recordings (as discretely as possible). This step cannot be limited merely to a simple observation; it must be completed by an institutional analysis so as to identify the areas of constraints bearing on the relations between the actors. The observer must collect a maximum of information and capture a particular, inside view of the situation. Stem 3. Identifying and cutting up typical sequences: The events observed during the second step are analysed in terms of characteristic sequences. Each recording is 'cut up' just as one would edit a film into sequences. The observer identifies the "evaluative assertions", that is to say, the sentences which convey an explicit or implicit value judgement. Typical Principal evaluation techniques and tools sequences and their analysis are concentrated on these assertions, and reveal the way in which the policy is judged in the field. Used in this way, the tool can shed important new light on the validity and effectiveness of the policy. Step 4. Analysis of typical sequences with the actors: The typical sequences and assertions are rewritten or modified to make them anonymous. They are then given to the people representative of those that have been observed, for the purpose of collecting their comments and reactions. This step serves to verify that no bias has been created by taking the sequences out of their context. It gives, for each sequence, keys for interpretation which are recognised and validated by the "community" under study. Application to the Structural Funds To our knowledge, ethnographic observation has been used very little in the context of the evaluation of socio-economic programmes. Yet this technique is well suited to certain Structural Funds interventions, for example those intended for publics that are difficult to observe by means of more traditional inquiry techniques (e. g. the long-term unemployed). Ethnographic observation is a generic tool that involves the collection, interpretation and comparison of data. It shares these characteristics with the case study technique. It is therefore particularly well suited to the analysis of the effects of an intervention that is innovative or unfamiliar, and especially the influence apparent success or failure of the community interventions evaluated. the clarification of confounding factors that Strengths and limits in practice • Ethnographic observation serves to reveal the discrepancy between the way in which public intervention is understood high up at the decision­ making level, and the way in which it is understood in the field; it highlights the interpretation made of it by individuals in an operational situation. • The observation is limited to a small number of situations. Generalisation is therefore possible only if the intervention is sufficiently homogeneous. • It is based on spontaneous information, gathered by an independent and experienced observer. The reliability of the observation depends to a large extent on the professional know-how of the observer-analyst. Tools for observing changes • Despite its advantages, ethnographic observation requires meticulous preparation to enable the observer to fit into the observed context without disturbing anyone, as well as considerable time for data collection because the observer has no control over the schedule of her/his observations. • This technique allows data to be collected in difficult situations where other survey techniques cannot be used. Further reading Coulon A. (1987), L'ethnométhodologie, Paris: PUF, Que sais-je? An approach which is both theoretical and practical, illustrated by case studies. Garfinkel H. (1967), Studies in ethnomethodology, Englewood Cliffs, NJ: Prentice Hall A return to the origins of ethnomethodology. Jorgensen D. L. (1989), Participant Observation, a Methodology for Human Studies, London: Sage Publications, Applied Social research methods Series, No 15, Complete manual on a variation of ethnographic observation in which the observer puts her/himself personally in the situation of an actor or addressee of the programme. Tools for analysing data When an evaluation concerns a limited domain, it is possible to carry out an in-depth analysis of the causal links between the action and its effects. Several tools may be used in this context: • Variance analysis (not presented in this Volume) and factor analysis are used to reveal similarities and dissimilarities within a sample of observations, to create typologies, and to identify exogenous factors associated with the production of the impacts. • The Delphi survey is included in this section because it was designed for estimating impacts. It is well suited to ex ante evaluations that rely on secondary data. The tool mobilises and analyses data through the intervention of experts. It is therefore similar to the expert panel that is also suitable for data analysis. • Comparison groups are used to estimate net effects by noting the difference between a group of addressees and a group of non- addressees. • Regression analysis is used to estimate net effects and to determine whether the causal links between the intervention and its effects are statistically significant. • Case studies, group interviews and ethnographic observation mentioned above as tools for observation, are also tools which can be used flexibly for comparing and analysing data. It is even possible to estimate effects (in the form of a range, from minimum to maximum effects) by means of case studies carried out on a selection of projects. Tools for analysing data Factor analysis Factor analysis is a multidimensional statistical analysis technique used to sort large volumes of quantitative data concerning groups of individuals about whom there is information about numerous characteristics (variables). The tool identifies similarities or links (factors) between individuals and between variables. It enables one to "trade off Λ In­depth evaluation S Medium technical complexity Φ Medium cost Θ O O Use stabilised Retrospective use Prospective use information for legibility", by producing a simplified graphic representation of available data, showing the phenomena to be analysed. Although its origins can be traced back to the beginning of the century, its recent development can be attributed to significant advances in data processing, owing to computer technology. An example: Employment impacts in the Catalan co­operative sector This example corresponds with Box 20. The use of factor analysis can be illustrated with respect to an ex post evaluation project, oriented towards the employment effects (creation and maintenance of jobs) of subsidies in the sector of Catalan co­operatives. Evaluation in this case was complicated by the diversity of co­operatives and the consequent heterogeneity of the population. The problem was twofold: first, the relevance of measures taken by the region had to be determined in view of this situation; secondly, the importance of impacts on those co­operatives which received assistance had to be assessed. Given the variety, as well as the lack of knowledge of the population and of impacts, factor analysis was chosen for its ability to reveal the main characteristics of the population under consideration (about 3,000 co­operatives). The first step in the evaluation consisted of sending a questionnaire to all the co­ operatives. This was completed through interviews. The data thus gathered were first analysed using traditional statistical methods (search for the main characteristics of the population, drawing up of cross­referenced tables). The second step in the data analysis consisted of a factor analysis with a view to isolating variables likely to explain variations in employment in the co­operatives: including the size of the entity, age, sector of activity, technological level, etc. Source: Comité Assessor para l'Estudi de l'Organització de l'Administració, Generalität de Catalunya, 1995 Principal evaluation techniques and tools What is the purpose of the tool? Factor analysis can serve various purposes. In the evaluation context it is used to: • analyse in detail the reactions and characteristics of the beneficiaries, and thus gain a better understanding of the relative efficiency of the interventions; • explore the correlation between phenomena and thereby identify possible confounding factors that have amplified or reduced the impact of the programme. In which cases can it be used? Factor analysis is a data processing tool used by national statistics offices and survey institutes. It has the advantage of processing quantitative data in such a way that typologies can be constructed. The tool is therefore very useful when the evaluation has to construct classes within the population concerned (e. g. : qualification, socio-professional category, criminality, urban concentration, etc. ). It is important to note that the classifications established by factor analysis are not chosen randomly by the evaluation team. On the contrary, they are drawn from a vast corpus of data, and constructed in such a way as to allow the best discrimination possible within these data. How should it be implemented? Step 1. Constitution of a table of data: the data processed are provided by monitoring information or, more commonly, from a questionnaire survey. All the data collected are grouped together in a data matrix, like that resulting from each survey or census. A row corresponds to the data relative to a person (firm, etc. ) and a column groups together values of a single variable for different individuals. Steps 2 and 3. Choice of a space of individuals and a space of variables: These two steps are fairly similar. In both cases the sample is examined, extreme or incorrect values sought, and the variables and groups of persons relevant to the analysis extracted. It is here that the analysis is oriented towards those variables on which the study is to focus. Step 4. Multi-dimensional analysis: This analysis is a step that is facilitated today by the existence of specialised statistical software. It is, Tools for analysing data however, a crucial one, particularly because several technical choices must be made in relation to the available data and desired results. In this step the number of observations will be reduced and relevant factors identified. These will then be used for differentiating the population. Step 5. Projection into two (sometimes three) dimensional space: Statistical software produces part of its results in the form of a card which may feature individuals and/or variables. Box 25 - Example of result of factor analysis Example of the evaluation of aid to Catalonian cooperatives u ω xxxx xxxx xxxx Cooperatives which. invest. , receive aid y''' are medium- sized ι XXXX X3CX < xxxx are far from the coast have been In operation for years factor 2 ,'xxxx , xxxx *»„ ', ' xxxx xxxx Step 6. Interpretation of results: factor analysis does not reveal the significance of the factors nor that of the groups formed on the map. The analyst has to interpret them and to name them. Ideally, the key stakeholders in the evaluation should be associated with this step, since it provides an opportunity for them to enhance their understanding of the functioning of the intervention. Application to the Structural Funds Factor analysis may be used as an ex ante evaluation tool, for example for identifying typologies of the target population, based on the population as a whole. It can thus be used to set up procedures for selecting beneficiaries or for identifying the nature of the socio-economic problems facing this population. Factor analysis may also be used to highlight the similarities and differences between groups of individuals or geographical areas. This is particularly Principal evaluation techniques and tools useful for making comparative analyses of regional development or for evaluating the reactions of the beneficiaries of specific actions. On the other hand, factor analysis can not be used to estimate socio­ economic effects. It is above all a tool for data comparison and inductive analysis. Strengths and limits in practice • Factor analyses are a tool for clarifying raw material resulting from large- scale surveys because they help to reveal relevant dimensions (factors explaining the differentiation or resemblance of a group of data). • The results of factor analysis are easily accessible and are presented very clearly (diagrams), aggregating a large number of data. • Although factor analysis is used to illustrate the existence of groups of variables around certain categories of the population, it is not a tool that can explain the existence of cause and effect. • Factor analysis requires access to large amounts of data, which may prove time consuming and costly to collect. • Factor techniques require specific qualifications and methodological precautions. Used wrongly, they could provide tautological results. • Factor analysis cannot be conceived without the use of a large number of data, which may prove long and costly to collect. Further reading Bry X. (1995), 'Analyses factorielles simples', Paris: Economica, 11 l p. Although this book is of general interest, it is preferable to have some knowledge of statistics. Dazy F and Le Barzic J. F. (1996), 'L'analyse des données évolutives, méthodes et applications', Paris: Technip Written form of a training seminar, requiring sound basic knowledge in statistics. Grange D. and Lebart L. (1994), 'Traitements statistiques des enquêtes', Paris: Dunod A book which focuses on the essential points. Tools for analysing data Jaccard, J. (1998), Interaction Effects in Factorial Analysis of Variance, London: Sage publications. Kim, J-O. , Mueller, C. W. (1979), Introduction to Factor Analysis, London: Sage publications. Rossi, P. H. (1997), 'Advances in quantitative evaluation, 1987-1996', in New directions for evaluation, Winter 1997 (76). San Francisco : Jossey- Bass Publishers. Advances in data collection, measurement and statistical analysis which are appropriate to the evaluators' needs. Tools for analysing data Delphi Survey Evaluations that rely on the views of experts may be carried out in the framework of ex ante evaluation. The Delphi method is a case in point. It is presented in the form of an anonymous and iterative consultation procedure by means of postal surveys. The absence of prior consultation within the group allows for the A In­depth evaluation Β High technical complexity G φ) High cost © Prospective use Innovative use clarification of differences of opinion between experts, and thus for a better understanding of the nature of each person's expertise. The method is recommended when the questions posed are simple (a programme with few objectives, of a technical nature) and for the purpose of establishing a quantitative estimation of the potential impacts of an isolated intervention (e. g. increase in taxes or in the price of energy). Example: AIDS vaccination policy: a scenario analysis using the Delphi method The Delphi method was used to examine likely answers to the introduction of a first AIDS vaccination in Switzerland. Thirty persons concerned by this vaccination strategy contributed anonymously to the study. The study was based on a scenario developed beforehand, describing the characteristics of a first preventive, partially effective, vaccination against AIDS. The Delphi consultation was carried out in three stages: • list the objectives to be achieved in the first five years, • evaluate the acceptability and feasibility of proposals concerning the development of a public health strategy and the AIDS vaccination, • estimate the potential use of the vaccination by different groups of users. These results allowed for a) significant conclusions to be drawn on the strategies and recommendations to be developed in the framework of AIDS prevention campaigns and b) the provision of a frame of reference for the setting up of a future AIDS vaccination strategy. Source: Zuber, P. (1994) Introducing a first AIDS vaccination in Switzerland: A Delphi Policy Analysis. Lausanne: Institut de médecine sociale et préventive. Principal evaluation techniques and tools What is the purpose of the tool? Like the other expert-based approaches, the Delphi survey is useful for producing, in the evaluation context, recommendations of which the credibility depends on the legitimacy of the experts themselves. It may be used in forward planning to establish hypotheses about the realisation of events and on their socio-economic implications. Above all, the method serves to shed light on the evolution of a situation, to identify priorities or to draw up prospective scenarios. In which cases can it be used? This technique may be used when significant expertise exists on the subject, i. e. in the case of programmes that are not innovative. It is also recommended in an ex ante evaluation context if the evaluation concerns public intervention of a technical nature. Thus, it was very often used in the framework of energy policies, for example, for prospective studies on the impact of changes in taxation. It may be used to ensure that experts do not influence one another. Anonymity and the absence of interaction favour an in-depth analysis of the reasoning behind the opinions expressed. How should it be implemented? The approach consists of questioning the experts by means of successive questionnaires, in order to reveal convergence and any consensus there may be. The main stages of this process are: Step 1. Determination and formulation of questions: Particular care must be given to the choice and formulation of questions, so as to obtain useful information. Step 2. Selection of experts: They must have specific knowledge on the subject and be prepared to become involved in this type of procedure. The panel is generally composed of about fifty persons. Step 3. Formulation of a first questionnaire that is sent to the experts: The first questionnaire must contain a reminder of the nature of the study and include two or three semi-open and open questions. Tools for analysing data Step 4. Analysis of the answers to the first questionnaire: The answers are analysed in order to determine the general tendency and the most extreme answers. Step 5. Formulation of a second questionnaire that is sent to experts: Each expert informed of the results of the first round is asked to provide a new answer and to justify it if it differs from the general tendency. Step 6. Sending of a third questionnaire: This questionnaire is intended only for those experts whose answers were "extreme". They are asked to criticise the arguments of those who supported the opposite point of view. The comparison of opinions has a moderating influence and facilitates the appearance of convergence between the points of view. Sufficient convergence of opinions generally appears with the fourth questionnaire. If that is not the case, the cycle continues. Step 7: Summary of the process and drawing up of the final report. Application to the Structural Funds In the case of the Structural Funds, the Delphi inquiry is recommended for obtaining macro-economic estimations when the phenomena involved are complex; for example, to quantify the impact of a major infrastructure project. It may also be used to specify relations of causes and potential effects in the case of innovative interventions. It is particularly useful when a very large territory is being dealt with since there are no experts' travel expenses, only postage costs. Strengths and limits in practice • Unlike the panel approach, the conclusions drawn from this type of approach facilitate the taking into account of minority points of view. The method furthermore allows for a better understanding of the elements on which the experts' opinions are based. • The use of this technique has limited flexibility because the various iterations are time-consuming (over three months in all). • As in the case of expert panels, the reliability of the Delphi survey depends on the subjectivity of the experts' opinions, itself a function of their knowledge. Principal evaluation techniques and tools Further reading Godet M. (1985), Prospective et planification. Paris: Economica, pp. 119- 125. The Delphi method is presented and illustrated in the first part of this book devoted to forecasting. Nadeau M-A. (1988). L'évaluation de programme. Laval, Québec: Presses de l'université de Laval, pp 349-352. A manual presenting the methods and instruments used programme evaluation. The Delphi method is described briefly. in Witkin B. R and J. W Altschuld (1995), Planning Conducting Needs Assessments, Thousand Oaks: Sage, pp 193-203. Collection of methods to be used in the evaluation of needs. A part of the chapter on group techniques is devoted to the Delphi method. Tools for analysing data Comparison groups The comparison group technique involves comparing a "participant group" (e. g. SMEs benefiting from services) with a similar group which has been unaffected by the intervention. Ideally, the two groups should be identical in all respects except for their participation/non- participation in the programme. Where this applies, the group that A In-depth evaluation 0 Medium technical complexity Θ © High cost O Use stabilised Retrospective use has not been affected by the intervention is referred to as the 'control group' and fulfils the same function as a 'control' in the natural sciences. In practice however it is rare to find groups which are identical in every respect and in the social sciences and impossible in the context of European Structural Funds, which is why it is not considered in this document. A modified version of the approach is to use the "matching pairs" technique. Example: Estimation of the impact of a training and consultancy programme in firms in the UK This example corresponds to that of Box 21. The TEC (Training and Enterprise Council) programme in England was evaluated by means of the comparison group technique. The effect of the programme was estimated through a "before and after" comparison between firms using the programme services and a set of matched pairs of companies which had not participated in TEC programmes. Matches were made according to the following criteria: type of activity of the firm, size of the firm in terms of number of employees, age and geographical location. A total of 180 pairs of beneficiary and non-beneficiary firms were questioned. The report examines both the practical and theoretical difficulties of the comparison group techniques. The main practical difficulty identified was that of matching the pairs. Theoretical problems related to the fact that, even if the pairs linked together were apparently exact matches, there remained imperceptible differences between them which might account for differences in performance which were unrelated to the programme being evaluated differently. Despite these difficulties, the comparison group technique made it possible to define norms for other regional or national factors that had influenced the results of the firms. Two data collection techniques were applied. One consisted of a questionnaire on the employment impact of the TEC. To avoid results being biased by 'instrumental, Principal evaluation techniques and tools self interested "strategic" responses' (e. g. a positive reply given in the hope of obtaining additional support), firms were asked not only to estimate the employment effects of the TEC services, but also to give details of their turnover and changes relating to employment. This was designed to enable objective comparisons between the matching samples. Source: MEANS Handbook N°3, pp. 41-42 What is the purpose of the tool? The aim of the comparison group technique is to estimate the net effects of a public intervention, that is to say, the effects directly attributable to this intervention, and not those which are the result of exogenous or confounding factors. In which cases can it be used? The comparison group technique is usually used when the programme offers subsidies or services to a relatively homogeneous group of beneficiaries and when these beneficiaries have already been reached (retrospective situation) and there exists a similar group which has not been offered or has not taken up assistance from the programme which is being evaluated. To produce credible results, the two groups being compared must be sufficiently similar in respect of likely 'confounding factors'. This means that one can identify and grade these factors before constituting the groups to the compare. The tool mechanisms producing effects are already fairly well known. is therefore reserved for situations in which The comparison group technique does not require the use any specific data collection technique. In fact several approaches to primary data collection (questionnaire surveys, interviews, etc. ) and/or secondary analyses are equally applicable. How should it be implemented? Step 1. The construction of comparison groups: Three ways of constructing comparison groups exist. • Creation of matching pairs by selecting, for each individual in the beneficiary group, an individual not affected by the programme but as Tools for analysing data similar as possible in all other respects. The difficulty is finding individuals not affected by the programme. It is for example likely that a firm operating in a similar market to a participant of the programme may have been affected adversely by assistance given to what is, in effect, one of its competitors. Equally, \jery little is likely to be known about non- addressees, and since checking out their characteristics and finding individuals to fit a specific profile is likely to be difficult and long, the creation of matching pairs may be expensive. • Comparison with individuals known to the programme but not benefiting from it (candidates who withdrew their application, candidates not selected, selected candidates who dropped out of the programme midway). In this case, the group of non-beneficiaries must be adjusted to resemble the group of beneficiaries as closely as possible. However, a bias remains, because there are likely to be reasons that caused them to withdraw or to be rejected which may also affect their performance as a whole. • A generic comparison may be made with a hypothetical 'comparison' group based on national statistics. This is done when the programme has affected only a small group of beneficiaries very specifically, and the rest of the country has not benefited from similar programmes. Step 2. The actual comparison: The principle consist of comparing the changes observed in the two groups and then deducing, from the difference, the net effect of the programme. Different disadvantages, and varying degrees of validity. forms of comparison exist, each with its advantages and • An "afterwards only" comparison involves analysis of the differences between the two groups after the subsidy or service has been received. Its advantage is its simplicity. The disadvantage of this approach is that it disregards differences that existed between the two groups at the outset. Participation in the programme Situation after the programme Participant group Comparison group yes no X2 X'2 Estimation of the effect X 2 - X '2 Principal evaluation techniques and tools A "before-after" comparison avoids this problem by taking into account possible differences between the two groups before as well as after the intervention. It therefore focuses on the relative improvement of the beneficiaries compared to non-beneficiaries. Its drawbacks are that it requires the availability of data on the situation before the application of the treatment (e. g. through monitoring or in the form of statistics), and such data does not always exist. Moreover, there are situations in which the evaluation does not start until the treatment is underway and in these cases the 'before' state may not be known. There are also many programmes in which the beneficiaries and non-beneficiaries are not known at the outset and information about their differences prior to the programme are not available. In all these cases data about the 'before' situation have to be gathered retrospectively and are therefore unlikely to be entirely reliable. Participation in the programme Situation before the programme Situation after the programme Participant group Comparison group Estimation of effects yes no (X2-X1) ( Χ ' 2 - Χ Ί) X1 ΧΊ X2 X'2 • The "before-after" trend comparison is considered more reliable. The performance over time of both groups is compared in order to test the extent to which beneficiaries display greater improved performance relative to their own previous situation than do non-beneficiaries. Application to the Structural Funds The comparison groups technique is suitable for ex post evaluations for which the identification of impacts is essential. In particular, it enables the estimation of deadweight, that is to say, changes which would have occurred in the target group in any case, even without the programme, and which therefore can not be considered as an impact of the programme. In the context of the Structural Funds, it is however often difficult to determine comparison groups. The public concerned must be sufficiently homogeneous and a significant proportion of the potential beneficiaries must not have participated in the programme (if this is not the case, it is said that there is "saturation " of the population exposed to the programme). Tools for analysing data One approach is to compare the performance of beneficiaries with those of select comparison groups from outside eligible areas. However, few studies have so far employed this technique. Strengths and limits in practice • A well­designed comparison group analysis provides a convincing estimate of a range of programme effects. • One of the problems posed by this tool stems from the need to take into account, from the outset, factors that are likely to have a decisive influence on the production of effects. If important variables are overlooked (often factors outside the influence of the programme) it becomes difficult to establish strong causal inferences and the analysis loses some of its validity. • Another shortcoming stems from its inability to explain the observed effects. It focuses on the measurement of effects rather than on mechanisms that produced them (implementation, quality of services provided, reactions of beneficiaries). The programme and its effects are treated as a "black box". This may be partially compensated by using other tools (e. g. case study), giving the comparison a qualitative dimension. Further reading Boruch, R. F. (1997), 'Randomized Experiments for Planning and Evaluation. A practical guide'. In Applied Social Research Methods Series, 44. London : Sage Publications. The book deserves to be called what is written as its second title "a practical guide". Matalon Β. (1988), Décrire, Expliquer, Prévoir­ démarches expérimentales et terrain. Armand Colin ­ collection U. pp. 219­252 More detailed and technical volume ­ see the third part. Annex 1 considers the particular case of the construction of comparison groups by drawing lots. Mohr, L. B. (1995) 'Impact Analysis for Program Evaluation' (2nd edition), London: Sage Publication. Monnier E. (1992), Evaluation de l'action des pouvoirs publics. Paris: Economica, pp. 9­51 Good illustration of comparative techniques. Treasury Board of Canada, Office of the Contrôler General (1991), 'Program Evaluation Methods'. Treauusury Board of Canada. General approach ­ see chapter 3. 1 ­ 3. 35 Principal evaluation techniques and tools More in-depth: control groups and the experimental approach In order to eliminate the risk of bias, individuals, families or enterprises can be randomly assigned to one or the other of the two groups. This practice, known as the experimental approach, is the most rigorous way of establishing supposed causal links between public interventions and their expected impacts. This approach assumes the comparison of a group exposed to the programme (experimental group) and a strictly identical group (control group) that allows what would have happened in the absence of the programme to be observed. The experimental approach is used to verify the relations of cause and effect between observed socio-economic changes and the evaluated programme. It allows the state of a given social situation following the implementation of a programme to be empirically compared to what would have been the situation without the benefit of the programme. This approach thus allows the evaluation of the net impact of a programme by controlling factors. Since the use of comparison groups that are statistically equivalent strengthens the internal validity of. the technique, it can be considered the most rigorous way of establishing causal relations in relation to the results of a programme. influence of confounding the to combat alcohol Experimental approaches can be used to evaluate a wide range of public interventions, but have been mainly used in educational programmes (programmes consumption, smoking, etc. ). Experimentation can be appropriate when it is possible to perform a random assignment without fear of social injustice occurring from the exclusion of the control group from the benefits of public intervention. It is equally justified in the case of costly, innovative programmes for which the most reliable data possible are required. Implementing an experiment Experimentation is based, first and foremost, on the random assignment of the units of observation, divided between the experimental group and the control group so as to constitute groups that are statistically equivalent. Thus, the probability will be very high that the descriptive statistical parameters of any variable describing these two groups will be similar. The second step consists of applying a given treatment uniquely to the experimental group (the only difference between the two groups). This step is followed by making measured observations of the two groups before the Tools for analysing data application of the treatment (E (1) and E' (1)), and then after (E (2) and E' (2)). Finally, the gap between the two groups' initial and final states is compared to provide a measure of the impact of the treatment. Whether the impact is significant or not is judged on the basis of the extent of the difference. The table below summarises the implementation of this approach: Initial measurements Application of treatment Measurement in fine Variation in measurements E(1) X E (2) E(2)-E(1) Experimental group Control group E'(1) E'(2) E' (2) - E' (1) The limits of experimentation to is difficult The experimental approach implement. Often, random assignment cannot be achieved since it is sometimes impossible to define the variable that would allow the constitution of a comparison group (e. g. when the target population benefits in totality from the programme). Random selection of the members of the groups studied is rarely possible for the totality of the target population, while the use of volunteers is a source of bias, known as "autoselection". Furthermore, the artificial creation of a control group harms the external validity of the experimental approach. Whatever method is used to select the participants, external validity may be compromised. Contextual factors linked to the choice of sites also limit the representativeness of the experiment. Many elements therefore confirm that the experimental approach reduces the extent to which findings can be generalised to an entire programme or to other contexts. Ethical and political problems can furthermore arise, when it is considered the that only the members of the experimental group benefit from programme to the detriment of individuals in the control group. Moreover, the experimental approach requires significant financial, intellectual and institutional resources, and the constitution of statistically equivalent groups is in itself a demanding task. Finally, this approach does not provide information about the effects of sudden events. How to use the tool for evaluation The experimental approach is justified in a certain number of cases where causal relations are tew, simple and known, when the objective is to ^ ^^ Principal evaluation techniques and tools measure impacts that are stable, and not to explain complex phenomena that are in a state of flux. This tool is therefore not particularly well suited to evaluating Structural Funds and an example of its application within this framework cannot be the found. Practical difficulties uncertainty of its usefulness can explain this absence. Structural Fund programmes are applied over a relatively long period. Thus, the two groups, experimental and control, could be exposed to certain variations and obvious differences would therefore occur between the groups. implementation and relating its to Further reading Boruch R. F. (1997), Randomized Experiments for Planning and Evaluation, Thousand Oaks, CA: Sage, 265 p. A book which presents a measured argument in favour of the use of the experimentation technique. Tools for analysing data R e g r e s s i on a n a l y s is Regression analysis is used to obtain a quantitative estimation of the influence of several explanatory factors (e. g. intervention to be evaluated, confounding factors) on a variable that has to be explained (e. g. the result of an intervention). A logical explanatory m o d el is required if regression analysis is to be used. The tool calculates the O Overall evaluation 0 Medium technical complexity © φ Medium cost O Innovative use Retrospective use parameters of this model, adjusting them as accurately as possible in relation to developments observed in the past. Regression analysis can be applied in various forms, depending on whether the variables of the model are continuous or not, and on whether their relations are linear or not. The tool has been used for about a century by statisticians, particularly in biology. It was generalised with the development of microcomputers and statistical software enabling users to easily adjust c o m p l ex models to fit numerous data. Example: Evaluation of ESF training courses in Spain In 1995 the Departments of Economic Analysis and Applied Economics, of the Faculty of Economic Science at the Independent University in Madrid, evaluated the effects of certain training measures set up with the assistance of the European Social Fund. This evaluation was to provide answers to the following questions: How many participants found a job after attending the course? What is the profile of the participants who found a job? What is the influence of the regional context? A survey was carried out on a sample of 13,127 real or potential trainees throughout Spain (with the exception of three regions). An explanatory model was constructed with the following variables: Variables to be explained (dependent variables): the employment rate and the length of time looking for a job; Explanatory variables: gender, age, qualifications, place of residence, type of unemployment (long-term, search for a first job), and professional experience; A special explanatory variable was added: participation in the programme (duration of ESF training). This model comprises continuous variables such as age or length of time seeking employment, as well as discontinuous variables such as gender or the fact of finding Principal evaluation techniques and tools a job. Regression analysis was used to adjust the model to the results of a questionnais survey carried out on a sample of 13,127 real or potential trainees throughout Spain (with the exception of three regions). The regression analysis was used to estimate the impact of the programme on the employment of the trainees. Moreover, it was used to produce additional estimations such as: • Men improve their chances of finding a job by 5%; • Regional locality is important: trainees living in the Balearic Islands have about 20% more chance of finding a job than those in Galicia; • People with secondary school education improve their chances of finding a job by over 12% compared to those who only have primary school education. Source: F. Saez-Fernandez, M. Toledo, "Education, Labour Market and Employement Policy - Empirical Evidence and Implications', 7th EALE Conference, Madrid 1995. What is the purpose of the tool? When applicable, regression analysis for estimating effects. It serves to attribute part of the changes observed to the evaluated programme to explanatory or confounding factors. (net effect) and is an excellent technique the rest In which cases can it be used? The tool is applicable only if one already has a credible and sound explanatory model. It must then be used in cases where the effects of the intervention are well identified and where the production process of the effects is known. Regression analysis may typically be used when one of the effects of the evaluated programme is given precedence (e. g. placement rate). In order to apply the tool, a considerable volume of quantitative observations is required: between one thousand and ten thousand units (trainees, enterprises, farms, etc. ). The observations must be detailed according to all the variables of the explanatory model. How should it be implemented? Step 1. Construction of the causal model: Construction of the explanatory model is a difficult step that must be carried out in line with reflection on the action theory of the intervention. Several kinds of variable Tools for analysing data may exist. They may be specially created, for example to take into account the fact that an individual has benefited from support or not (a dummy variable, taking values 0 or 1). A variable may also represent an observable characteristic (having a job or not) or an unobservable one (probability of having a job). The model may presume that a particular variable evolves in a linear, logarithmic or probabilistic way. All the explanatory models are constructed on the basis of a model such as: Y:f(X0, X1, X2. ,Xn) in which: Y is the change that the programme is mainly supposed to produce (e. g. employment of trainees) f is a function of several explanatory variables XO is a variable representing the fact that an individual benefits or not from the intervention. X1,. , Xn are the other variables likely to explain the change. Box 26- Performing a regression analysis S t e p l. Construction of the causal model «- ►! Y =f (Χ1. Χ2 This relation is not given by the method Xn) ; ; ; I Step 2. Construction of a sample I Step 3. Collection of data I Step 4. Calculation of the coefficients of the relationship model i Step 5. Test of the coefficients obtained (adjustments quality) I See questionnaire survey » J Significance of parameters Analysis of residue ' Step 6. Generalisation to the total population (inference) Principal evaluation techniques and tools A good model has to select few variables with substantial explanatory power. Important factors must not be forgotten. Causal relations that function both ways must be avoided, e. g. X1 explains Y but Y explains X1 (circular causality). If two explanatory variables are linearly dependent and provide the same information, it is futile to consider both of them. In fact phenomena of co- linearity weaken the adjustment. For example, if the women questioned have experienced periods of previous unemployment which are systematically longer than those of men, it will not be possible to separate the influence of the two explanatory factors: gender and duration of previous unemployment. Some forms of regression analysis (path analysis) allow for the separation of direct and indirect effects. Indirect effects occur through one or more intermediary variables. Given the complexity characterising most socio­ economic phenomena, this priority is important. Step 2. Construction of a sample: To apply multiple regression, a large the sample questionnaire survey for the construction of samples. is usually required (2,000 to 15,000 individuals). See Step 3. Data collection: Data are generally taken from the monitoring system, from a questionnaire survey or from a combination of both. Step 4. Calculation of coefficients: This phase in the analysis is facilitated to a large extent by statistical software that is both affordable and can be used on a microcomputer (e. g. : SAS, SPSS, SPAD, etc. ). Step 5. Test of the adjustment made: The preparation and adjustment of the model are aimed at trying to explain as much of the variability of the observed changes as possible. The part that remains unexplained represents the residue. Thus, the smaller the residue, the better the quality of the model and its adjustment. The analysis of residues is a very important step: it is here that one sees the degree to which the model has been adapted to the phenomena one wants to explain. It is the residue analysis that enables one to tell whether the tool has made it possible to estimate the effects in a valid way or not. If significant anomalies are detected, use of the tool must be considered as a failure and the evaluation must be stopped before Step 6. Step 6. The purpose of this last phase is to generalise the coefficients of the model to the population as a whole. It is therefore used to produce an estimation of the effects. Tools for analysing data Application to the Structural Funds limits. The In the context of the Structural Funds, use of this technique soon encounters requires quantitative knowledge of several thousand individuals. It is therefore also interventions time-consuming and expensive and is applied only to affecting a large number of people. is unwieldy because technique it Strengths and limits in practice • Use of regression analysis affords an opportunity to specify hypotheses concerning the nature of effects (action theory), as well as confounding factors. • When it is successfully concluded (statistically valid adjustment), regression analysis produces a quantitative estimate of net effects and solves one of the most difficult problems in evaluation. • The observations must present sufficiently contrasted evolutions to allow for adjustment. For example, if all the observations concern the 30-40 age group, it will not be possible to estimate the influence of age on employment. • Regression analysis is likely to reach the conclusion that there is a strong link between two variables, whereas the influence of other, more important, variables has not been estimated (this error is called "data snooping"). • Relations between the different explained and explanatory variables are often circular (X explains Y and Y explains X). In this case, the method is inapplicable. • The implementation of the tool is both time-consuming and expensive. • The tool has to be used with great care: errors in data collection and in the explanatory model affect conclusions and this can easily be overlooked. Further reading Wonnacot T. H. , Wonnacot R. J. (1984), Statistique, Economie, Gestion, Sciences, Médecine, Paris: Economica, 790 pp. Manual explaining the basic principles of statistics in the social sciences. Clear and illustrated with numerous examples. Principal evaluation techniques and tools Mohr L. B. (1995), Impact analysis for program evaluation, London: Sage Publications, 310 pp. This book addresses the main evaluation. Clear, avoids a mathematical mode of expression. techniques for programme Tools for making a judgement In the framework of in-depth evaluation, evaluators can use any available tool for making a judgement, particularly those that have been developed for project evaluation. • Cost-effectiveness analysis judges the efficiency of the intervention by quantitative comparison with one or more other interventions. It favours an objective considered to be a priority. It is particularly well suited to ex post evaluation. • Benchmarking judges the efficiency of the intervention by qualitative comparison with another intervention recognised as being excellent (i. e. a benchmark). With benchmarking, several effects can be taken into account and synthesised. • Cost-benefit analysis judges the efficiency of the intervention by taking into account all its effects and by making a quantitative synthesis. Cost- benefit analysis can judge an intervention in the absolute, without any comparative reference to other interventions. It is generally used ex ante, but all the conditions required for its efficient application are rarely met. • Multicriteria analysis (presented above) compares the effectiveness and efficiency of several projects or interventions. The tool takes into account and synthesises several effects, quantitatively and/or qualitatively. This tool can be used to take into account any differences in the points of view of the partners. The tool is generally used ex ante. Its flexibility makes it suitable for use in many situations. • The expert panel (presented above) is a very flexible tool which can be used to produce a synthetic judgement based on qualitative and quantitative data, even if these are incomplete. With this tool any differences between the points of view of the partners can be taken into account. To a lesser extent, the Delphi survey (presented above) can provide similar assistance in the formulation of a judgement. • Finally, the colour vote (presented above) is a tool which may be used during the final synthesis of an evaluation, for validating conclusions and recommendations. Tools for making a judgement Cost-Effectiveness Analysis Cost-effectiveness analysis originates from certain types of evaluations, notably in the domains of health and road safety. Characterised by its pragmatism, it focuses directly on the main effect of a programme in relation to its cost, purposely excluding other potential effects. It is a simplified A In-depth evaluation Θ Q Medium technical complexity O Φ Medium cost O Use stabilised Retrospective use Prospective use form of "value-for-money" evaluation (VFM) in which the "value" of the programme is expressed as a single dimension. Cost-effectiveness analysis differs from cost-benefit analysis, particularly in so far as the effect of the programme is expressed in physical quantities (e. g. number of deaths avoided) and not as a monetary value. Example: Creating and safeguarding jobs in Wales This example corresponds to that of Box 3. The British National Audit Office (NAO) carried out an evaluation, based on cost- effectiveness analysis, of policies for the creation and safeguarding of jobs in Wales. These policies had been implemented over the preceding twenty year period by three key agencies: the Welsh Office (WO) responsible for urban development, the Welsh Development Agency (WDA) responsible for industrial development and the Development Board for Rural Wales (DBRW) charged with social and economic development in the Northern and Central regions of Wales. The evaluation was set up jointly by the NAO and the WO and lasted for 5 months. It focused on the effects of these policies on a sample of 303 beneficiary firms, which were questioned about these effects. The results of the interviews were used to calculate the cost per job of the measures, on the basis of the foreca. t number of jobs created, the deadweight and displacement effects and the beneficial effects on suppliers of these enterprises. The overall cost of the programme was then divided by the net benefits to obtain a measure of cost-effectiveness; i. e. cost-effectiveness = cost of the programme / (number of gross jobs created - deadweight - displacement + supplier effects). The absence of uniformity in the implementation of regional development actions in Wales made comparisons of costs per job created difficult. Nevertheless, the NAO concluded that the employment effects of the programmes were often over- evaluated, particularly since deadweight and displacement effects were not taken Principal evaluation techniques and tools into account, but that the programmes had nevertheless created a significant number of new jobs. Source: S. Roberts and Ch. Pollitt, (1994) Audit or Evaluation? a National Audit Office VFM Study. Public Administration What is the purpose of the tool? Cost-effectiveness analysis is used to evaluate the effectiveness of a programme in terms of a single criterion that is considered preponderant or the key objective of the programme. On the basis of this criterion, it is possible to compare the effectiveness of the programme with that of similar programmes in other regions or alternative approaches focusing on the creation of similar results in the same region. the Cost-effectiveness analysis may constitute benchmarking of programmes or actions implemented in similar contexts, but which apparently have a better cost-effectiveness ratio than the programme under analysis. first step the in In which cases can it be used? Cost-effectiveness analysis is used in both ex ante and ex post evaluation. It is used when all the expected effects have been defined and are homogeneous and/or can be measured in terms of a key result (e. g. the number of jobs created). How should it be implemented? Step 1. Determination of the programme objective: This first step is one of explanation of the programme logic. The following questions are answered: What are the goals to be achieved? What are the programme outputs? What are the expected impacts? Among them, which one may be considered predominant? Step 2. Definition of the total cost of the programme: At this stage the cost of all the public resources of the programme are added up to obtain a total cost. Generally speaking, only direct resources that have a well- defined monetary value are included. Less tangible costs, such as the time passed by staff, and in particular, supervision costs, are not usually taken into account. The cost of a programme is sometimes measured in relation to the benefits that could have been obtained by allocating public monies to other ends (opportunity costs). Tools for making a judgement Step 3. Measuring the impact: This step is the trickiest one. Numerous studies, including the evaluation of the NAO, presented above; use empirical approaches based on the collection of primary data, in order to gather information on the positive effects of a programme under evaluation. It is also possible, however, to estimate impacts on the basis of secondary data and/or the modelling of the implementation of the programme. Whatever the method used, it is important to have an exact picture not only of positive effects, but also of deadweight, displacement effects, crowding out effects, etc. Certain evaluators have estimated that indications should be given about multiplier and unexpected secondary effects. In reality, it is extremely difficult in practice to accurately evaluate these secondary effects. Consequently, the majority of evaluations does not take them into account or gives estimations of their level. This solution is preferable given the importance of secondary effects. The lack of accuracy of the estimation of these impacts can be attenuated by the use of sensitivity analysis to indicate the probable level of benefits. Step 4. Calculation of the cost-effectiveness ratio: The last phase is that of the calculation which will give the final result. It consists of a simple division, for which the formula will be of the following type: Programme cost / (gross effects - deadweight and displacement effects, and should the occasion arises + multiplier effects on suppliers and indirect addressees). Application to the Structural Funds The cost-effectiveness technique is difficult to apply to Structural Fund programmes that often have multiple objectives. From this point of view, multicriteria analysis is often more appropriate. It may, however, be a focused on suitable employment effects, effects on the creation of SMEs, etc. , or to compare approaches that are aimed at producing similar impacts. It is nevertheless important to ensure that the basis of the calculation and the reliability of the data relating to the costs and net benefits are consistent. thematic evaluations such as those tool for Strengths and limits in practice • The analysis must be preceded by reflection on the programme objectives, on its main objective and on the indicator applied to that objective. Principal evaluation techniques and tools • Cost-effectiveness analysis tends, by its nature, to focus on direct results that occur over the short to medium term, but does not look in general at more long-term impacts. Yet it on these that the overall effectiveness or the lack of effectiveness of programmes and policies depend. For example, the effectiveness of Structural Fund programmes depends on achieving a combination of multiple objectives aimed at promoting growth and autonomous development within a region. • Cost-effectiveness analysis can serve to compare programmes only when their implementation is straightforward and their impacts of a similar nature. This type of case is far rarer than one might imagine. Is it possible to compare jobs created without introducing differentiation concerning their stability, quality, required qualifications, etc. ? • Since cost-effectiveness analysis is based on an estimation of the programme impact in relation to its main objective, it presents the advantage of producing easily understood findings, which concentrate on the main preoccupations of key groups (including politicians and decision-makers). • If the analysis is based on specific links between inputs and outputs that are relatively well established, the tool can facilitate the description of the actual functioning of programmes. This can be useful for the refining of existing policies or improving the effectiveness of future interventions. • Cost-effectiveness analysis can only be used to compare programmes that are simple to implement and of which the impacts are the same type. Yet, this situation is much rarer than is imagined. Thus, for example, even programmes targeted on an identical major objective, such as the creation of jobs, can create employment opportunities that are qualitatively different (e. g. in terms of longevity, security, remuneration, probability that they will be accessible for the inhabitants of targeted regions, etc. ). • Establishing causal links of this type requires the ability to obtain detailed data from programme managers and addressees. When these are not collected through the monitoring of programmes and projects, the evaluation will involve the painstaking collection of primary data. • Ideally, cost-effectiveness analysis should be used in combination with other techniques of economic analysis in order to analyse longer-term impacts on the regional GDP and competitiveness. Tools for making a judgement Further reading Creating and Safeguarding Jobs in Wales, A Value for Money Audit (1993), Brunei University, MEANS A pilot evaluation of the MEANS programme, 20 ρ Garrabé M. , Ingénierie de l'évaluation économique, Paris: Ellipses, 255p. A book on economic evaluation, describing the main economic evaluation tools. Among these, features cost-benefit analysis, which includes cost-effectiveness analysis. Accessible. Gray A. W. (1995), EU Structural Funds and Other Public Sector Investments. A Guide to Evaluation Methods, Dublin: Gill & Macmillan,106 p. A book on economic evaluation techniques, specifically oriented to cost-benefit analysis. Contains a complete description of cost- effectiveness analysis, its logical articulation and the conditions of its implementation. Highly accessible. Sanders J. R (1994), The Program Evaluation Standards, How to Assess Evaluations of Educational Programs, Second Edition, London: Sage Publications, 222 p. A manual devoted to the main steps in the evaluation of programmes. Contains a brief description of cost-effectiveness analysis. Highly accessible. Tools for making a judgement Benchmarking Benchmarking w as developed in the 1980s by the major American electronics firms that wanted to improve their competitiveness and quality in the face of Japanese c o m p e t i t i o n. T he t e c h n i q ue is b a s ed on t he e x c h a n ge of information relating to t he o u t c o m es a nd organisation of certain segments of production, O Overall evaluation A In-depth evaluation 0 Low technical complexity O @> High cost © Prospective use Innovative use between firms with a reputation for excellence in a given field. In the field of "New Public Management" this practice has been translated into the management of public services a nd municipal administrations. It may also be relevant t h o u gh in t he o n g o i ng evaluation of public interventions. Example: evaluation of urban management The Bertelsmann foundation in Germany organised a competition, open to German towns. The programme judged the quality of administrative reforms implemented by the candidate towns, by comparing them to three benchmarks. In order to have a standard for comparison, the foundation called on a panel of experts which defined the criteria to be used to judge the quality of administrative reforms. This jury then identified cases of "best practices" in terms of the chosen criteria, i. e. three towns, situated in New Zealand, the UK and Switzerland, well known for the originality and effectiveness of their administrative reforms. Each of the towns identified was considered to be "exemplary" and served as a "benchmark" for at least one of the criteria defined by the jury. The criteria were as follows: modalities of commercialisation of public services; quality strategy applied to the public services; strategic management concerning the organisation of services; and financial management. Apart from the fact that it provided a clear set of criteria for judging the competition, the benchmarking technique made it possible to evaluate, on a broader scale, the progress of administrative reform in many German towns. The process was seen as highlighting the backwardness of German towns in implementing reforms of this kind. Source: Frieder Naschold, "Internal Modernisation, Competition, Fiscal Consolidation. International Experiences with Local Government Administrative Reform", WZB, 1996. Principal evaluation techniques and tools What is the purpose of the tool? The aim of benchmarking is the identification of the strengths and weaknesses of an organisation by comparing it to other organisations renowned for being the best in the field in which progress must be made. The approach consists of the comparison of performances situated at different levels, depending on the benchmark chosen: • External benchmarking is used to compare the different functions of the production process of an organisation to similar functions in other organisations. the methods applied within • Internal benchmarking is used to study best practice and to use it to improve Internal benchmarking is applied to the divisions of a parent organisation. It is sometimes considered to be a less demanding approach because it involves comparisons with a relatively small group as opposed to the 'best of the rest'. the organisation. The advantage of benchmarking lies in the decomposition of the productive logic into elementary functions the performance of each of which is then compared. The exchange of information on these elementary functions thus becomes possible between and within very different organisations. This type of approach provides a more objective view of current practice in an organisation, by comparing it to practices elsewhere. It is a way of learning from others' experience. Practised on a larger scale, benchmarking may promote the spread of good practice. Although benchmarking may be used for the evaluation of measures and projects. it was originally designed to be applied to organisations, • In which cases can it be used? As an evaluation tool, benchmarking is applied to the evaluation of measures. It consists of identifying measures (or projects) recognised as best practice and using them as a standard of comparison for evaluating another measure (or project). This enables one to clarify the functioning of the programme, by detailing the different measures characterising it, and to identify the strengths and weaknesses of particular approaches. The technique may be used to determine benchmarks in terms of effectiveness and efficiency - that is to say, programmes recognised for their quality - and thus to implement the changes to be made in the Tools for making a judgement framework of the evaluated programme to enable performance of the best performers. it to match the How should it be implemented? Box 27 - Benchmarking in four steps 1. Select those parts of the process which will be subject to benchmarking - 2. Identify a partner for the benchmarking - 3. Observe the process in the partner's situation * 4. Analyse the discrepancies in performance Step 1. In the preliminary phase the nature of the problem must be identified and the problem presented clearly. A first step consists of the conceptualisation of the process to be analysed. The benchmarking will be able to focus only on some of the steps in the process. These will be chosen in relation to the nature of the problem: they are the benchmark criteria. A second step consists of constructing indicators which can be used to make the evaluation criteria measurable. Step 2. The choice of the organisation(s) with which the comparison will be made depends on the criteria chosen for the benchmarking. The comparators need not necessarily have similar structures, but will be chosen because they are seen as relatively good performers in terms of the benchmarking criteria. Benchmarking is, however, based on a form of reciprocity. Both partners must get something out of it. If this is not the case, the benchmarking will be of limited interest. Step 3. It is at this point that the benchmarking takes place. It consists of an in-depth analysis of the criteria identified in the partner organisation, with a view the solutions implemented to meet these criteria. to establishing a systematic comparison of Step 4. This analysis may have different forms: face-to-face contact through visits in the organisation, the sending of a questionnaire, use of specialist consultants or panels composed of members of the organisations involved. Whilst all benchmarking exercises include these four stages, they can take a variety of different forms. In some cases benchmarking is used simply to compare the outputs or outcomes achieved by different organisations or Principal evaluation techniques and tools policies. Here the focus is upon assessing relative effectiveness and the result is often some form of ranking of performance through for example the publication of 'league tables'. More and more agencies are though going a step further, using benchmarking as a diagnostic tool to enable programme managers not just to observe differences in performance but also to analyse the reasons for these and to learn from each other about strategies for enhancing effectiveness. This latter approach is known as 'process benchmarking'. It requires analyses of the different management styles, organisational cultures and environmental contexts within which programmes operate as well the means by which policies are being implemented. Its value lies in the ability to identify good practice and to transfer this to or modify it for implementation in other settings. It is often argued that comparisons between similar programmes operating in comparable environments are the most valuable because their performance might be expected to be broadly comparable and they benefit from adopting similar practices. Unlike output based benchmarking however this is not always the case and some programme managers have found that they are able to gain important new insights from benchmarking against organisations which are quite different to their own. Exchanges of good practice between public and private sector agencies represent an increasingly common example of the latter. In the context of the structural funds examples might include process benchmarking to compare the operation of similar programmes in different member states or comparisons of EU funded programmes and domestic initiatives within the same country. Application to the Structural Funds The application of benchmarking to the Structural funds consists of the identification of projects recognised for their good practice, and of their comparison with projects implemented in the framework of the Structural Funds. International Organisations, Universities or other public or private organisations. It may concern measures implemented in It is also possible to use benchmarking during the preparation of a new cycle of Structural funds, which may result in changes of a regulatory, strategic and organisational nature. The tool may also be applied for the purpose of improving the evaluation systems themselves. Tools for making a judgement Strengths and limits in practice • Benchmarking is a flexible method which can be used to formulate solutions to organisational problems and to highlight strengths and weaknesses. • The implementation of benchmarking must involve a team that is big enough for the results to be accepted within the organisation, but small enough to maintain its flexibility and adaptability vis-à-vis the changes to be made. In large institutions this poses problems. • The procedure must be initiated within an organisation and be constructed on a participative basis if it is to be effective. If the work is formal and is carried out by outside consultants without any real participation from within, the benchmarking will inevitably be a limited exercise with a very limited impact in the organisation. • The identification of a partner for the benchmarking may be problematical in the framework of public programmes, and it may be necessary to use panels of experts to identify the best practices. • Benchmarking is often based on the sharing of strategic information. It may therefore constitute a risk for the organisation that agrees to participate if it is a public enterprise or a research laboratory in a strategic political or economic sector. Further reading Balm, G. J. (1994)"Evaluer et améliorer ses performances. L'étalonnage", AFNOR, Paris, 159 p. Sets out the steps of benchmarking as undertaken by IBM in the United States. Sometimes a little dense. Camp, R. (1990) 'Benchmarking: the search for industry best practices that lead to superior performance', ASQC Quality Press. Chang, R. Y, Kelly, P. K. (1995) "Progresser avec le Benchmarking", Les presses du management, Paris, 122p. A narrative description of the setting up of benchmarking in a firm. Very simple and lively. Karlöf & Partners (1995). "Pratiquer le Benchmarking", Les éditions d'organisation, Paris, 144p. Practical guide which provides a lively illustration of the different steps in benchmarking, from the viewpoint of firms. Principal evaluation techniques and tools Naschold, F. (1996), 'Internal Modernisation, Fiscal Internal Experiences with Local Government Consolidation: Administrative Reform'. Berlin: Science Center. An article which shows how to use benchmarking as an evaluation method. Competition, Tools for making a judgement C o s t - b e n e f it analysis A Θ © Medium cost In-depth evaluation Medium technical complexity Cost-benefit analysis (CBA) is a tool for analysing public projects. Its purpose is to determine whether is realisation of a project the desirable from the point of view of society as a whole. The technique takes into account and assigns a monetary value to all the positive that and negative effects the project has on the different components of society, throughout the life of the project. In order to validate the appropriateness of the project, it is important not only to assess its feasibility but also to evaluate alternative options in order to select the most appropriate solution for the community. Θ G Prospective use Use stabilised Use of cost-benefit analysis was generalised during the 1960s in the framework of the PPBS (Planning and Programming Budgeting System) in the United States. As a decision-making tool, it can be used to justify the choice of a project among a set of possibilities. Some experiences in the systematic use of CBA have been carried out, for example by the World Bank (in the 1970/80s) and the British government (in the 1980s). Today cost-benefit analysis plays a significant role only in the evaluation of large- scale projects such as major infrastructure projects. Example: The bullet train in the Madrid-Seville corridor - a cost-benefit evaluation The Spanish bullet train (AVE) connected Madrid to Seville in 1992. During its construction, several technological options existed. There was a choice between a complete high-speed train requiring a specific infrastructure, and using pendular technology necessitating less investment in infrastructure but resulting in lower travelling speeds than those of the AVE. Cost-benefit analysis was applied retrospectively to this project, taking into account the structure of the costs and rates observed since 1987. The following costs and benefits were taken into account: • cost for the Spanish and European tax payer • cost for the passengers of the train • cost in terms of pollution for people living near the railway line • advantages in terms of time saved by passengers (calculated as a monetary equivalent). Principal evaluation techniques and tools Given that costs are concentrated mainly during the construction and that benefits are spread out over a long period, all the costs and benefits were calculated by measuring their net present value. The study resulted in a retrospective validation of the project by showing that its benefits were, on the whole, greater than its costs. Source: G. De RUS, Facultad de Ciencas Económicas de Tarifa, 1996 What is the purpose of the tool? Cost benefit analysis is used essentially to compare all the impacts of a public project during its lifetime, when these impacts, even if they are widely diverse, can be measured in monetary terms. In which cases can it be used? Cost-benefit analysis is widely used in the framework of procedures prior to the construction of transport infrastructure. It is used when the expected impacts of a project extend beyond the private sphere, are differentiated, and concern a heterogeneous population. It is carried out within the framework of projects where the addressees and those who stand to lose from the project can be identified, and projects for which all the effects, even those not related to the market, can be identified and attributed monetary values. How should it be implemented? Step 1. Definition of the project and its cost: This first phase serves to situate the project in its context. It is during this step that the inherent constraints in the realisation of the project are determined, and that the different possible variants and the duration of the project are examined. This is the phase of the determination of costs, with the application, where relevant, of fictive prices (e. g. the fictive price of unskilled labour is nil if there is a high level of unemployment). Four categories of variables define the CBA: • costs (C): not only the monetary but the social costs generated by the project must be taken into account (e. g. the real cost of renting land made available free of charge by a public institution to a user); • benefits (B): both the strictly economic spin-off and the social impact of into account (e. g. reduced congestion the project must be taken following the construction of a highway); Tools for making a judgement • time (t): since any investment is characterised by a lifespan, it is important to extend the analysis to the entire period concerned by the future of the project (e. g. each of the years during which a hospital will function); the time itself is limited by the lifespan of the investment which is equal to n (e. g. 20 years for infrastructure and 10 years for productive investments); • discount rate (r): since the costs and benefits are spread out over time, it is essential to discount values so as to be able to total the values calculated (e. g. the assessment of an advantage at its current price does not allow for the correct comparison of two distinct periods, given the phenomenon of inflation). Box 28 - Steps in a cost-benefit analysis Step 1. Identifying the project, checking for its cost. , ' ' Step 2. Identifying benefits and losses for social groups ' r Step 3. Valuing benefits and drawbacks in euros 1 r Step 4. Evaluation of the project I What are the possible costs of running the ^ are the limits of the project? Ί How long Is the envisaged duration of the I project? i ¡ J I Identification of impacts for winners and losers. Quantitative estimates of impacts. ! I Use of social values, willingness to pay and , discount rate ; I Add present value of all gains and losses In accordance to the compensation principle. J ; Net present value of the project. ; Social rate of return. ' ! ; Step 2. Identification and physical quantification of the advantages and disadvantages of the project. This phase consists of the identification of the advantages and disadvantages of the project, which may affect different groups of persons (e. g. the time saved by people using the new railway line or the amount of disturbance suffered by the people living nearby). CBA differs from the evaluation of private projects on two fundamental points: Principal evaluation techniques and tools • the costs and benefits to all members of society are taken into account, income of the agency and not only the monetary expenses and responsible for it; • the social rate of return may differ substantially from the private rate of return after phenomena of externalities have been taken into account, because the latter concern the positive or negative effects that society has to accept as a consequence of the realisation of the project. Externalities concern the positive or negative effects that society has to accept as a consequence of the realisation of a given economic activity. They refer to the positive or negative impacts on society, without these impacts being taken into consideration uniquely within the functioning of the market. For example, the improvement of transport infrastructure is reflected in a reduced accident rate, time savings, reduced transport costs and less pollution. These effects benefit everyone, without everyone necessarily having contributed to the realisation of the project. Step 3. Monetarisation of advantages and disadvantages: This phase must not be confused with that of the definition of costs. Here the monetary value must be given to the different impacts (positive or negative) of the project, the main ones being outside the sphere of market relations. The principle of cost-benefit analysis is that each impact is given a value in proportion to what those who benefit from it are prepared to pay to obtain it (presuming they are in favour of the impact) or to avoid it (if they are against the impact). For example, a passenger is prepared to pay 100 Euro to save an hour, a resident is prepared to pay 10,000 Euro less for a house exposed to pollution, etc. If an advantage or disadvantage cannot be evaluated in the form of willingness to pay (What is the value of a biotope? What is the value of a human life? What is the value of knowledge gained? etc. ), social values fixed by discretion are used. Social value should be taken here to mean the value fixed through an explicit political process (though more often implicit). The advantages and disadvantages are spread out over time and have to be related to the different stages of the project. For this purpose, time itself is considered to have a social value, i. e. the discount rate. Step 4. Evaluation of the project: Once all the preceding steps have been completed, the evaluation as such can be started. This is when a judgement will be made on the project. It is at this stage that the principle of compensation applies. If the project is soundly based, the winners should Tools for making a judgement be able to compensate the losers, which translates into the following principle: the monetary value of the disadvantages and costs must not exceed the monetary value of the advantages. This principle is obviously of an abstract nature, for compensation is never really paid. Since the results of CBA may vary considerably, depending on the working hypotheses, it is necessary to complete the evaluation of the project with a sensitivity analysis. This will be used to check the extent to which the profitability of the project is affected by variations in essential variables. Two indicators summarise the final evaluation of the project • the net present value of the realisation of the project, which is no more than the difference between the benefits and the socio-economic costs of the realisation of the project, measured by taking account of the duration of the project, and calculated as comparable values on the basis of the discount rate; • The the internal rate of return of the project, which relates to determination of the discount rate for which the present net value of the project is nil, and which makes it possible to assess the social utility of the project. Application to the Structural Funds Cost-benefit analysis is applied to the evaluation of large projects. It is a regulatory obligation for the prior appraisal of major European projects. It is carried out by the Member States that have to submit the analysis to the Commission before any decision can be taken to finance the project. Cost- benefit analysis is rarely used in other situations. For the period 1989-1993, the internal economic rate of return of major projects financed by the Structural Funds are as follows: Sectors Average rates Number of projects Energy Water and environment Transport Industry Other services Total 13. 7 21. 7 19. 2 18. 4 18. 7 19. 3 5 21 63 14 5 108 Source: Guide de l'analyse des coûts et avantages des grands projets, Commission européenne Principal evaluation techniques and tools Strengths and limits in practice • The setting up of a cost­benefit analysis forces designers to ask essential questions such as: Who are the persons concerned? What are the advantages and disadvantages of the project? What part of the population wins and loses and how? In this sense, it helps to clarify the project. • Cost­benefit analysis makes it necessary to ask about the impacts of a project in clear terms. However, it cannot serve to analyse impacts, which have to be examined using other evaluation techniques. • It is not always possible to give a credible and legitimate value to certain advantages and disadvantages (e. g. the destruction of a rare biotope). • Cost­benefit analysis does not directly take into account the objectives of the policy being carried out. All the advantages and disadvantages are included, whether they are important or not in relation to the political objectives. • The full complexity of a large project, the number of persons concerned, the definition of impacts, the indirect effect, etc. make the quantification of advantages and costs very difficult. • The principle of compensation is never applied in reality, the use of this technique is therefore far removed from the pure theoretical constructions (public economy and welfare economy) which serve as its basic justification. Further reading. Chervel M. (1987), Calculs économiques publics et planification. Les méthodes d'évaluation de projets, Paris: Publisud, 268 p. A detailed presentation of all techniques for public economic calculation. The main aspects are addressed and a classification proposed. Commission Européenne (1997), Guide de l'analyse Coût­Avantages des grands projets dans le cadre de la politique régionale communautaire, DG XVI, Bruxelles, 90 ρ A guide intended for cost­benefit analysis prior to the implementation of major projects. Describes the main steps of the method. Diamond D; and Spence N. (1983), Regional Policy Evaluation. A Methodological Review and the Scottish Example, Gower International Library of Research and Practice, Aldershot, 170 p. Tools for making a judgement A presentation of the main evaluation tools taken from economic theory applied to regional development policy. A critical approach to cost-benefit analysis. Highly accessible. Gauthier G and Thibaut M (eds. ) (1993), L'analyse coût-avantages, défis et controverses, Paris: Cetai, Economica, 526 p. Accessible only to the initiated. A compilation of texts by various specialists on the advantages and limits of cost-benefit analysis, particularly the epistemologica! aspects. Nas T. F. (1996), Cost benefit Analysis. Theory and application, London: Sage Publications A manual that considers all the micro-economic foundations of cost-benefit analysis and includes case studies. Accessible only to the initiated. More in-depth: contribution of the technique to the evaluation of projects Once the project has been clearly identified, its objectives defined, its feasibility established and alternative options considered, the evaluation team is in a position to launch the different phases of quantification of the project: financial analysis, cost-benefit analysis and sensitivity analysis. This description of concepts is completed by a synthetic presentation of the observations made on the basis of an analysis of the major projects co- financed by the Structural Funds. Financial analysis the that investment project requires The financial analysis of the programming of this investment and the expected income and expenses be taken into account in such a way as to assess as satisfactorily as possible the annual inputs and outputs related to the realisation and management of the project. This approach is used to evaluate the financial weight of the project and to ensure that there is no risk of it being stopped at any stage due to a lack of liquidity. It is on the basis of the results of this analysis that into account numerous the CBA will be carried out, by complementary components of a socio-economic nature, through which the social impact of the project can be measured. taking To assess the net present financial value of a project, the values updated to a discount rate "r" of the future financial inputs (I) and outputs (O), (including the residual value of the investment at the end of the period) are taken into account: Principal evaluation techniques and tools VFAN = Øo-Cg+^-Cg /(1+r)+(l2-02) /(1+r)2 + (On-On) /(1+r)n The discount rate represents the opportunity cost of capital, i. e. the economic value of the capital invested in its optimum alternative use. The rate used in practice is often equivalent to the long-term rate of return on the capital invested in financial markets. Sometimes the discount rate is chosen in a rather arbitrary way. When comparing several projects, it is worth using the internal financial rate of return "rr", which is the value of "r" which satisfies the following equation: (lo-Cg+fjVCg /(1+r)+(l2-02) /(1+r)2 + (ln-On) /(1+r)"=0 factors, In the case of public investments, it often occurs that the rates of return observed are weak or even negative. This can be explained by a multitude including price distortions, problems with pricing, and of externalities. Hence, the need to extend the analysis to criteria other than financial variables, in order to judge if the public investment is opportune. This involves the identification and monetary evaluation of all the costs and benefits of the project. Cost-Benefit analysis In a similar way to financial analysis, the net present economic value is calculated by means of the following formula: VEAN = ( B Q - C O M B ^ C ,) /(1+r)+(B2-C2) /(1+r)? + (Bn-Cn) /(1+r)" in which B¡ represents all the benefits expected in period i and C¡ all the costs to cover during the same period. If the net present economic value is more than zero, then the evaluation concludes positively about the project. The coefficient "r" represents the social discount rate used to update all the costs and benefits as a function of the reference year, defined in this case as year 0. The analysis of major European projects of the first and second generations has shown discount rates varying from 3% to 10%. Generally, the advised rate is 5%. Still by analogy with financial analysis, the rate of internal economic return on the project "rr" is the value of "r" which satisfies the following equation: ( B o - C J + ^ - C ,) /(1+r)+(B2-C2) /(1+r)2 + (Bn-Cn) /(1+r)"=0 Since all the costs and benefits of a project cannot systematically be translated into monetary terms, the analysis will be complemented by an in-depth study on the non-monetary aspects related to the realisation of the project. Tools for making a judgement Analysis of sensitivity and risk Depending on the case, the results of the cost-benefit analysis can be extremely sensitive to a change in the working hypotheses. The analysis of sensitivity consists of varying the value of several key variables, so as to assess the impact of changes in hypotheses on the project's rate of return. If the modification of a key variable results in a significant change in the rate of return, it is essential to improve the reliability of the measure obtained for this variable. Major public projects are no more exempt from uncertainty than are private investments. Since cost-benefit analysis is based on assumptions about the expected impacts of the project, it is essential to assess the degree of risk involved in the project. Risk analysis will be carried out by linking probabilities to alternative values of crucial variables, by determining the economic rates of return associated with them, and then by calculating the distribution of probability of these rates of return. This permits an evaluation of the probability of the project having an economic rate of return that is smaller than the minimum rate of return required. Risk analysis enables decision-makers to make their final decision with full knowledge of the risks facing the project. Rafes of return of European projects Sectors Mid­ term Number of projects (*) Average internal financial rate of return Number of projects (*) Average internal economic rate of return Number of projects (*) Energy Water and environment Transport Industry Other services Total 24. 7 29. 1 26. 6 8. 8 14. 2 20. 1 9 47 127 96 10 289 7. 0 -0. 1 6. 5 19. 0 4. 2 11. 5 6 15 55 68 5 149 12. 9 15. 8 17. 1 18. 4 16. 3 16. 8 6 51 152 14 10 233 (') Projects for which data is available Source: Guide de l'analyse des coûts et avantages des grands projets, Commission des communautés européennes Principal evaluation techniques and tools For the programming periods 1989-1993 and 1994-1999 corresponding to the first and second generations of programmes, resulting from the reform of the Structural Funds, the mid-term horizon (economic lifespan of the project), the average internal financial rate of return and the average internal economic rate of return of a set of 400 major projects co-financed by the Structural Funds were analysed. This highly instructive synthesis enables the most important aspects of the contribution of CBA to the analysis of large projects to be pinpointed. Firstly, the average lifespan of projects varies considerably from one field to the next. With an average of 20 years, the lifespan varies from 9 years for industrial infrastructure to 30 years for environmental infrastructure. An assessment of the lifespan of a project is important if the potential mid- and investments are to be correctly long-term estimated. impacts of the planned Secondly, the estimation of the internal financial rate of return is used to define the financial needs of the project throughout its life. This financial analysis is also vital for the programming of investments. Whereas financial returns are high for productive infrastructures, which usually generate substantial income (rents or selling prices of property and facilities, selling price of services, etc. ), and are generally around 19%, the financial rate of return of environmental facilities, which are very rarely priced at market prices, is very low (-0. 1 % on average). For projects characterised by a low financial rate of return, financial programming is all the more important when the question of the viability of the project and of its real impact on public finances is asked. Thirdly, the assessment of costs and socio-economic benefits of the project is used to evaluate its real economic value and its contribution to economic and social welfare - the ultimate aim of a CBA. An initial observation is that the internal economic rate of return differs significantly from the internal financial rate of return. For projects co-financed by the Structural Funds, the average internal economic rate of return varies between 12. 9% in the energy sector to 18. 4% for the industrial sector. These figures highlight the advantages of the cost-benefit approach which enables the analysis to be extended beyond the strict framework of financial analysis by taking account, in particular, of the externalities that the project will generate. IV Recommendations for selecting and assembling tools (1) prospective The most suitable tools must be identified for each specific evaluation. Part four IV provides assistance for designers of evaluations, by describing typical situations: examination of a multi-sectoral programme: (2) retrospective examination of a multi-sectoral programme; (3) more in-depth in-depth treatment of a retrospective analysis. In each of these four situations, the appropriateness of tools presented in this Volume is reviewed. This Volume concludes with recommendations for the assembly of several tools within the same evaluation. treatment of a prospective analysis; (4) more Guide for identifying the most appropriate tools The identification of the most appropriate tools must be done on a case by case basis, in relation to the nature of the object to be evaluated and, above all, in relation to the questions asked by the commissioner. It is therefore out of the question to provide a general "blueprint". However, the designers of evaluations can be assisted by the description of several typical situations. First, the object to be evaluated may be an entire multi-sectoral programme or, by contrast, a particular evaluative question that needs to be considered in greater depth. From the point of view of the choice of tools, it is also important to take into account the time dimension of the question posed. Is the analysis to be prospective or retrospective? An analysis is retrospective if it can be answered on the basis of observations of effects in the field. This is the case of ex post evaluation. It is prospective in the opposite case, particularly for ex ante evaluation. Intermediate evaluation may comprise a mixture of prospective analysis (e. g. what is the expected impact of projects that have already been selected?) and retrospective analysis (e. g. what are the first results obtained in the field?). The combination of these distinctions allows the definition of the evaluation situations presented in Box 29. Box 29 - Typical situations Overall evaluation of a multi-sectoral programme More in-depth study of an evaluative question prospective analysis retrospective analysis Situation 1 Situation 2 for example: ex ante or intermediate evaluation of a regional programme or a CSF for example: intermediate or ex post evaluation of a regional programme or a CSF Situation 3 Situation 4 for example: ex ante or intermediate evaluation of a measure or a sectoral programme for example: intermediate or ex post evaluation of a measure or sectoral programme The guide presented below proposes an aid for the selection of tools in these four situations. It is limited to those tools figuring in an information Principal evaluation techniques and tools sheet in this Volume. The guide's approach could easily be extended to other tools. It does not recommend the systematic application of any specific tool to a particular situation, but proposes several tools from which only those (or the one) best suited to the situation in question should be chosen. Tools for the overall evaluation of a programme, prospective analy­ sis Box 30 - Overall evaluation of a programme, prospective analysis là SWOT Analysis Impact mapping Colour vote O <> -Ó- Individual interview <> -o- -Ó- Focus group Case study Geographical information system Input-output model Macro-economic model Expert panel Multicriteria analysis -o o o -ó- O Tools ( J Appropriate O Application subject to certain conditions In this situation, the main lines of the programme have already been proposed. The evaluation must help to ensure that the planned programme has objectives that are soundly justified, clear, graded, and logically connected to the planned actions. If necessary, the evaluation may include field observation to check the nature and importance of needs. It may also Selecting and assembling tools include a forecast of effects, particularly for helping to quantify objectives. An evaluation of this type warrants the use of fools, if adequate funds and time are available. Tools for the overall evaluation of a programme, retrospective analysis Box 31 - Overall evaluation of a programme, retrospective analysis li SWOT Analysis <y- Impact mapping Colour vote Individual interview Focus group Case study -<>- <^ Geographical information system —Ò- Shift-share analysis Input-output model Macro-economic model Expert panel Multicriteria analysis Benchmarking -o- <^-o- o- -Ò- Tools ( ) Appropriate O Application subject to certain conditions In this situation, the evaluation must observe, analyse and judge the effects produced by the programme In the field. These effects are numerous and are the result of multiple processes that differ, depending on the components of the programme. Moreover, the evaluation may have to take into account the differences in the partners' points of view. This situation presents constraints, which limit, to an important extent, the choice of tools. Principal evaluation techniques and tools Tools for in-depth evaluation, prospective analysis intended In this situation, an aspect or part of a complex programme is evaluated in depth. This corresponds to the case of a simple intervention, consisting of similar actions, for a relatively homogeneous public and possessing a primary objective. The prospective analyses that are generally performed in this situation examine the feasibility and the relevance of the planned intervention. Consequently, the evaluation must help to explain how this intervention will produce these effects, what the estimated extent of the effects will be and to what extent the planned intervention responds to a real need. Box 32 - Tools for in-depth evaluation, prospective analysis J 55 O <> <> Logical framework Metaplan® Colour vote Delphi survey Individual interview Focus group o- -o—o- Cost-effectiveness analysis -Ó- <y- Benchmarking Cost-benefit analysis Multicriteria analysis Expert panel -Cr- -o— Tools Cj Appropriate O Application subject to certain conditions Tools for in-depth evaluation, retrospective analysis In this situation, the evaluation observes, analyses and judges the effects obtained in the field. There are few of these effects and they are produced Selecting and assembling tools by processes that can be schematised. It is therefore possible to study them in-depth. Time may also be devoted to looking for suitable benchmarks for judging whether the effects obtained are sufficient or not. Most classical evaluation tools are applicable to this situation. Box 33 - In-depth evaluation, retrospective analysis fiS Logical framework Metaplan® Colour vote Questionnaire survey Focus group Case Study Ethnographic observation Factor analysis Comparison group Regression analysis Cost-effectiveness analysis Benchmarking Expert panel -o- <> <y- -o- -o—o -o- o—o Tools \^) Appropriate O Application subject to certain conditions Recommendations for assembling several tools Procedure for assembling tools The choice of tools and their assembly is generally a process of trial and error. At first, the nature of the evaluation (a multi-sectoral programme or a less complicated intervention) and that of the analysis to be performed (prospective or retrospective) suggest that certain tools might be appropriate. The preceding guide can help to identify these tools. At this stage, a temporary method is constructed and its quality checked by means of the following questions: • Are the eight standard functions of an evaluation dealt with? Are some functions irrelevant? Have some functions been forgotten? • Which tools have been chosen? What functions do they fulfil? Are they well suited to the functions to be dealt with? Does their use comply with the established procedures? • Is the method as a whole sound and homogeneous? The latter question is perhaps the most important, for an evaluation method can be compared to a chain. Each function is a link in the chain and each link is stronger when treated with an appropriate tool. The metaphor of a "technical chain" highlights the interdependence of the different tools assembled within an evaluation method. A link is strong if the technical underpinning of the corresponding function is sufficient in relation to the risk of the evaluation results being challenged. A link does not necessarily have to be "tool(s)-based" to be strong. However, if it is, its strength depends on the appropriateness of the tool in relation to the question posed and the situation. The entire technical chain is as strong as its weakest link. It is therefore of little use to employ a sophisticated and expensive tool to estimate effects, if the data to which the tool is applied has been collected in a contestable manner. The soundness of an evaluation method is only partially related to the quality of the tools used. It is less the highly technical way in which certain functions are dealt with, than the fact that all the functions are treated with the same level of technical underpinning, which makes an evaluation good. Thus, when a temporary evaluation method has been constructed, it is essential first to check whether it has any weak links. Principal evaluation techniques and tools Box 34 - The technical chain is as strong as its weakest link Clarifying/grading effects and criteria Choosing observation Instruments Defining the observation field Collecting data Comparing data Estimating effects Judging in terms of the different criteria Formulating a synthetic judgement [ ) -— - Weak link Tools fj Strong link The evaluation method is then constructed by successive improvements, until the following requirements are met: • all the relevant functions have been dealt with • the selected tool(s) are suited to the situation and the function they fulfil • the method constructed is homogeneous and has no weak points • the method chosen is compatible with the cost and schedule constraints of the evaluation. It is not possible to seek maximum robustness, nor to choose numerous and sophisticated tools, If the time and budgetary constraints are too severe. The realism of a method demands that its strength, but also its cost, are consistent with the importance of the evaluation for its clients. When the method proposed does not meet this demand for realism, it must be adapted by eliminating certain tools or replacing them by others that are easier to implement. Selecting and assembling tools A problem that is often forgotten is that a method is not constructed for each evaluation but for each question within an evaluation. Assuming a single evaluation contains the following two questions: A. Was the impact of the programme sufficient in terms of reducing unemployment in the region? B. Which measures of the programme are the most efficient for creating jobs, and why? Question A must be treated by estimating the impact of all the measures, if necessary by means of an econometric model. On the other hand, for question Β it would be better to use a method based on observation and analysis of particularly successful projects in terms of job creation (case study). In some cases a single method may answer two or more questions, but in general the answers will not all be equally robust. It could therefore be said that the cost of an evaluation increases in proportion to the number of questions in the terms of reference (and to which a real answer is given). Three examples of assembling tools The following three propositions are based on real examples presented in the preceding chapters. The first example is the overall evaluation of the Hainaut regional development programme, presented in the information sheet relating to multicriteria analysis (page 113) and in Volume 4 of this Collection, devoted to innovative techniques developed in the framework of the MEANS programme. This intermediate evaluation was intended to compare the effectiveness of the main measures of the programme. It ended with a multi-criteria analysis that was used to make a synthetic judgement. This judgement was based on a qualitative estimation of the effects of each measure. In order to strengthen the estimation of impacts, the method could be developed in the following way: • clarification of the effects to be evaluated by means of an impact map, • collection of indicators and available secondary data on each measure evaluated, • organisation of one or more focus groups for each of the evaluated measures; qualitative estimation of impacts, Principal evaluation techniques and tools • comparison between measures and synthesised judgements, by means of a multicriteria analysis. Box 35 - Example of the combination of tools for the intermediate evaluation of an entire programme Clarifying and grading the effects to be evaluated, defining criteria Choosing observation instruments Defining the observation field METHOD Impact mapping \ Monitoring indicators Collecting data Comparing data Secondary data Tools Estimating effects Focus groups Judging in terms of the different criteria Formulating a synthetic judgements Multicriteria analysis The second example is based on the evaluation of job creation in Wales, described on page 178 (see the information sheet on Cost-effectiveness analysis) and in Box 3. This ex post evaluation aimed at making an in-depth analysis of an impact. It included a questionnaire survey on assisted businesses, used to estimate deadweight and displacement effects, and a cost-effectiveness analysis, used to judge the efficiency of the programme, compared to a Scottish programme. The weak links in this evaluation were probably the estimation of deadweight effects and the comparison of efficiency. To enhance the evaluation on this point, the method could have been improved in the following way: • definition of the field of observation in three groups: Welsh businesses assisted by the programme, non-Welsh businesses assisted by other programmes considered to be efficient, and Welsh businesses not affected by the programme. Selecting and assembling tools • collection of data by means of a questionnaire survey • estimation of effects by means of comparison groups • cost-effectiveness analysis and judgement in relation to other successful programmes. This method would of course have been far more expensive. Box 36 - Example of the assemblage of tools for an in-depth ex post evaluation Clarifying and grading the effects to be evaluated, defining criteria Choosing observation instruments Defining the observation field M E T H OD Indicator of job creation Recipient and non-recipient firms inside and outside the eligible area Collecting data Comparing data Questionnaire M survey ΤηηΙς Estimating effects Comparative * analysis /. / Formulating a synthetic judgements Cost-effectiveness analysis v > impact of the probable The third example is based on the ex ante evaluation of European support to the Republic of Ireland, presented in Box 13. This evaluation aimed at estimating the assistance on economic development, particularly in terms of the per capita GDP. The evaluation comprised a first series of micro-economic estimations (impact on public spending, on productivity and on production costs). These estimations were then introduced into a macro-economic model. Estimations of micro- economic impacts on productivity and production costs were probably the weak links of this evaluation. Moreover, the evaluation stopped at the estimation of macro-economic effects and did not formally go so far as to judge whether the effects were sufficient or insufficient. Principal evaluation techniques and tools To improve the homogeneity of the evaluation method, a little less money could have been spent on the econometric model, and the weak links could have been strengthened in the following way: • clarifying and grading the probable micro-economic impacts of the different measures, if necessary by means of an impact matrix; • requesting an expert panel to estimate the main micro-economic impacts; • estimating the economic consequences at the national level, by means of a macro-economic model as was done in practice; • forming an expert panel and asking it to decide whether the probable impacts were sufficient or insufficient. At little extra cost, this method would have been more credible and conclusive. Box 37 - Example of the assemblage of tools for the ex ante evaluation of an entire programme METHOD Clarifying and grading the effects to be evaluated, defining criteria Identification of major macro-economic impacts Estimating effects Expert panel + Econometric model Tools Formulating a synthetic judgement Expert panel Conclusion The use of tools has numerous advantages in terms of providing assurances about the results to be obtained, of certainty about the contents of the evaluation, of controlling costs, of comparability of results and of quality control. A more technical evaluation is often more straightforward to commission, easier to control, more credible in the eyes of the readers of the report and more predictable for the commissioner. It can be understood that all these advantages make evaluation tools very attractive. technical complexity Since the value of tools used in is attractive, evaluations is often exaggerated. An entire work of evaluation is sometimes summarised by the name of the tool employed (e. g. cost-benefit evaluation, macro-economic takes on an emblematic dimension. evaluation), which thus It is however dangerous to overstate the value of a tool while forgetting that it most often fulfils a single evaluation function, is adapted to a certain context, presents constraints specific to the availability of data, and has a certain cost and duration of use that cannot be compressed. To choose a tool without taking all these dimensions into account can have serious consequences: time overruns, absence of conclusions, conclusions lacking in credibility, irrelevant conclusions, etc. This Volume should therefore not be considered as an advertisement for highly technical evaluations. Its ambition is rather to adapt the level of technical complexity to the strategic importance of the questions posed and the budget and time scale allocated for the evaluation. It should thus be considered as assisting the mastery of the technical aspects of evaluations. Index Index analysis of causality 27 approach 18 Β benchmarking 115, 193, 201 Beutel model 98 casestudy 59, 73, 8 1, 132, 151, 167 colour vote 39, 55, 115, 135, 193 comparison group 8 1, 132, 167, 179, 229 concept mapping of impacts 39, 47, 135, 227 cost-benefit analysis 193, 207 cost-effectiveness analysis 19, 23, 193, 195,229 criterion 25, 28 deductive approach 27 Delphi survey 167, 175, 193 descriptor 26 design 18 factor analysis 8 1, 131, 167, 169 field 26 focus group 59, 67, 227 formative evaluation 29 function 18, 24, 225 generic tool 24 geographical Information system 36, 81,83 group interview 151, 167 Η HERMIN model 29, 112 I impact matrix 39, 135, 230 in-depth evaluation 30, 219, 222, 228 indicator 26 individual interview 59, 6 1, 132, 151 inductive approach 27 input-output model 8 1, 95 intermediate evaluation 30, 219, 227 logical framework 40, 135, 137 economic technique 12 ethnographic observation 151, 161, M 167 ethnomethodology 162 evaluative question 25 ex ante evaluation 30, 219,229 ex post evaluation 30, 219, 228 experimental technique 12 expert panel 115, 117, 131, 193, 230 macro-economic model 29, 37, 8 1, 103, 230 METAPLAN® 20, 135, 145 method 18, 225 methodology 19 multicriteria analysis 115, 123, 193, 228 Principal evaluation techniques and tools N naturalistic approach 12 net effect 27 recapitulative evaluation 29 regression analysis 20, 8 1, 167, 187 overall evaluation 30, 219, 220, 221, 227, 229 secondary data 27 shift-share analysis 8 1, 89 specialised tool 24 structuring 25 SWOT analysis 2 1, 39, 4 1, 115 pragmatic approach 12 primary data 26 procedure 18 qualitative analysis 27 quantitative analysis 27 QUEST model 111 questionnaire survey 59, 131, 151, target public 31 technical chain 225 technique 18 thematic evaluation 31 tool 18 V 153,229 variance analysis 167 Acknowledgements The preparation of this Volume took place over several years from 1995 to 1998. Preparatory works included the following steps : • Five case studies were carried out on the basis of evaluations selected for their use of well-known evaluation techniques. Five European scholars took part in these studies which allowed procedures for the use of the evaluation tools presented in this Volume to be determined. • Nearly 300 desks officers from the European Commission attended training seminars which included a session specially devoted to evaluation tools. The contents of these training seminars evolved over time to meet participants'needs. This Volume includes the results of these interactions. • Several experts participated in the writing and verification of the information sheets. The production of this Volume was coordinated by Miguel Benito Alonso (D. G. XVI. G. 2 : Coordination of Evaluation) and involved several Directorate-Generals, notably D. G. II, V, VI, XIV and XVI and in particular Philippe Goybet, Anastassios Bougas, Martin Hallet, Andrea Mairate and Julius Op de Beke. This Volume has been written by the Centre for European Evaluation Expertise, notably with the participation of Kristian Colletis-Wahl, Olaf Rieper and Kevin Williams. It was translated into English by Liz Libbrecht. The authors wish to thank all those who contributed, either directly or indirectly, to the production of this Volume, and in particular the following experts: Henri Capron (Université Libre de Bruxelles, Belgium), Dr. Basil Cracknell (consultant, UK), Prof. Massimo Florio (CSIL, Milan), Christophe Kellerhals (Evaluanda, Genève,), Steve Martin (Warwick Business School, UK), Thanos Skouras (Athens University of Economics and Business), Evert Vedung (Uppsala Universitet). European Commission Evaluating socio-economic programmes Principal evaluation techniques and tools Luxembourg: Office for Official Publications of the European Communities, 1999 1999 — 237 pp. — 14. 8 χ 21 cm ISBN 92-828-6626-2 Venta · Salg · Verkauf · Πωλήσεις · Sales ■ Vente · Vendita · Verkoop * Venda · Myynti · Försäljning BELGIQUE­BELGlE ÖSTERREICH Jean Do Lan noy Avenue du Roi 202/Koningslaan 202 B­1190 Bru ielle s. Brussel Tel (32­2) 538 43 08 Fax (32­2) 538 08 41 Email: iean. de lanrtoyOinloboard. be URL. hrtpJ/wwwjean­de­iannoy be La librairie européenne/ De Europese Boekhandel Rue de la La 244/Welstraai 244 B­1040 Bruì eil es/Brussel Tel (32­2) 295 26 39 Fax (32­2) 735 08 60 E­mail mailOlibeurop be URL http ' www hbeurop be Moniteur belge/Bel g lach Staatsblad Rue de Louvam 40­42 Le uve π so wog 40­42 B­1000 Bruiellos,Brussel Tel (32­2)552 22 ti Fax (32­2) 511 01 B4 DANMARK J. H. Schultz Information A/S Herstedvang 10­12 DK­2620 Albertslund Tlf (45)43 63 23 00 Fai (45) 43 63 19 69 E­mail schultzOschuttz dk URL http //www. schuitz. dk CEU TSCHLArJD ^ _ ^_ Bundeaanzelger Vertag GmbH V e ri no b saoi e> lung Amsterdamer Straße 192 D­50735 Köln Tel (49-221)97 66 80 Fax (49­221)97 66 82 78 E­Mail vertrieb θ bunde sa nzei ger de URL lmp_//www bundesanzeiger de ΕΛΛΑΔΑ/GREECE G. C. Eleftheroudakis SA I n le mal lonal Book store PanepistimKXj 17 GR­10564 Alhina Tel (30­1)331 41 ΒΟ. Ί/2/3/4'5 Fax (30-1) 323 98 21 E-mail elebooksOnelor gr ESPANA Boletín Oficial del Estado Trafalgar. 27 E 28071 Maand Tel ( 3 4 ) 9 1 5 1 8 21 11 (Libros). 913 B4 t7 15(Suscnp) Fax (34) 915 38 21 21 (Libros), 913 84 17 14(SuSCnp) E­mail dientes β com boe es URL: http//www boe es Mundi Prensa Libros. SA Castellò. 37 E ­28001 Madrid Tel (34)914 36 37 00 Fax (34) 915 75 39 98 E­mail: libreria βmuridrpronsa es URL http//www mu ndipr en sa com FRANCE Journal officiel Service des publications des CE 26, rue Desaix F­75727 Paris Cedex 15 Tel (33) 140 58 77 31 Fax (33) 140 58 77 00 URL http /fwww |oumal­orficiel gouv fr IRELAND Government Supplies Agency Pubi ca lion s Section 4­5 Harcourt Road Dublin 2 Tel (353­1)661 31 11 Fax (353­1) 475 27 60 ITALIA Licosa SpA Via Duca di Calabria. 1/1 Casella postale 552 1­50125 Firenze Tel (39) 055 64 83 1 Fax (39) 055 64 12 57 E­mail lieosaOtlbcc il URL hrtp /fwww ftbcc lUicosa LUXEMBOURG Messageries du livre SARL 5. rue Raiffeisen L­2411 Luxembourg Tfri (352)40 10 20 Fax(352)49 06 61 E­mail ma. lOmdl lu URL http //www mdl lu NEPER LAN0 SDU Servicecentrum Uitgevers Chnstorfel Plantons!raat 2 Postbus 20014 2500 EA Den Haag Tel (31­70) 378 98 B0 Fax (31­70) 378 97 83 E­mail sduOsdunl URL http /,­www sdu nl M a n i s c he Verlage­ und Universitatsbuchhandlung GmbH Kohl markt 16 A­1014Wien Tel (43­1)53 16 11 00 Fax (43­1) 53 16 11 67 E­Mail bestellen O manz coat URL http//www manz at/mdex htm PORTUGAL Distribuidora de Livros Bertrand L d. ' Grupo Bertrand. SA Rua das Torras dos Vales. 4­A Apartado 60037 P­2700 Amadora Tel. [351­1)495 90 50 Fax (351­1)496 02 55 Imprensa Nacional­Casa da Moeda, EP Rua Marqués Sá da Bandella. 16­A Ρ­1050 Lisboa Codex Tel (351­1)353 03 99 Fax (351­1) 353 02 94 E­maJ defíneme mail. teiepac. pt URL: httpi/www incm. pl SUOMI FIMLAND Akateeminen Kirjakauppa/ Akademiska Bokhandeln Keskuskatu 1 /Centralgatan 1 PL/PB 128 FIN­OOlOt HelsinkiHelsingtors P/tfn (358­9) 121 44 18 F/lax (358­9) 121 44 35 Sähköposti akatilaus β akateeminen com URL: http^/www akateeminen com SVER ι. 1 __ BTJ AB Traktorvägen 11 S­221 82 Lund TIn (46­46) 18 00 00 Fax (46­46) 30 79 47 E­post btteu­pubObt) se URL htlp. //www bl| so UNITED KINGDOM The Stationery Office Ltd International Sales Agency 51 Nine Elms Lane London SW8 5DR Tel. (44­171)873 90 90 Fax (44­171) 873 84 63 E­mail, ipa enquiñes θ tneso. co. uk URL http. //www lhe­stallone ry­o ff ice co uk ISLAND Bokabud Larusar Blondel SkóiavôrduslKj 2 IS­IOt Reykiavik Tel (354)551 56 50 Fax (354) 552 55 60 NORGE S weis Norge AS Ostenjoveien 18 Boks 6512 Etterstad N<06O6Oslo Tel (47­22)97 45 00 Fax (47­22)97 45 45 SCHWEIZ/SU ISSE. 'SVIZZERA Euro Info Center Schweiz c/o OSEC Stampi e nbachsttaße 85 PF 492 CH­B03S Zurich Tel (41­1)365 53 15 Fax (41­1)365 54 1t E­mail eicsOosoc eh URL http //www osee ch/eics BÀLGARUA Europres a Euromedia Lid 59. Orvd Vitosha BG­1000 Sofia Tel (359 2)960 37 66 Fax (359­2) 980 42 30 E­mail Milena O mboxcit bg CESKA HEPUBLIKA ÜSIS NIS­prodejna H a velkova 22 CZ­130 0 0 P r a h a3 Tel (420­2)24 23 14 86 Fd. (420­2)2­1 23 II 14 E­mail nkpospOdecms cz URL htlpy/usiscrcz CYPRUS Cyprus Chamber of Commerce and Industry PO Box 1455 CY­1509 Nicosia Tel (357-2)66 95 00 Fax (357­2)66 10 44 Email: demetrapOcco org cy EESTI Eesti Kau band u s ­ TM st us k oda (Estonian Chamber of Commerce and Industry) Toom­Kooli 17 EE­0001 Tallinn Tel (372)646 02 44 Fax (372) B46 02 45 E­mail einfoOkoda ee URL http//www koda ee HRVATSKA Mediatrade Ltd Pavia H alza 1 HR­10000 Zagreb Tet (385­1)481 94 1 Fax (385­1) 481 94 1 MAGYARORSZAG Euro Into Service Europa Haz Marnitszirjöt PO Box 475 H­1396 Budapest 62 Tel (36­1)350 80 25 Fax (36­11350 90 32 E­mail ouromfoOmail maiav. hu URL. http//www euromlo. hu/index. htm MALTA Miller Distributors Ltd Malta International Airport PO Box 25 Luqa LOA 05 Tel ( 3 5 6 ( 66 44 88 Fax (356) 67 67 99 Email: gwirthöusa net POLSKA A rt Polona Krakowskie Przedmiescie 7 Skr pocztowa i 0 0i PL ­00­950 Warszawa Tel (48­22)826 12 01 Fax (48-22)826 62 40 E­mail ars_poiebovy hsn. com pi Euromedia Str. G r al Berihelot Nr 4i RO­70749 Bucuresti Tel (40­1)315 44 03 Fax (40­1) 314 22 86 ROSSIYA CCEC 60­leliya Oktyabrya Av 9 1 17312 Moscow Tel (7­095) 135 52 27 Fax(7­Q95) 135 52 27 Centrum VT1 SR Nam Slobody, 19 SK­8l223Braiisiava Tel (421­7)54 41 Θ3 64 Fax (421­7) 54 41 83 64 E­mail, eutopOtbbi sltk. iluba sk URL htlp /'www sltk stuba sk SLOVENIJA Gos podara k i Vestn I k Dunajska cesta 5 SLO­1000 Ljubljana Tel (386)613 09 16 40 Fax (386)613 09 16 45 E­mail europOgvestnik. si URL httpy/www. gvestnik. si TURKIYE Dunya Infoiai AS 100. Yil Mahallossi 34440 TR 80050 Bagcilar­Istanbul Tel (90­212)629 46 89 Fax (90­212) 629 46 27 E­mail. inlotelOdunya­gazelecom tr AUSTRALIA Hunter Publication! PO Box 404 3067 Abbotsford. Victona Tel (61­3)94 17 53 61 Fax (6t­3)94 19 71 54 E­mail ipdaviesOozemailcomau CANADA Les éditions La Libert« Inc. 3020, chemin Sainte­Foy G1X 3V Samte­Foy. Québec Tel (1­418)658 37 63 F a i ( t ­ B 0 0 ) 5 67 54 49 E­mail liberteOmediom qc ca Renouf Publishing Co. Ltd 5369 Chemin Canolok Road Unit 1 KU 9J3 Ottawa, Ontano Tel ( 1 - 6 1 3 ) 7 4 5 26 65 Fax (1-613)745 76 60 E­mail order dept β re nou (books com URL http //www renoutbooks com EGYPT The Middle East Observer 41 Sherif Slreet Cairo Tel (20­2)392 69 19 Fax (20­2) 393 97 32 E­mail mafoudaOmeobserver com eg URL http//www meobserver com. eg INDIA EBIC India 3rd Floor. Υ Β Chavan Contre Gen J BhosaloMaig 400 021 Mumbai Tel (91­22)282 6064 Fax (91-22)285 45 64 E­mail ebicttgiasbmOl vsnl net in URL' httpy/www ebiandia com ROY international 41. MishmarHayarden Street PO Box 13056 61130 Tel Aviv Tol (972-3)649 94 69 Fax (972­3) 648 60 39 E ­ mail. royl β netvision. nel. il URL: httpy/www royint. co. il Sub­agent for the Palestinian Aulhonty: Index Information Services PO Box 19502 Jerusalem Tel (972­2)627 16 34 Fax (972­2) 627 12 19 PSklapan Asahi Sanbancho Plaia «206 7­1 Sanbancho, Chiyoda­ku Tokyo 102 Tel (81­3)32 34 6 9 21 Fax (81­3) 32 34 69 15 E­mail booksOpsi­|apan co ¡ρ URL: httpjfvtww psi­japan com EBIC Malaysia Level 7, Wisma Hong Leong 18 Jalan Perak 50450 Kuala Lumpur Tel. (60-3)262 62 98 Fax (60­3) 262 61 98 E­mail ebic­kiemot. net my Mundi Prensa Mexico. SA de CV Rio Panuco No 141 Colona Cuauhtemoc MX­O65O0 Mexico. DF Tol (52­5)533 56 58 Fax (52­5) 514 67 99 E­mail. 101545 23610compuseive. com EBIC Philippines 19th Floor. PS Bank Tower Sen Gil J Puyat Ave. cor Ttndalo SI Makati City Metro Manilla Tel (63-2) 759 66 BO Fax (63-2) 759 66 90 E­mail ocepcom© globo com. ph URL: htlp //www. eccp. com EBIC Sri Lanka Trans Asia Hoi el 115 Sir chittam palam A Gardiner Mawalha Colombo 2 Tel (94­1)074 71 50 78 Fax (94­1)44 87 79 E­mail ebicslOitmin. com EBIC Thailand 29 Vamssa Building, eth Floor Soi Chidlom Ploenchit 10330 Bangkok Tol (66-2)655 06 27 Fax (66­2) 655 06 28 E­mail: ebicbkkOksclS. thcom URL: htTpïwww ebicbkk org UNITED STATES OF AMERICA B erna n Associates 4611­F Assembly Dnve Lanham MD20706 Tel (1­800) 274 44 47 (loll (roe lelephono) Fax (1 ­800) 865 34 50 (Ion (roe lax) Email queryObeman. com URL: http://www bernan com Bitte wenden Sie alch an ein Büro Ihrer Wahl/ Please contact lhe s a l si o t i l e· ol your choleo/ Veuillez vous adresser au bureau de vente de votre choix Office for Officiât Publications ol the European Communities 2, rue Mercier L­2985 Luxembourg Tel (352) 29 29­42455 Fax (352) 29 29­42758 E­mail: info mfoOopoce cec be URL' http://eur­op ou mt The MEANS collection: "Evaluating socio-economic programmes" This six-volume set gives wider public access to the lessons of the MEANS programme which the European Commission introduced with a view to impro­ ving and promoting evaluation methods. Its publication meets a growing need for reliable benchmarks to deal with the problems of evaluation encoun­ tered when implementing public measures in general and Structural Fund ope­ rations in particular. In this context, the set constitutes a unique methodological guide that can solve the technical and organisational problems arising in the course of evaluation work and for which the existing literature provides little help. The collection is aimed not only at professional evaluators, for whom it will provide a valuable handbook, but also at non-specialists, in particular manag­ ing authorities and structural programme Monitoring Committees. Evaluation design and management Each volume is designed to be independent of the others, yet they are also complementary. Together, they deal with all aspects of the evaluation of socio­ economic programmes: I. II. Selection and use of indicators for monitoring and evaluation III. Principal evaluation techniques and tools IV. Technical solutions for evaluation within a partnership framework V. Transversal evaluation of impacts on the environment, employment and other intervention priorities VI. Glossary of 300 concepts and technical terms OFFICE FOR OFFICIAL PUBLICATIONS OF THE EUROPEAN COMMUNITIES L-2985 Luxembourg