Evaluating socio-economic programmes : volume 4, Technical solutions for evaluation within a partnership framework.
uctural Funds Evaluating socio-economic programmes Technical solutions for evaluation 3worl? MEANS Collection Volume 4 EC Structural Funds Evaluating socio-economic programmes Technical solutions for evaluation within a partnership framework Volume 4 European Commission MEANS is a programme of the European Commission. It is aimed at improving Methods of Evaluating Structural Policies. The MEANS Programme and Collection was supervised by Phillipe Goybet and Miguel Angel Benito Alonso, Unit for the Coordination of Evaluation in the Directorate for Coordination and Evaluation of Operations of DG XVI Coordination-evaluation@fmb. dg 16. cec. be The realisation of the Programme was entrusted to the Centre for European Evaluation Expertise (C3E) under the direction of Eric Monnier and Jacques Toulemonde. Responsibility for the opinions expressed in this book lies solely with the authors of the text: C3E - 13 bis Place Jules Ferry - F - 69006 Lyon - France - c3e@c3e. fr A great deal of additional information on the European Union is available on the Internet. It can be accessed through the Europa server (http://europa. eu. int) and on the Inforegio Website (http://inforegio. cec. eu. int). Cataloguing data can be found at the end of this publication. Luxembourg: Office for Official Publications of the European Communities, 1999 ISBN 92-828-6626-2 © European Communities, 1999 Reproduction is authorized, except for commercial purposes, provided the source is acknowledged. PRINTED ON WHITE CHLORINE-FREE PAPER Printed in Italy ^r Foreword is directly Developing good evaluation practices the has become a priority area for relevant Commission and to the effective use of Community resources. Indeed, the sound financial management principles which the Commission seeks to promote via its SEM2000 (Sound and Efficient Management) Programme rely, specifically, on the regular evaluation of the Community activities. The Directorates General responsible for structural policy have considerable expe­ rience in this area. Since 1988 they have made evaluation an integral part of each stage in the process of Community Structural intervention. Thus, the programme development stage includes a prospective evaluation; the implementation stage a strengthening of the monitoring systems and, at the end of the intervention, evaluation establishes what has been achieved in terms of programme impacts. However, the capacity of evaluation to contribute to the effectiveness and transparency of structural interventions rests ultimately on the integrity of the methods adopted to verify the credibility of results. This is why, four years ago, on the initiative for the Directorate General for Regional Policy and Cohesion (DG XVI), the MEANS programme was initiated as a response to the demand for developing evaluation methodo­ logy. This also include practices combining both general scientific requirements and the specific requirements of Community interventions whilst, at the same time, remaining accessible to a large audience. One of the more tangible results of the MEANS programme has been the publication of this six volume set by a group of independent experts, focu­ sing on the principal elements of socio-economic programme evaluation. A short summary of this work can be found at the beginning of the present volume. This collection covers the various methodological approaches and innovations that have taken place in the context of structural funds eva­ luation. It will enable those responsible for evaluation to develop their approach in a flexible and independent manner and for the various programme partners to exercise greater control over the evaluation pro­ cess. Attending the conferences of evaluation specialists and "clients" convened in Brussels (1995), Berlin (1996) and Seville (1998), I gained some impression of the interest that new evaluation techniques can gene­ rate. I would stress that the development of evaluation needs to progress in tandem with efforts to incorporate evaluation findings in the decision making process. This implies evaluation as a means of establishing whe­ ther policies are equitable, effective and relevant to the identified needs. In satisfying these requirements, evaluation should be seen as a valuable instrument affording both Member State and Community authorities the necessary guarantees that Community funds are being deployed in the best interests of the EU citizen. This is a cost worth paying when the goals of Economic and Social Cohesion and the European Union enlargement are at stake. Finally, I wish to extend my thanks to the members of the Committee of Independent Experts established at the launch of the programme. Their expertise and standing have contributed greatly to the quality and relevan­ ce of the MEANS programme. I therefore thank: E. Chelimsky (USA), ex- President of the "American Evaluation Association"; K. Kennedy (IRL), Director of the "Economic and Social Research Institute" (ESRI); J. R. Cuadrado Roura (SP), President of the "European Regional Science Association"; C. Seibel (F), Director of the "Statistiques démographiques et sociales de l'Institut National de la Statistique et des Etudes Economiques" (INSEE); L. Senn (I), Director of the Department of Economic Regional of the University Bocconi; E. Stern (U. K. ), Director of the unit of evaluation of the Tavistock Institute; A. Sorber (t) (NL), Head of the "Policy Analysis Department of the Ministry of Finance"; L. I. Strom (SV), Statens Institut för Regionalforsking (SIR); L. Tsoukalis (GR. ), professor at the University of Athens; H. Wollmann (D), professor at the University of Humboldt - Berlin and founder member of the "European Evaluation Society". E. Landaburu The MEANS Collection: "Evaluating socio-economic programmes" of European Union The evaluation structural interventions has entered into a phase of maturity, attested by the extension of work carried out by both the Member States and the Commission. Regulations based on decisions by the Council have largely contributed to the development of evaluation practice and have stimulated methodological research. initiated by the Commission Over the past ten years, the Commission has striven to promote and improve methods for evaluating Structural Funds. The main methodological conclusions have been debated and disseminated as part of the MEANS in 1991. programme Achievements to date provide enough material for a complete and accessible publication intended for a broad public: a six volume Collection covering all aspects of the evaluation of socio-economic programmes. These Volumes are written with a perspective that is broader than the current administrative framework, for this is likely to evolve more rapidly than professional standards. information on In general, the Collection is aimed at a readership of non- specialists seeking essential the evaluation of socio-economic development programmes. These programmes have multiple objectives, a partnership dimension and a multi-sectoral content - all characteristics which call for suitable evaluation methods and solutions not yet provided by the international technical literature. The complete Collection consists of 6 volumes designed to be complementary and to provide essential for professionals wanting to independently develop their own evaluation practices. Each volume can, however, be read and understood without having to refer to the other volumes in the Collection. recommendations information and Technical solutions for evaluation within a partnership framework The first Volume in the MEANS Collection entitled "Evaluation design and management" serves as an introduction both to the Collection and to the evaluation of socio-economic programmes. It enables readers to understand the implications of evaluation and to know what they can and cannot expect from an evaluation at a given time in a particular context. It covers all the elements needed to steer an evaluation project from A to Z: sharing of responsibilities, defining of the aims and method of an evaluation, establishing a schedule and budget, assessing the quality of the work performed and, finally, managing the use of conclusions. The second Volume focuses on the "Selection and use of indicators for monitoring and evaluation". It deals with the subject of indicators used for monitoring and evaluation. It shows how indicators can be used to support policy decisions, improve management and measure the overall performance of programmes. The volume proposes a methodological framework and a practical guide for the use of indicators in these different situations. The third Volume presents a selection of "Principal evaluation techniques and tools" applicable to the evaluation of socio-economic programmes. These tools are taken from relevant scientific disciplines: economics, sociology, management, geography, etc. Each one of the 23 tools is described in the form of an information sheet featuring a general description, the principles of its implementation, a list of its strengths and weaknesses for the evaluation of Structural Funds, an example of its application and a list of references. A number of methods adapted to the evaluation of socio­ economic programmes are the object of a more detailed description in the fourth Volume entitled "Technical solutions for evaluation within a partnership framework". Several pilot evaluations were implemented within the MEANS programme to design and test the following four innovations, the application of which is described in The MEANS Collection detail in this Volume: impact mapping, cross-impact matrix, score cards and multi-criteria analysis. The fifth Volume deals with "Transversal evaluation of impacts on the environment, employment and other intervention priorities" of the European Union and Member States. Among the questions addressed are impacts in terms of equal opportunities between men and women and the competitiveness of SMEs. The Volume provides the elements required for an analysis of those impacts which are not necessarily part of the explicit objectives of all the actions evaluated. Finally, the last Volume is a "Glossary of concepts and technical terms". The compilation of the Collection has necessitated the rigorous standardisation and definition of the terms used. Each of the first five volumes has an index of terms which, along with hundreds of definitions, are brought together in the glossary. Contents Introduction I Concept mapping of impacts A solution for evaluating different measures by means of common criteria Instructions for use of the concept mapping of impacts Using the concept mapping of impacts II Constructing and using scoring scales A solution for evaluating different measures in a homogeneous manner Instructions for using scoring scales Using ratings for synthesising the evaluation of measures III Analysing the programme using a matrix of cross impacts A solution for analysing synergy within a programme Instructions for using the matrix of cross impacts Empirically verifying the main synergies identified IV Using multicriteria analysis within a partnership framework A solution for judging the overall success of a programme Instructions for using multicriteria analysis Using the analysis to produce useful conclusions Conclusion Appendix Index 9 15 17 29 43 47 49 55 67 71 73 81 87 91 93 99 115 117 125 129 Introduction Introduction Socio-economic development programmes are often co-funded and managed within a partnership framework. This is the rule especially for Structural Fund programmes. As a result, the evaluation of these programmes is subject to particular constraints, not only in so far as the sharing of decision-making increases the number of stakeholders, but also because it tends to multiply the number of objectives and to diversify the domains of intervention. Until recently, evaluation developed essentially in the context of interventions financed by a single institution, in a specific domain. Evaluators consequently lack the know-how required in cases of multi- objective, multi-sectoral programmes involving several partners. Evaluation teams tend to view complex programmes as a series of partial analyses, and therefore analyse each of the components successively (axes, measures or actions). This approach leads to difficulties, however, when synthetic conclusions have to be produced and strategic recommendations made on a programme-wide scale. With traditional evaluation methods it is rarely possible to justify the reallocation of budgets because the different components of the programme cannot be compared in terms of common evaluation criteria, acceptable to all the decision-makers. In recent years, several evaluation teams working on the Structural Funds have distinguished themselves by introducing methodological innovations in cases of "multi-objective and multi-sector" evaluations. Pilot evaluations performed as part of the MEANS programme have also helped to develop several operational solutions applicable to such cases. By drawing on experience acquired in recent years, this Volume describes and provides detailed instructions for the use of four new technical solutions now available to professionals. Two of these techniques (concept mapping of impacts and multicriteria analysis) are also featured In Volume 3 of the Collection, devoted to evaluation tools. The intention of that Volume is to assist the making of methodological choices; the tools are therefore presented more synthetically, without the detail provided in this Volume. The first technique presented here is the concept mapping of impacts. This tool can be used to define the main intermediate impacts of a programme in the same way for all the different stakeholders. Concept mapping offers applicable cross-sectional evaluation criteria for an entire multi-objective and multi-sector programme. It also provides a precise description of these impacts, which can be used to construct scoring Technical solutions for evaluation within a partnership framework scales and indicators. To date, no manual exists for the implementation of this technique in an evaluation context. The second technique presented is the construction of scoring scales. Scoring is being increasingly used for project selection, particularly in Northern European countries. The originality of this Volume lies in the fact that scoring is applied to the synthesis of conclusions obtained at the measures level, and is directly related to the main evaluation criteria of the programme. Apart from the evaluation of each measure, the interaction between measures must also be analysed. It can be expected that the impact of a coherent set of measures will be greater than that of the same measures Implemented separately. The third tool presented in this Volume is an aid for the evaluation of these effects of synergy. It is the matrix of cross impacts, a key element in the approaches conceived and tested by several European teams whose work was included in the MEANS programme. Finally, each of the funding authorities would like to see its own criteria integrated into the evaluation report. Multicriteria analysis, the fourth technique presented, is used as a basis for synthesising conclusions which take into account the expectations of each partner. Multicriteria analysis is typically used for ex ante evaluation, with a view to classifying or sorting competing projects or the variants of a single project. By using the tool for the overall evaluation of a socio-economic development programme, this Volume proposes an original use. Among the multiple technical variants of multicriteria analysis, it presents the one which is most appropriate for co- funded programmes. The four tools covered by this Volume are presented with detailed instructions for their use. For each tool, a particular effort was made to present several variants which are either very easy to apply or, on the contrary, involve more technical and rigorous procedures. The reader will note that, in the least complicated variants, the proposals in this Volume are little more than "organised common sense" and that, in many cases, they correspond to current practice. The advantage of presenting these four techniques in the same Volume is that they have numerous common points, e. g. : high level of participation by the partners in the programme; capacity to process quantitative and qualitative information; several points of view taken into account; etc. All four techniques can, moreover, be grouped together within a single Introduction evaluation method. The conclusion to this Volume presents two practical suggestions in this respect. This Volume is intended mainly for evaluation teams (it is the only Volume in the Collection devoted primarily to them), but also for evaluators who wish to enhance their knowledge on the subject. By extension, it is likely to be of interest to those whose work concerns partnership evaluation, outside the domain of the Structural Funds. This Volume can be read in several ways, as suggested in the following box. Box 1 - Possible uses of this Volume If you wish to prepare an overall evaluation of a socio-economic programme by using all or part of the four techniques: >■ Read the Conclusion (see page 115), then refer to the different parts of the Volume as and when necessary. If you wish to study the coherence of a programme, explore its implicit objectives and find transversal evaluation criteria for all the measures: >■ Read Part I on the conceptual mapping of Impacts (see page 15) or simply its instructions for use (see page 29). If you wish to evaluate all the measures of a programme with the same homogeneous and transversal logic: >■ Read Part II on the construction and use of scoring scales (see page 47) or simply its instructions for use (see page 55). If you wish to analyse the synergy within a programme and understand how the impacts of different measures are mutually reinforcing: >· Read Part III on the matrix of cross impacts (see page 71) or simply its instructions for use (see page 81). If you wish to conclude the overall evaluation of a programme, classify or compare the measures of which it is composed in terms of their success, and take into account all the criteria which are important for the partners: >■ Read the part on multicriteria analysis (see page 91) or simply its instructions for use (see page 99). I Concept mapping of impacts The concept mapping of impacts, as presented in this chapter, is an adaptation of the technique known as "concept-mapping", used for the past ten years in evaluation. The tool is used to define the main intermediate impacts of a programme, by seeking the greatest possible degree of transversality between the different measures. A concept map of impacts serves to clarify the main (explicit and implicit) objectives of a programme and to base an evaluation on clear criteria accepted by all the partners. It also provides a precise description of expected impacts which can be used operationally in field surveys, for example to construct scoring scales and indicators. A solution for evaluating different measures by means of common criteria Overall evaluation of a programme Programmes financed by European cohesion policy are all aimed at accelerating socio-economic development in one of its dimensions (e. g. income, employment; quality of life). Thus, the main impact is generally expected at the macro-economic or macro-social level. To obtain this impact, most programmes adopt multi-sectoral strategies consisting of various kinds of measure (e. g. facilities, consultancy, subsidies, training, communication) in diverse fields (e. g. transport, research, tourism). Within the framework of European economic and social cohesion policy, a typology consisting of three main domains of intervention is used: improvement of basic infrastructure (transport, communication, public facilities in general); support for the productive environment, including support for economic actors (to increase, for example, their ability to control quality and to export, their integration into innovation networks, etc. ) and, finally, the development of human resources (essentially through training). This wide range of heterogeneous fields of intervention is frequent in socio­ economic programmes. However, with the Structural Funds this diversity has reached a level rarely equalled elsewhere, which could lead to confusion if the programmes were not highly structured. The mode of structuring opted for has three main levels, as well as one or two intermediate levels: • The overall programme level is associated with the global objective, for example economic development of employment. This level generally comprises several priority axes which define the main strategic dimensions of the global objective. • The measures level (between about a dozen and several dozen measures) is the basic unit of programme management, in which each measure has its own management apparatus. In principle, a measure corresponds to the smallest homogeneous component of the programme and groups together similar projects. However, in some cases certain measures may group together several categories of different projects. An additional subdivision is then introduced, called "action" to denote each homogeneous project category. Technical solutions for evaluation within a partnership framework • The project level (often a few hundred) is the unit of implementation in the field. Each project is the point of articulation between the programme and its addressees. All the projects in a single measure first produce outputs (in the example in Box 2, kilometres of roads improved), then results on direct addressees (time and money savings for users of the improved roads). Finally, impacts on increasingly distant concerned groups materialise in successive stages through multiple, interlinking mechanisms (e. g. easier international travel for regional businesses or better access for tourists to tourist sites). The very existence of the programme assumes that all these mechanisms converge towards the global objective of development. The fact remains that this convergence is achieved in numerous and highly complex ways. Box 2 ­ Structure of a programme (extracts) Programme (consisting of 4 axes) Priority axis: Communications (consisting of 5 measures) Measure: Roads (consisting of several dozen projects) Project : Road between A and B Reducing the lag in development Enhanced competitiveness of enterprises Enhanced competitiveness of enterprises Effective time savings for the transport of goods and people travelling in the region φ o ë culo" o Ii t. , α> a. e fi ç tu <υ o Í2­ *2 crt αζ to Q. O Effective time savings for travellers and goods transported on the regional road network Potential time savings for vehicles using the regional road network Potential time savings for a vehicle travelling between A and B Length of the new or improved road The evaluation of a programme can be performed in very different ways, depending on how impacts are to be analysed. In this Collection, three situations have been presented as follows: Concept mapping of impacts • Overall evaluation which has to examine all the impacts of all the measures. • More in-depth analysis of macro-economic impacts. • More in-depth analysis of a particular impact, for example, of a measure. This Volume was prepared from the viewpoint of the first situation, i. e. the overall evaluation of all the impacts of a multi-sectoral programme. Volumes 1 and 3 of the Collection can be referred to for an introduction to evaluation techniques applicable to the other two situations. Synthesis of specific evaluations Each measure is naturally evaluated in terms of its own criteria, for example on the basis of its results and specific impacts. Box 2 suggests that a measure for improving roads will be evaluated, for instance, in terms of the time actually or potentially saved by the vehicles using the road. As is often the case, it can be assumed that the evaluation team will estimate the total number of hours of transport saved annually by vehicles using the road. The number of hours saved can be related to the cost of the measure, so that comparisons can be made with road improvement programmes in other similar regions. The indicators mentioned above are specific to a particular domain of intervention (roads), just as the approach adopted is that of an evaluation specifically for the road transport sector. Specific evaluations can be performed for all sorts of measures: railways, harbours, airports, but also for sectors of intervention as varied as the training of the long-term unemployed, the rehabilitation of run-down neighbourhoods or the establishment of innovation networks. Volume 2 in the Collection presents numerous examples of interventions, with a description of their results and specific impacts and suggestions for appropriate indicators. A standard procedure in the overall evaluation of a programme consists of performing specific evaluations, measure by measure, and then of synthesising the results at the axis and programme levels, as shown in Box 3. This bottom-up procedure does not involve any major problems when it is used to evaluate a measure by synthesising the evaluations of the projects of which it is composed. Since all the projects of the same measure are similar, it is possible to proceed by way of aggregation, calculation of Technical solutions for evaluation within a partnership framework averages, comparisons between projects, identification of exemplary projects, and so on. Box 3 - Synthesis of specific evaluations Programme Axes Measures Projects Evaluation by synthesis can, in contrast, involve serious methodological shortcomings if it is performed on heterogeneous measures evaluated in a specific manner. Ultimately, each specific evaluation uses its own indicators, analyses distinct impacts, and applies different judgement criteria. In such situations, the overall evaluation of a programme can only be the juxtaposition of partial evaluations of measures. The very notion of synthesis therefore loses its meaning. An evaluation report can be produced in which a long chapter is devoted to each measure, but in which the overall conclusion is nothing more than a few lines with very poor or tenuous contents. In these circumstances it is impossible to make relevant recommendations for the entire programme. For example, it is not possible to justify recommendations for the reallocation of budgets between measures because there is no argument showing that one measure is more effective or successful than another. Evaluation using transversal criteria To produce genuine synthetic conclusions, evaluation criteria have to allow the comparison of the value of measures in relation to one another. A Concept mapping of impacts the speed at which frequently used element of comparison implementation takes place, reflected, for example, in an absorption rate or completion rate indicator. This element of comparison plays an important part in monitoring. It is too often used as the main argument for reallocating budgets, when the evaluation team is unable to base its recommendations upon common criteria of effectiveness, efficiency or relevance. is To enhance evaluations, a comparison of measures in terms of the attainment rate of their specific objectives could be conducted (see the following example). The fact that a measure rapidly achieves its objectives may be an argument in favour of its effectiveness or success, but this argument is necessarily weak. If the objectives were set very carefully or, by contrast, very ambitiously, the conclusions could be distorted. The evaluation of the degree to which objectives have been achieved relates to a question of ex ante evaluation: Can it be said that the objectives of all measures were set with the same level of ambition? This question relates to another fundamental question considered in this Volume: In terms of what common criterion can the level of ambition of the different measures of a programme be compared? Example: Comparison of the rate of achievement of objectives The aim of one measure was to retrain 2,000 employees threatened with unemployment, and that of another was to create 500 new international contacts for regional SMEs. When the mid-term evaluation was performed, the first measure had achieved 53% of its objectives and the second only 25%. Could it be said, on this basis, that the second measure was a relative failure? In reality, the first measure was the continuation of a former intervention of which the objectives had not really changed, while the second measure was innovative and ambitious. Thus, the comparison of the two rates cannot be used to compare the effectiveness of the two measures. In the context of overall programme evaluation, this Volume tries to promote a genuine synthesis based on an analysis of the contribution of the different measures to socio-economic development, and not on the speed at which they are executed. Contribution to the achievement of the global objective of a programme Each programme must contribute to the European Union's global objective of economic and social cohesion. The global objective of a programme is Technical solutions for evaluation within a partnership framework itself expressed in terms of socio-economic development, with a weighting that will depend on a particular aspect of that development, e. g. sectoral diversification, competitiveness, social integration, sustainability of growth, protection of the environment, etc. As the following example shows, the global objective describes the ultimate impact that the programme is supposed to obtain. In a programme designed coherently, each of the measures contributes towards the production of the global impact of the programme. Example: Global objective and intermediate objectives The Belgian province of Hainaut receives European Union assistance as part of a development programme the priority of which is its economic adjustment. The global objective set at the start of the programme, in 1994, is the increase in the per capita GDP, with a view to attaining 79% of the Community average in 1999. The programme comprises four axes of intervention, each of which is sub-divided into measures. The objective of the priority Axis 1 is to boost economic activity. It consists of stimulating the modernisation of the economy and the endogenous development of the region, of increasing its technological R&D capacity, of providing a favourable environment for new business creation, and of transforming the economic growth resulting from these efforts into jobs. The objective of Axis 2 is to enhance the attractiveness of the region. The Hainaut province is still scarred by its industrial past. Making the area attractive, providing quality living conditions and protecting the environment are therefore a specific challenge. Axis 3 is aimed at completing the transport infrastructure within a perspective of integration, in relation to the other programme measures. The objective of Axis 4 is to open up opportunities for everyone. The labour market problems in the Hainaut province justify the implementation of an employment policy aimed at promoting the development of an economy in which competitiveness and solidarity are priorities. Although it may be desirable to judge the measures of a programme in terms of their contribution to the achievement of the global objective, practical obstacles make this impossible. As Box 4 shows, the impact of a given measure on economic and social cohesion involves too many complex mechanisms to be the object of reliable evaluation. Moreover, reasoning. An some of these mechanisms require macro-economic evaluation team may certainly provide useful knowledge on these mechanisms, but it will rarely be able to analyse them, in a differentiated way, measure by measure, with enough robustness to justify strong recommendations such as the reallocation of budgets. In practice, the empirical observation of impacts must stop at an intermediate level which Concept mapping of impacts often extends beyond direct addressees, but never to the entire regional economy. Beyond this intermediate level, extrapolation coefficients or economic models are used, providing results which are useful but do not allow measures to be distinguished in terms of their relative effectiveness. Box 4 - Complexity of impacts and levels of objectives Measure for the amelioration of roads Time saving for users living in the region Specific impact Intermediate impact Growth of business activity and employment in the region Global impact Better economic and social cohesion Identifying the main intermediate impacts for the techniques have been developed Several identification of intermediate causal relations between the specific results of a public intervention and its more global impacts. The most simple and well-known is the logical framework. Originally designed for the ex ante evaluation of projects in developing countries, this tool gives a structured presentation of the evaluated intervention and its expected effects. By applying the logical framework to the example cited in Box 2, a table is obtained which describes expectations concerning outputs (bottom left in Box 5), as well as the main logical steps separating outputs from the global impact (top left). With each logical step, the evaluation team stipulates which means have been provided or planned for achieving the objectives (verifiable indexes of Technical solutions for evaluation within a partnership framework success). It simultaneously defines what must be considered a success or failure (sufficient / insufficient outputs, results and impacts). By requiring that this second column of the matrix be filled with precise qualitative indicators or descriptors, the technique often provides a useful clarification which can then be used to provide a more precise description in the first column. The third column, entitled "conditions and risks", establishes a list of exogenous factors which can facilitate or hinder the advancement from one step to the next. More details can be found on the logical framework technique in Volume 3 of the Collection. Box 5 - Classification of objectives by means of the logical framework Road project between A and Β - See Box 2. Statement on expectations Verifiable success indexes Conditions and risks Global impact Social and economic development Specific impact Result Output Savings for regional businesses Time savings for a typical journey Improved roads The structure of the logical framework, consisting of only two impact levels, does not lend itself to a representation of intermediate impacts. A second tool, better adapted to the complexity of socio-economic programmes, is the objectives tree. As Box 6 shows, this technique is used to determine the logical contribution of each of the objectives at the most specific level, to an objective on a more global level, and so on right up to the global objective of a programme represented by the tree trunk. Compared to the logical framework, the objectives tree provides two interesting new elements: as many branches can be created as there are Concept mapping of impacts measures in the programme, and the intermediate levels can be multiplied to account for the complexity of the mechanisms of impact production. Box 6 - Objectives tree Airports Improvement of roads Accessibility fortravellers Harbours Training / tourism Accessibility for freight Diversification Reduce the peripheral nature of the region Economic and social development In practice, the objectives tree is drawn up in such a way that measures are situated on the left and the global objective on the right The objectives tree also reveals the intermediate objectives common to several measures. In Box 6, the objective "reduce the peripheral nature of the region" can serve as a common reference forjudging the effectiveness of the three measures situated on the top left-hand side of the tree. In fact, the criteria common to several measures appear at each fork in the tree. However, each fork creates a possibility for comparison limited to the branches which are linked to it. To compare measures situated to the right and left of the tree, it would be necessary to go right down to the trunk, but this is not generally possible. Thus, because of its tree-like structure, the technique has little potential for synthetic and transversal evaluation of the multiple measures of a programme. Identifying cross-sectional intermediate impacts To achieve a genuine synthetic evaluation, the image of a tree has to be abandoned and a network of objectives has to be considered. The Technical solutions for evaluation within a partnership framework appropriate tool is the logical diagram of impacts which is used to consider a multiplicity of logical relations between measures and intermediate impacts. Thus, in the example in Box 7, the intermediate impact "viability of enterprises" is indirectly related to all the programme measures, which makes it possible to undertake comparisons between all the measures on the basis of their contribution to the viability of enterprises. Box 7 - Logical diagram of expected impacts Measures Intermediate impacts Global impacts R&D Tourism Industrial wasteland Enterprises Agriculture Training Opening up to the outside\ Enhanced viability of entreprises Strengthened economic fabric Improved employability Improved productive system Structural balance of employment The logical diagram reveals a series of intermediate impacts which will serve as a basis for the overall evaluation of the programme. Among the intermediate impacts, it is necessary to look for those which meet the following requirements: • they must not be too global to allow for empirical analysis based on observations in the field; • they must be connected to as many measures as possible to allow for comparison and aggregation within the programme. The logical diagram of impacts can be constructed in various ways. One approach consists of constructing the objectives tree, based on programme documents, and then of transforming the tree into a network, based on interviews with programme managers and certain operators. The diagram is then presented to the evaluation steering group to be discussed and validated. This approach is not always satisfactory, especially if the evaluation team identifies ambiguities or inconsistencies in the intermediate objectives. In Concept mapping of impacts the past, evaluation teams would sometimes mention such problems to argue that the programme as a whole was therefore not évaluable and that it wished to stop its work at the stage of the specific evaluation of measures. In such situations, where intermediate objectives are not defined clearly enough, the evaluation team should not give up; instead, it should attempt to reconstruct implicit objectives. Box 8 - Reconstruction of implicit intermediate objectives Specific measures and Impacts New tourist products Diversification of entreprises Development of services to entreprises Increased use of natural gas Creation of technological entreprises Development of research -enterprise contact Intermediate transversal impacts Global Impact Socio-economic development ? ? ? ? This approach consists of constructing the logical diagram of impacts based, not on the objectives tree, but on the impacts expected by the managers and operators of the programme. In order to do so, the evaluation team starts by schematically representing the measures, their results, their specific impacts (see the white boxes in the left of Box 8) and the global impact (box on the right). It identifies and describes precisely a series of expected intermediate impacts (shaded intermediate impacts). These should make it possible to link specific and global impacts through the densest possible network of logical relations. This approach was the object of a pilot evaluation in the framework of the MEANS programme in 1996. The work was part of the mid-term evaluation Technical solutions for evaluation within a partnership framework of the socio-economic development programme In the Valencia region (Spain), co-funded by the regional authorities, the Spanish government and the European Commission. The evaluation team used the concept mapping technique, for which instructions for use are given below. Instructions for use of the concept mapping of impacts The concept mapping of impacts is an adaptation of the technique known as concept-mapping, used for the past ten years in evaluation. The technique was first used to help define the strategies or the reorganisation of firms, before being applied to public-policy making, especially in the field of social programmes. The application developed within the framework of the MEANS programme was especially designed for specifying the explicit and implicit intermediate objectives of a socio-economic development programme and for facilitating the choice of related indicators. The concept mapping tool is based on collective reflection and favours the diversity and richness of different points of view. Initial ideas are expressed and intuitively sorted into groups by the participants themselves. A qualitative data processing program synthesises individual representations in the form of a collective graphic image (the concept map). This technique ensures that the evaluation team does not impose its own logical structure which would not necessarily correspond to that of the participants. Constituting one or more w o rk groups The technique may be used within the evaluation steering group, provided that the group has at least ten members. It may also be applied within the framework of a series of work groups. In this case, it could involve up to two hundred people, as in the following example. Example: Collective reflection with a large number of participants The concept mapping technique was used in Northern Ireland to evaluate a programme for reconciliation between the Catholic and Protestant communities. The evaluative question was: "What types of actions contribute best towards enhancing mutual understanding between the communities?". As a result of collective reflection involving 200 people, a large number of proposals were made (e. g. "support for the Irish language as an integral part of our cultural heritage"). In a smaller meeting, the proposals were grouped together by theme. Each participant sorted them into coherent piles and gave them a score, between 1 and 5, depending on their relevance. The groups thus formed by the partners were synthesised in the form of concept maps. The result was used to group the proposals into six categories which were named (e. g. "cultural traditions") and which served to articulate the rest of the discussion. Technical solutions for evaluation within a partnership framework To constitute work groups, the participants can be grouped together by category, for example: a group of programme managers; a group of representatives of local authorities; a group of company directors; etc. Example: Work groups by category of participant The concept mapping technique was used in the United States in a nursing care programme. The aims of the study were to provide managers with information on the kinds of nursing care practised, and to review theories and research results on the subject. Three distinct groups were formed: a group of nurses, a group of patients, and a group of nursing care theoreticians. The concept mapping technique was applied concurrently to each of these groups. One way of creating several work groups is based on the domains of intervention of the programme. For example, one group is constituted for interventions concerning interventions concerning the productive environment, and a third group for interventions concerning human resources. infrastructure, another one for It is preferable for each group to have between 10 and 15 members. The groups must be formed so that their participants have the same level of knowledge about the programme, its logic and its impacts. Finally, the participants must undertake to be present throughout the group's meetings. Two or three meetings over a one- or two-week period should be planned. Depending on the circumstance, a part of the work could be organised in the form of virtual meetings consisting of an exchange of e-mail. Identifying expected impacts in the p r o g r a m me d o c u m e n ts The evaluation team starts its work by consulting all available documents likely to reveal the potential impacts of the programme. It first analyses the programming document, lists the objectives and identifies those associated with a description of expected impacts. The names of measures, sub- programmes or priority axes may help it in this respect. Statements, facts or motives regarding the programme rationale may also inspire the team as it attempts to define what the programme is supposed to produce. Apart from official programming documents, the evaluation team may extend its analyses to include the preparatory work of the programme and, in particular, the plan drawn up by the Member State, the SWOT analyses Concept mapping of impacts and the justifications put forward for the eligibility of the territory or target group, in view of the regulations governing the Structural Funds. The Community Support Framework, when relevant, Is also a useful source of information. Earlier evaluation reports may also be used, as may information from management and monitoring documents, such as technical information sheets describing project selection criteria. Reviews, operating reports, the minutes of monitoring committee meetings, press communiqués and official statements on the programme and its expected impacts are similarly analysed. The analysis of these documents serves to compile a list of extracts describing or mentioning the effects that the programme must or is supposed to produce. If several groups have been formed to cover different domains of intervention, the evaluation team draws up one list per group, by putting together all the extracts likely to concern the different groups. A single extract may concern several groups. Asking for the programme actors' opinions on the expected impacts The second stage in the evaluation team's work consists of a qualitative survey of between 10 and 20 participants in the work groups. The aim is to reveal the implicit objectives of the managers and operators who implement the programme. The basic questions are of the following kind: • With which of the programme interventions are you most familiar? • For those interventions with which you are most familiar, could you say which groups are affected directly or indirectly? • What is the nature of the programme effects on these groups? • Through which mechanisms are these effects produced? • Should the effects produced be considered as positive and desirable? To obtain the actors' opinions on programme effects, the evaluation team holds individual interviews either by telephone or face-to-face. The focus group technique may also be used. In this case, it would be in the form of an initial work group meeting. The interview should not, however, be structured nor make use of a questionnaire. The idea is to enable as many ideas as possible to emerge, by fully respecting the expressions of the interviewees rather than asking them to fit into pre-established frames. Technical solutions for evaluation within a partnership framework The impacts concerned are those which the programme actors expect to observe in the short or medium term. These impacts may or may not be forecast in the programme documents. They are generally positive, but the evaluation team must not hesitate to note those which are negative. For example, the following two impacts could be noted: "assisted firms increase their productivity" and "jobs are lost in assisted firms due to increased productivity". Experience shows that participants are reluctant to mention negative impacts; evaluation teams must therefore emphasise the advantages of examining It must also ensure that these as well. interviewees do not mention only those impacts which are "easy" to obtain. The analysis of interviews is concluded with a list of sentences, taken from the interviewees' answers, which describe the effects of the programme as expected by its actors. If several groups have been formed, a list of sentences likely to interest each group is compiled. Establishing an h o m o g e n e o us list of statements on impacts All the extracts of texts and statements gathered during the two preceding stages must be rewritten in an homogeneous form by the evaluation team. Thus, the team has to reformulate all the elements collected, in short sentences called "impact statements". An impact statement is always in one of the following forms: a particular group changes its behaviour in a particular way; a particular economic variable is modified in a particular way; a particular element of the environment evolves in a particular way. For example, if a programme objective is to "reduce the number of young people leaving rural areas", the following statement will be written down: "more young people settle in rural towns". The statement implies that the change or evolution is a consequence of the programme, but this causal relation does not have to be referred to each time. Thus, if an impact is stated in the form "more young people settle in rural towns", different measures may have different levels of success from that point of view, e. g. : "a significant number of young people settled and stayed" or "a few young people settled exceptionally" or "no young people settled". These levels of success will be discussed later in this book, where they are referred to as "impact descriptors" (see page 61). A statement always applies to one impact only. For example, if a programme objective is to "increase the income of farmers through the diversification of activities", the following two impact statements will be made: (1) "farmers' activities are diversified" and (2) "farmers' incomes are rising". Concept mapping of impacts The evaluation team must bear in mind the aim of the exercise which is to record intermediate impacts. At this stage, it has to eliminate impact statements which are too specific and obviously cannot concern more than one measure. Similarly, impacts which are too global should be deleted when it appears that they cannot be empirically connected to the different measures of the programme. At the end of this stage, the evaluation team sorts the statements to eliminate those which are obviously duplications. It thus obtains a list of statements (see the example below) which must not exceed one hundred items. If the list is too long, the impacts which cover a particular measure too specifically are put aside. If several work groups have been formed, a list is compiled for each group. Example: Impacts stated by the managers of a measure for direct aid to SMEs SMEs have better management systems The qualifications of craft enterprise employees are increasing Supply in the tourism sector is becoming diversified A larger number of firms comply with environmental standards Firms are replacing oil-derived energy sources by natural gas Research centres are developing R&D projects adapted to the needs of regional industry Firms are improving their production processes Firms are increasing their export sales The financial situation of SMEs is improving Jobs are being created in priority areas Firms are reducing their production of waste Validating the statements The following step is carried out in a steering group meeting or in several meetings if there are several work groups. In the latter case, each group examines the list of impact statements prepared especially for it. The chairperson of the meeting starts by: • presenting the principles of the technique • describing the reasoning behind the grouping together of the participants • explaining what is understood by an impact statement and how these statements were determined • presenting the list of statements. Technical solutions for evaluation within a partnership framework The list is then discussed in depth, particularly around the following points: • Is there a contradiction or not between the statements drawn from programming, management and monitoring documents, and those taken from interviews? • Can some statements be written again so that they become applicable to more measures? • Are the statements understood correctly and in the same way by all the participants? If not, how can they be specified? • Is it possible to combine statements which are similar and add nothing in terms of impacts? • Should important statements be added? Weighting the statements After this first meeting, the evaluation team finalises the list of statements. The statements are distributed randomly so that sequences of similar statements are not created. The evaluation team prints a list of numbered statements for each participant. The work group holds a second meeting shortly after the first one. If there are several work groups, each of them holds two successive meetings. The main aim of these meetings is to grade the expected impacts in terms of their importance for the overall success of the programme. The meetings must be run according to the following instructions: • Distribute the list of statements to each participant. • Ask the participants to give a score to each statement, depending on the the importance of the corresponding programme (1 = relatively unimportant. , 4 = essential impact). This scoring is also valid for unfavourable impacts (4 = highly unfavourable impact,. , 1 = minor unfavourable impact). The scores must be written on the list of impact statements which must then be collected. for the success of impact, impact Grouping together conceptually similar impacts The collective work is continued with the grouping together of impact statements into several main clusters. This is where the concept mapping technique, in the strict sense of the term, is applied. The evaluation team Concept mapping of impacts prints, for each participant, a set of cards with a statement and its number on the same side of each card. The meeting is continued on the basis of the following instructions: • Distribute the complete series of impact statements to each participant. Box 9 - Distribution of impact statements • Ask the participants to sort the cards into several piles, according to their similarities. Each member of the work group will thus put together impacts which, in his or her view, belong to the same cluster of impacts. The instruction is: "Put those impacts which you think are similar onto the same pile". Participants are free to create piles as they wish. The only constraint is that all the cards must not end up on the same pile and that there must not be as many piles as there are cards. • The participants are asked to explain, on the back of the first card in each pile, the logic behind the constitution of the pile. • The piles of all the participants are collected, stapled together and put into an envelope on which the name of the participant is written. The piles produced by each member of the group are analysed statistically. Clusters of impacts are formed, reflecting as closely as possible the groups constituted by the participants. This technique consists of counting the number of times that two impacts are associated with each other, and of Technical solutions for evaluation within a partnership framework considering this number as an index of their conceptual proximity. The conceptual distance is, in contrast, equal to the number of times these impacts were not associated. A square matrix is then constructed, with all the statements in lines and columns. Each cell shows the conceptual distance between is processed by multidimensional statistical analysis which produces two results: impacts. This matrix two • A concept map showing impacts and reflecting their conceptual proximity and distance as clearly as possible (see Box 10). • A hierarchical classification showing those statements which are closest together in conceptual terms (see Box 11). The entire process can be performed by statistical analysis software on a microcomputer. Several such software packages exist on the market. To facilitate the process, a programme developed especially for concept mapping can also be used (see bibliographic annex, W. M. K. Trochim, 1989). Box 10 - Concept map of impacts <1>SMEs management systems have improved <12> The financial situation of SMEs is improving <25> Enterprises are reducing their production of waste <7> Enterprises are replacing oil-derived energy sources by natural gas Based on this map, the statistical analysis can be continued by creating several clusters of impacts. Hierarchical classification will then be used, by cutting the tree at a given point. In the example in Box 11, two possibilities Concept mapping of impacts are shown: in the first, five impact clusters are created and in the second, six impact clusters (A,B,C,D,E,F). In this example, the division into five clusters is preferable because, as the example shows, if clusters C and D are combined, this group should also be extended to include cluster E. Once the clusters have been defined, it is possible to calculate a bridging index for each statement. The bridging index varies between 0 and 1 : a value close to one means that the statement played an important part in the construction of the impact cluster to which it belongs; a value close to 0 means, on the contrary, that the statement is not closely related to its cluster. Box 11 - Hierarchical classification of impacts Statements <FQ SMEs Is improving 12 The financial situation of 37 1 SMEs management systems 15 have improved 28 45 18 6 8 40 39 4 25 Enterprises are reducing 7 Í 0 19 23 Enterprises are replacing their production of waste oil-derived energy sources by natural gas In the pilot evaluation performed in Valencia, the "infrastructure" group selected three impact clusters, the "direct aid to enterprises" group selected eight, and the "support for the productive environment" group selected seven. If there are few participants and fewer than thirty impact statements, it is possible to proceed without a statistical analysis programme, for example by using a group discussion technique such as METAPLAN®, described in Volume 3 of the Collection. This tool serves to produce a visual grouping of impact statements, in a context of collective discussion, and a weighting of these statements. This simplified approach was applied in the framework of Technical solutions for evaluation within a partnership framework mid-term evaluation of a socio-economic development programme, as the following example shows. Example: Use of the METAPLAN® technique for creating impact clusters In the mid-term evaluation of an industrial redeployment programme in France, the first steering group meeting was aimed primarily at producing a list of expected impacts. The outside consultants responsible for drawing up the agenda and chairing the meeting used the METAPLAN® technique. The participants (13 people) were managers of measures and operators. The session lasted for about two hours. It revealed the main expected impacts, which were then grouped into five clusters labelled as follows: A - Reskllling of the workforce Β - Adaptation of training C - Attractiveness of the territory D - Redeployment/Diversification E - Creation of jobs through local initiatives. Validating the impact clusters obtained In the concept mapping process, each work group meets again, for a final time, to discuss and validate the impact clusters which appeared on the map. The evaluation team would, in the meantime, have drawn up and printed the map, as well as a list of statements sorted into clusters. For each cluster, the participants proceed in the following way: • Go through the impacts in a cluster and verify their meaning. • Reclassify, where relevant, the impact statements which have a low bridging index. • Identify, within the impact cluster, those statements which were given the greatest weight by the participants and those which have the highest bridging index. • Discuss the logic behind the creation of clusters, if necessary by referring to written notes made by the participants when they sorted the cards into piles. • Find a word or expression which synthesises all the impact statements in the cluster, particularly those which were given the most weight and which have the highest bridging index. Concept mapping of impacts After a discussion about each impact cluster, the work group chooses a vaguely abstract name, as shown in Box 12. Some participants may be concerned about the fact that, after intense collective work, the final result is so imprecise - especially if an effort has been made to clarify objectives considered vague. In reality, this type of criticism is not valid. Whereas the name of a cluster of impacts may be abstract, the statements comprising it are very clear and sufficiently precise to form the basis of the evaluation. Box 12 - Naming impact clusters Better management Human capital <1> SMEs management systems have improved <12> The financial situation of SMEs is improving <25> Enterprises reducing their production of waste <7> Enterprises are replacing oil-derived energy sources by natural gas Innovation Environmental viability Work with t wo levels of transversality An interesting variant was chosen in the case of the Valencia evaluation. It consisted of applying the technique in two phases. First, three work groups were formed, with about ten local officials (operators and managers of measures). A "transversal" group was then formed, with representatives from the Commission and from national and regional administrations, i. e. : the vertical partners (see Box 13). Technical solutions for evaluation within a partnership framework Box 13 - Work groups created in Valencia Sectoral groups Group A "Direct aid to enterprise" Group Β "Support for the productive environment" Group C "Infrastructure" Transversal groups "Central authorities" group 12 managers from 4 administrative institutions (tourism, trade & industry, finance, economy & enterprise) 11 managers from 6 institutions (trade, occupational training, tourism, economy, etc. ) 11 managers from 5 institutions (public works, transport, industry, environmental quality, etc. ) Representatives of the Valencia Independent Community, representatives of the Spanish State and desk officers from the European Commission For example, the work group "Direct aid to enterprise" consisted of managers and operators concerned by the following measures: • Investment support plan; • Modernisation of craft businesses and of their production systems; • Programme for providing assistance to priority areas; • Venture capital for SMEs; • Support for energy savings, diversification and/or the use of alternative energy sources by firms; • Plan for promoting services to businesses; • Technological plan; • Support for the improvement of supply in the tourist industry; • Support for businesses to comply with environmental standards. impact Each of the three sectoral groups compiled statements. These statements were systematically divided into two levels. The first level was closest to the specific impacts of the measures and the second level was closer to the global impact of the programme. its own list of The lower level (closer to the specific impacts of the measures) consisted of the following types of statement: "the number of entrepreneurial Concept mapping of impacts initiatives has increased", "new SMEs have improved management industries have systems", "the qualifications of employees of craft improved". The higher level (closer to the global impact of the programme) consisted of the following kinds of statement: "the number of jobs in regional enterprises has increased", "the areas which have benefited from assistance have not experienced population loss". Each of the three sectoral groups worked only on the impact statements on the lower level. All the impact statements on the higher level were grouped into a single list and processed by the fourth group. The impact clusters thus formed were therefore arranged on two levels, as shown in Box 14. Box 14 - Two levels of impact clusters Measure A Measure Β Measure C | Measure J I Measure Κ Measure L | Measure R I Measure S Measure Τ Impact cluster Using the concept mapping of impacts ■ Constructing programme evaluation criteria The collective reflection on which the concept mapping of impacts is based encourages the diversity and richness of different points of view. This approach, which takes into account the different partners' opinions, serves to produce evaluation criteria readily accepted by the different programme partners. The use of computers seems to be a major asset for ensuring that the common evaluation reference system is acceptable to all. Thus, the technique precludes the evaluation team from imposing its own logical structure which might otherwise be foreign to the participants. Box 15 - Evaluation criteria common to the different measures Impact cluster Environmental viability Evaluation criterion Effectiveness of measures in terms of environmental viability Evaluative question What was the contribution of the different measures to the environmental viability of the economic units concerned by the programme? Example of the evaluation method Survey of economic units targeted by the different measures: businesses, tourist agencies, farms, artisans, etc. Survey by questionnaire, interviews or case study. The questions asked refer to impact statements belonging to the environmental viability cluster, i. e. : "Has there been a reduction in the production of waste?", "Has there been a transfer in energy consumption towards natural gas?", etc. The main advantage of concept mapping is that it allows for the construction of effectiveness criteria which can be applied transversally to several measures. For example, by reverting to the step in which the clusters are named (see Box 12), the evaluation team could use the cluster "environmental viability" as indicated in Box 15. Through this example it can be seen that a very simple procedure leads from an impact cluster to an evaluation criterion, then to an evaluative question, and finally to a survey Technical solutions for evaluation within a partnership framework method which will provide answers to the questions in an empirical and pragmatic way. Furthermore, it is possible to use the statements comprising each cluster, to construct scoring scales. The evaluation will check whether the impact clusters are really transversal, that is, whether they apply to a large number of measures. The approach can be considered a success if each impact cluster is potentially connected to a large number of measures. The more connections there are, the more the evaluation will be able to make comparisons and aggregations between the measures, and the more it will be possible to perform a synthetic evaluation of the programme. Note that the concept mapping technique favours multiple connections because of the way in which the impact clusters are constructed. It is recalled that these clusters consist of all the intermediate impacts expected by the managers of the different measures, and that these impacts have systematically been sorted into groups. Construction of generic indicators A generic indicator is an indicator which serves to make a comparable quantification of several different kinds of measures within the same programme (see Volume 2 of the Collection). Such an indicator allows for the aggregation of data within the programme, in the form of a sum or average. Examples of generic indicators include absorption rate and completion rate. Certain generic indicators apply only to a part of the programme, e. g. : number of innovations (for all measures promoting technological development) or proportion of SMEs among the addressees (for all the measures benefiting business). The more heterogeneous the measures of a programme, the more difficult it is to define really generic indicators. If a concept map is created, the intermediate impact clusters constitute an excellent support for creating these generic indicators. A part of the last work group meeting can be devoted to this exercise. The discussion is then on each impact cluster, in turn, and follows these guidelines: • Run through the impact statements comprising the cluster. • Ask the participants to find indicators associated with these statements. • Redefine the indicators proposed, so that they are as generic as possible. For example, the indicator "jobs created in assisted firms" could be replaced by a more generic indicator such as "jobs created in assisted economic units". Concept mapping of impacts • Give priority to indicators associated with impacts which are the most heavily weighted. • Select those indicators that will apply for as many measures as possible. • 2 to 5 indicators could be proposed for each impact cluster. • Considering the importance of the question and the time required for an in-depth discussion on indicators, it is advisable to address the subject after a long break, or even to plan a half-day's work for such discussion. Example: Indicators associated with impact clusters Several Intermediate clusters corresponding to the generic criteria defined by the main group in the MEANS pilot evaluation are shown below. This evaluation concerned 25 measures of the socio-economic development programme in the Valencia region for the period 1994-99. Impact cluster Generic indicator Information source Programme Context Better national development Use of urban public transport Number of users per km of public transport services Transport operators assisted Transport operators assisted Better productive environment Improved qualifications of the working population Operators of training and education measures Density of banking institutions Statistics from the Transport Ministry Statistics from the Transport Ministry National Statistics Institute National Statistics Institute An appropriate technique for partnerships The concept mapping of impacts serves to initiate a group dynamic which is obviously an advantage in evaluation. The evaluation will be accepted far more readily by the stakeholders in the programme if they have collectively participated in defining a common evaluation reference system. Moreover, interaction between managers can only enhance their knowledge of actions and thus favour the effects of synergy. The participation of managers in the Technical solutions for evaluation within a partnership framework process favours a better awareness of the effects which the interventions are supposed to produce, beyond the outputs stage. The possibility of implementing the method and obtaining results in a relatively short lapse of time, such as one week for a series of meetings, makes the concept mapping of impacts compatible with the evaluation schedule. It does, however, have its drawbacks which can limit the utility of the technique under certain circumstances. One of the drawbacks of this method is that it mobilises a large number of managers or officials over a period of several days. Even if some meetings are replaced by an exchange of mail, this type of project demands that numerous officials devote time to it. Apart from practical problems related to agenda-setting, such mobilisation of public agents involves expense for the community. It seems justified only in so far as significant stakes motivate the evaluation. Furthermore the creation of groups is clearly a sensitive issue. In case of conflict or if all the partners do not agree to participate in the process, or if it is not possible to achieve a balanced representation, the implementation of the technique in the constitution of groups is also great when representatives of target publics are involved in the process. In this case, identification and balanced representation seem relatively complex matters. is compromised. The difficulty encountered Finally, the interplay between actors during group reflection has to be taken into account. The concept mapping technique makes it possible to ensure that a leader within the group does not influence the constitution of piles nor the scores given to statements. This influence is, however, possible during the collective production of statements and during discussions on the impact map. It is then up to the person chairing the meeting to ensure that the results are indeed those of the group as a whole. It is therefore preferable to have a chairperson who is thoroughly familiar with group leadership techniques, so that the interplay between the actors does not influence the results of the exercise. The fact that the implementation of the technique requires adequate knowledge on the evaluated domain of intervention and on relevant statistics, may cause the project to be entrusted to a multidisciplinary team. Constructing and using scoring scales Scoring systems are being increasingly used for project selection, especially in the United Kingdom. Scoring has the advantage of taking into account, with a great degree of flexibility, all the quantitative and qualitative information needed to judge a given criterion. Its field of application is therefore wider than that of indicators which are difficult to apply to certain criteria, due to their quantitative nature. The originality of the approach proposed in this chapter lies in the fact that scoring is used for the evaluation of measures and not for the selection of projects. This chapter is connected to the preceding one in so far as a scoring scale can be constructed for each impact cluster. It is also related to Chapter IV in so far as multicriteria analysis is usually based on evaluations of impacts carried out by means of scoring scales. A solution for evaluating different measures in a homogeneous manner Qualitative conclusions on the impacts of measures Overall evaluation of the measures of a programme aims at examining all the impacts of all the measures. In the case of a socio-economic programme, for example, with about forty measures to evaluate in terms of five impact clusters (competitiveness, employment, equal opportunities, environment, and quality of life), the evaluation team will have to express its opinions on at least 200 impacts. This figure shows that all these impacts cannot be analysed in depth. At most, the overall evaluation of a programme can include in-depth analyses of a few particularly sensitive impacts. In-depth analysis of a given impact requires observation in the field. The basic techniques applied to this type of analysis are, for example, comparison groups and regression analysis. Their use requires time and substantial resources, and the scope of observation must be limited to a single category of output and a single concerned group. In-depth analysis generally focuses on a micro-economic or micro-social impact mechanism. An impact can be analysed only at a late enough stage for it to have materialised (see Volume 1 of the Collection). All these constraints mean that a genuine impact analysis cannot be extended to a multi-sectoral programme. In such cases, pragmatic approaches, such as extrapolation coefficients for estimating impacts, must be given precedence. An extrapolation coefficient will make it possible to estimate the number of temporary jobs created per million euro of investments in infrastructure, or the annual value-added generated per million euro of investments in SMEs. A coefficient can be deduced from an in-depth survey carried out within the framework of the same evaluation or an earlier evaluation. It may also be borrowed from another evaluation or research on similar interventions, provided that the conclusions are considered applicable to other cases. A coefficient may also be established for the needs of an evaluation by one or more experts. In situations of mid-term or ex post evaluation, monitoring data on outputs can be used as a basis for extrapolation. Thus, for example, the number of beds in a recently constructed tourist facility can be taken as a basis for estimating the number of jobs created, by means of a ratio of number of Technical solutions for evaluation within a partnership framework jobs per bed. In certain cases, the addressees are asked to make a statement on the impacts at the time they enter into contact with the operators. They must, for example, indicate the number of jobs created at the time the last payment was made, or the number of jobs to be created is when recommended to use a correction coefficient to take dead-weight into account. This coefficient may vary from 0. 8, at best, to less than 0. 3 for certain interventions where there is substantial dead-weight. their application for assistance In this case, is filed. it Impacts can be estimated by extrapolation in certain conditions only: (1) it is necessary to have an extrapolation base in the form of indicators for monitoring outputs and results; (2) the impact to be estimated must be defined precisely and must be associated with an indicator and (3) existing literature or previous evaluations must provide ratios such as "impact / output" which can be generalised. These conditions are relatively severe and are rarely met for all impacts and outputs. The most frequent situation is therefore that in which the evaluation team cannot make an estimation, in figures, of the totality of impacts of all the measures. If it is essential for the evaluation to give an overview of programme effects, it will be able to provide only qualitative conclusions for at least part of the impacts. Even when an extrapolation approach serves to produce quantitative conclusions, estimations of impacts need to be refined with more qualitative interpretations. In particular, the evaluation team must explain and justify the extrapolation coefficients it has chosen, by considering all the context elements which condition the process of programme implementation, characterise the concerned groups and influence the performance obtained. Evaluation professionals generally recognise that indicators are only one of the bases among many enabling them to formulate their conclusions. In particular, indicators must be interpreted in light of all the accessible and relevant qualitative information. To conclude this rapid overview of impact evaluation, it should be recognised that it often produces qualitative conclusions. Synoptic presentation of qualitative conclusions By writing up the qualitative conclusions on the impacts of a given measure, the evaluation team naturally places itself in the perspective and the context of that measure. The conclusions therefore tend to be written within a logic and in a vocabulary peculiar to a specific measure, as the following example shows. This Constructing and using scoring scales creates problems in writing the chapter summarising the evaluation of the programme as a whole. As a result, evaluation reports tend to have a section on each measure, containing sound and well argued conclusions, but a final synthesis which is short, poor and without solid foundation. Example: Qualitative heterogeneous conclusions for different measures Conclusion on the impact of Measure A (training for the long-term unemployed): "On the basis of an examination of the post-training trajectories of a few trainees considered typical, it is estimated that employability has been improved in the majority of cases, with better results in tourism-related occupations and less satisfactory results in environment-related occupations (the latter account for a small proportion only of the training courses). On the basis of evaluations of comparable programmes, the placement rate can be estimated at between 30 and 50%. A positive impact has been identified in terms of the reorientation of women towards more secure jobs traditionally occupied by men. Yet a significant risk of substitution and skimming-off effects exists". Conclusion on the impact of Measure Β (assistance for new business creation): "On the basis of the evaluation of comparable programmes, the survival rate can be estimated between 20 and 30%. It seems that the survival rate is higher for firms created in the environment sector, to which 35% of the assisted firms belong. Moreover, it is considered that the majority of entrepreneurs who failed in their undertaking, substantially improved their employability. It is to be noted that 45% of the new businesses assisted were created by women, as opposed to a national average of 30%. The difference can be imputed to the programme". To facilitate the preparation of a final synthesis, it is useful to summarise all the qualitative conclusions in the form of a synoptic table organised in relation to the different impacts (see Box 16). Although the synoptic table is helpful for the evaluation team, a lot of work is still required to produce genuine synthetic conclusions. In theory, at least three approaches need to be undertaken: • First, synthetic conclusions have to be produced for each impact cluster. For example, the evaluation team can identify the measures which are most effective in terms of employment impacts, as well as those which have an insignificant or detrimental effect in this respect, and so on for the other impacts. This first approach is developed in the following pages devoted to scoring scales. • Secondly, the evaluation team must analyse synergy between the measures. For example, it could be shown that the two measures in Box 16 Technical solutions for evaluation within a partnership framework have impacts which mutually reinforce one another in the environmental domain. This second phase is developed in Chapter III, in connection with the matrix of cross impacts. Finally, a general synthesis covering the main impacts and synergy between the measures must be made. At the end of this third phase, the evaluation team should be able to classify the measures in terms of their contribution to the overall effectiveness of the programme. This point will be developed in Chapter IV, in relation to the multicriteria analysis technique. Box 16 - Qualitative synoptic table of measures and their impacts Measures Impact on Impact on Impact on employment equal opportunities the environment Training Employability was The training had a A small proportion of intended for improved in most cases positive impact in the training courses the long-term unemployed terms of reorientation resulted in On the basis of the evaluation of of women towards secure jobs qualifications in occupations in the comparable traditionally occupied environment sector. by men. programmes, the placement rate was estimated at between 30 and 50%. There is a considerable risk of substitution and skimming off effects appearing. However, very few trainees found jobs corresponding to these qualifications. Aid for new business creation On the basis of the evaluation of 45% of the new businesses assisted 35% of the businesses were comparable were created by created in the programmes, the survival rate can be women, as opposed to a national average environment sector. estimated at between of 30%. The difference It seems that their 20 and 30%. can be imputed to the programme. It is considered that the majority of business creators who fail, nevertheless significantly improve their employability. survival rate is higher than the national average. Etc Constructing and using scoring scales The present chapter is devoted to the first of these three approaches, i. e. to the writing up of synthetic conclusions on the different impact clusters. The idea is to formulate comparative judgements between the different measures, transversally, throughout the entire programme. The proposed technique is the scoring scale. It is a technique which produces conclusions that naturally lend themselves to comparison, aggregation and the calculation of averages. Although scoring scales are not comparable between different impact clusters, the technique does help directly in the writing of the final synthesis of the evaluation. Instructions for using scoring scales Scoring scales are used in many domains, often spontaneously in a relatively informal way. As a result, the technique is often criticised for its subjectivity. In some cases, however, rating is rigorously established and plays a part recognised by all. Well-known examples are international rating agencies which provide assessments on creditworthiness, and the economic role of ratings in gastronomic guides. In the domain of education, health or social integration policy, hundreds of scoring scales developed for research purposes are used in evaluation. Scoring is still used infrequently in the evaluation of socio-economic development policies. A noteworthy exception is, however, development aid where the technique has been used by numerous funding bodies since the mid-1980s (see the example on page 59). The following pages propose instructions for using this technique in the framework of European Union socio-economic programmes. Three variants of these instructions are proposed, depending on the time the scoring scale is established: Variant 1: the evaluation team has already formulated qualitative conclusions for each measure; the scoring scale helps to formulate a synthetic conclusion at the programme level. Variant 2: the evaluation team has already collected qualitative information on the effects of the measures, but has not yet written up its conclusions; the scoring scale helps to formulate conclusions, measure by measure, in an homogeneous way that is easy to synthesise. Variant 3: the evaluation team starts working; the use of a scoring scale in formulate homogeneous conclusions that are easy to synthétise. the data collection process makes it possible to Scales for synthesising qualitative conclusions In the first variant, the evaluation team has already formulated its conclusions. These are presented in the form of the synoptic table in Box 16, in which each column concerns an impact cluster and contains the conclusions drawn on all related measures. Box 17 is extracted from the synoptic table with regard to impacts on equal opportunities. Are the two impacts described in Box 17 comparable? Could one say, when reading these conclusions, that Measure 1 is the most effective in Technical solutions for evaluation within a partnership framework terms of its impact on equal opportunities? At this stage of its work, the evaluation team notes that the impacts are not directly comparable and that it cannot go much further, on its own, towards a synthetic conclusion. A possible way of formulating comparative judgements would be to get the steering group or an expert panel to express an opinion. The approach is as follows: • Organise a half-day seminar with an expert panel or the steering group. The steering group should preferably be extended, to form an evaluation committee, as proposed in Volume 1 of the Collection. • Present the evaluation conclusions to the participants, measure by measure. Verify the credibility of the conclusions and validate them. • Present the synoptic table of conclusions (see Box 16). • Examine the first column of the synoptic table and ask each participant to intuitively situate the impacts of each measure on a scale, as in Box 18. Compare the participants' classifications, discuss them and make them converge as much as possible. • Continue the classification with the other columns in the synoptic table. • After the seminar, convert the classifications into scores ranging from -10 (maximum negative impact) to +10 (maximum positive impact), through 0 (neutral impact). Convert the synoptic table in Box 16 into a table of ratings (impact scoring matrix). The construction of scoring scales allows for comparisons to be made within the same column (e. g. one particular measure has a better rating than another). On the other hand, scores in different columns cannot be compared (e. g. a score of 5 for employment is not comparable to the same score for an environmental impact). Box 17 - Synoptic presentation of conclusions relative to an impact 1 2 Measure Description of the equal opportunities impact Training intended for the long-term unemployed The training had a positive impact in terms of orientating women towards secure jobs traditionally occupied by men Aid for new business creation 45% of the new businesses were created by women, as compared to the national average which is 30%. The difference can be imputed to the programme. Other measures. Etc. Constructing and using scoring scales Box 18 - Classification of impacts obtained on an intuitive scale Measures Impact in terms of equal opportunities Support for the creation of new business 45% of the new businesses were created by women, compared to the national average of 30%. The difference can be imputed to the programme. Training for the long-term unemployed The training had a positive impact in terms of the reorientation of women towards more secure jobs traditionally occupied by men. Etc. Maximum positive impact Neutral impact Maximum negative impact In a pilot evaluation in the MEANS programme, a technically more rigorous procedure was proposed and applied. In this variant, the participants were not expected to intuitively place the impacts on a scoring scale as in Box 18 above. They were asked rather to express their preferences between all the impacts, considered in pairs. Their opinions were gathered through a series of questions such as: "which of these two situations do you prefer: • a situation in which a positive impact is observed in terms of reorientation of women towards secure jobs traditionally occupied by men or. • a situation in which it is observed that a greater proportion of women have created their own business than that observed throughout the rest of the country (45% of the assisted new businesses were created by women, compared to 30%). ? - As far as the situation you prefer is concerned, is your preference very strong, strong or weak?". The questioning is continued for all the impacts in the first column, compared in pairs, and then for the other columns. All the answers are processed by a computer program which attributes a score between -10 Technical solutions for evaluation within a partnership framework and +10 to each impact, so as to reflect the participants' preferences as closely as possible. This software, called "Macbeth", was developed by a team at Lisbon University (see the annexed bibliographic references: Bana e Costa, 1994). This software is also mentioned further on (see page 107), in connection with one of the variants of multicriteria analysis. Whatever the procedure opted for, the final result consists of a series of scoring scales (one per impact cluster). Each measure is given a score on each scale. All the scores can be combined in an impact scoring matrix (see Box 19) which complements the synoptic table of qualitative conclusions (see Box 16) and has the same structure. The combination of qualitative conclusions and scores makes transversal conclusions on a programme-wide scale, for each impact cluster. it possible to formulate Box 19 - Impact rating matrix Measures Employment impact Impact on equal opportunities Environmental impact Training intended for the long-term unemployed Aid for new business creation Etc +4 +8 +3 +7 0 +5 Scales for the formulation of structured conclusions As far as possible, it is preferable to construct scoring scales a little earlier in the evaluation process, i. e. before the formulation of conclusions for each measure. The evaluation team can then use the scoring scales to write conclusions which are more structured and more easily comparable between measures. When used for evaluation, scoring scales are generally designed in this way. In certain cases (see the example of the World Bank, below), standard scoring scales are made compulsory in all evaluations concerned by a particular policy. However, this approach is difficult to apply to socio­ economic programmes financed by the European Union. Because the different programmes are co-financed in partnership with different national and/or regional authorities, it hardly seems legitimate to standardise the ways in which their impacts are judged. Constructing and using scoring scales Example: Scoring scales used by the World Bank The World Bank performs about 200 project evaluations annually. All these evaluations have qualitative conclusions and conclusions in the form of ratings, structured according to a standard system. The scoring scales used include five dimensions of project performance: economic impact, sustainability, impact on financial performance of the bank, and financial institutional development, performance of the borrower. All the projects are evaluated by their managers (self- evaluation) at their conclusion, by means of 4-tiered ratings for each dimension ("highly satisfactory", "satisfactory", "unsatisfactory" and "highly unsatisfactory"). In many cases, and particularly in cases of doubt about the self-evaluation, a second rating is performed by the Bank's Operations Evaluation Department (OED). This second rating is always discussed with the project managers. If the judgements do not converge (and this is very rarely the case) an arbitration commission is asked to decide on final ratings. The World Bank, unlike other organisations, does not calculate a weighted global score to synthesise all the dimensions of performance. In other cases, the evaluation is performed by an expert panel which constructs its own scoring scales (see the British example below). The members of the panel receive all the information collected on the different measures, and formulate qualitative conclusions and conclusions in the form of scores. In order to do so, they agree on one or more scoring scales established during the first panel meeting. Each panel constructs its own scoring system. Expert panels are seldom used in the evaluation of European Union socio-economic programmes. Example: Rating by expert panels Rating by expert panels is an approach that has been used for the past decade in the U. K. for evaluating the performance of university research centres. In this country, the work of all research centres is monitored quantitatively and evaluated periodically by expert panels. In their evaluation, the experts take into account indicators (e. g. number of publications) as well as purely qualitative elements (e. g. quality of the four best publications of each researcher). They thus rate research teams (on a scale of 1 to 5) which serves as a direct reference for the allocation of public funds. In the domain of European cohesion policy, an interesting practice exists in the U. K. , in the selection of projects for funding. The following example presents the technique used in the Highlands and Islands region. This example is interesting in so far as the scoring scales are constructed in partnership and are used throughout the life of the programme. Many British programmes use similar scoring scales. Technical solutions for evaluation within a partnership framework Example : Scoring scales for project selection Within the framework of the Highlands and Islands regional development programme, two series of calls for projects take place each year. All the applications for funding are first examined in terms of their eligibility. The projects considered eligible are rated on eight scales common to the entire programme (relationship with objectives; probable effectiveness; respect for the environment; leverage effect; innovation; relations with other projects; intervention of the private sector; and support for local partners). Other scales are also used but are more specific to certain measures. Each project is rated on each of these scales. A weighting system completes the procedure, so that a global score can be given to each project. After comparing the applications for funds with available resources, an admission standard is established and a list of projects to finance is recommended. The final decision is taken by the programme managers. Since scoring scales are used frequently and satisfactorily for project selection, several recommendations have been made to extend the use of this technique to programme evaluation. Proposals generally suggest the creation of scales concerning the main impacts of the programme, and the use of these scales to rate measures rather than projects. This kind of approach was applied to the transversal evaluation of over one hundred socio-economic development measures in the Republic of Ireland. It was used to compare measures and to formulate recommendations on the reallocation of budgets, of which a large number were accepted. Example: Scoring scales for evaluating the relevance of measures The mid-term evaluation of a set of socio-economic programmes in the Republic of Ireland concerned about one hundred measures. For each measure, the evaluation team tried to ascertain whether its financing by public funds could be justified. Four categories of measures were established, corresponding to the four forms of justification. For each category the evaluation team created a scoring scale enabling it to assess several degrees of justification. Each scale was focused on a typical measure, an anchor measure, which was used as a benchmark for other measures. All the measures were then rated, mainly by means of the qualitative data collected beforehand. The rating made it possible to identify about twenty sensitive measures, likely to be concerned by important budgetary decisions. These sensitive measures were evaluated in depth. Finally, the evaluation team recommended that a series of measures be either abandoned or strengthened. In a significant number of cases, the decisions taken were in line with the recommendations. Constructing and using scoring scales This example applies to an evaluation of the relevance of measures. The creation of scoring scales for impacts can be inspired directly by it. The procedure leads to the creation of scoring scales such as the one presented in Box 20. It is used in the following way: • The evaluation team makes an initial, rapid recapitulation of the information collected and the analyses performed. It identifies a few impact clusters, such as impacts on employment, equal opportunities, the environment, etc. in such a measure benefited from • For each impact cluster, the evaluation team selects sentences describing the different aspects of the observed impacts, such as: "the women who participated the reorientation towards secure jobs traditionally occupied by men" or "a certain measure had a positive impact for its addressees and the proportion of women who benefited from this impact was particularly high". The evaluation team rewrites these sentences as generally as possible so that they can apply to most of the measures. Each sentence clearly and precisely describes a situation considered successful in terms of its impact. These sentences are called "impact statements". For a more rigorous reconstruction of impact statements and clusters, a concept map of impacts can be used. • Each statement describes a situation considered successful from the point of view of its impact. In reality, certain situations will reveal partial success, an absence of impacts or even counter-performance. To be able to treat all these different possible cases, the evaluation team must describe several possible levels of success. Scales of impacts with several levels between 0 (no impact) and 10 (excellent impact) can thus be constructed. The sentences describing the different levels of impact are called "descriptors". For each impact statement, several descriptors for positive impact levels must be written. Negative impact levels must also be provided for. • The evaluation team then groups together on the same scale all the descriptors concerning an impact cluster, as shown in the example in Box 20. If the same scale refers to several statements, the range of variation of each statement is reduced (for example, from -10/+10 to -5/+5), so that all the scores of the same sheet are always between -10 (case in which all the most unfavourable descriptors are chosen) to +10 (case where all the most favourable descriptors are chosen). • A half-day seminar is then organised with an expert panel or with the enlarged steering group. Technical solutions for evaluation within a partnership framework The evaluation team presents its work to the participants, starting with the first impact cluster and continuing through to the last. For each impact cluster, all the statements and descriptors are discussed and improved, until they are understood without any ambiguity. The evaluation team presents its proposed scoring scale. This scale is discussed by the participants and improved until it correctly reflects their opinion. Box 20 ­ Scoring scale consisting of impact descriptors First Choosing impact —»­ a statement descriptor Reorientation of women towards secure jobs traditionally occupied by men +5 (maxi) Intermediate levels Maintenance of women in threatened jobs traditionally occupied by them ­5 (mini) Positive impact for the addressees; a particularly large proportion of women benefited from this impact +5 (maxi) Second impact statement Choosing ■a descriptor Intermediate levels Positive impact for the addressees but far too few women benefited from this impact ­5 (mini) Total score for the impact concerning equal opportunities between ­10 and +10 After the seminar, the evaluation team once again summarises the information collected and analysed. For each measure, it chooses the descriptor(s) on the scoring scales which seem(s) to reflect its impacts most accurately. This enables it to rate the impacts of the measure (see Box 21). The descriptors chosen do not need to be adjusted very precisely to fit the conclusions that the evaluation team thinks it has to draw from its observations. Apart from the rating, the team also writes up complete qualitative conclusions, which gives it full latitude for qualifying Constructing and using scoring scales its statements. The main advantage of scoring scales is to provide a framework for studying all the measures according to an homogeneous structure which will facilitate the writing of synthesised conclusions on a programme-wide scale. Box 21 -Evaluation of a measure by means of a scoring scale relative to an expected impact First impact statement Choose -»- a descriptor Second impact statement Choose a descriptor Reorientation of women towards secure jobs traditionally occupied by men +5 maxi) Descriptor chosen +2 Maintenance of women in threatened jobs traditionally occupied by them -5 (mini) Positive impact for the addressees; a particularly large proportion of women benefited from the impact +5 (maxi) Descriptor chosen +3 Positive impact for the addressees but far too few women benefited from the impact -5 (mini) Total score for the impact concerning equal opportunities +5 The evaluation team thus obtains a synoptic table of the qualitative conclusions and an impact scoring matrix, in precisely the same form as those described above (see Box 16 and Box 19). The use of this matrix is described later in this section. From scoring scales to indicators If multiple scoring scales are constructed specifically for the needs of the overall evaluation of a programme and are used only once, they should be Technical solutions for evaluation within a partnership framework as light as possible; that is to say, there should be very few descriptors per scale. A reasonable number of descriptors could be considered to be between four and eight. The same scoring scale can be used on several occasions, as in the following situations, for example: • A scoring scale was created for a mid-term evaluation of measures and was subsequently to be used in the framework of project selection. • A scoring scale was created for the mid-term evaluation of measures and was subsequently to be used again for ex post evaluation. • A scoring scale was created for the evaluation of a programme and was to be used again for the evaluation of another programme. If a scoring scale is to be used several times, it warrants more precision. The scale in Box 21 could, for example, be increased from four to twelve descriptors, to attain the detail shown in Box 22. Box 22 - Refining and harmonising scoring scales What is the proportion of women among the participants? What proportion of the women participants obtained a qualification for a job traditionally occupied by men? What proportion of the women participants enhanced their employability? What proportion of the women participants found a secure job corresponding to their qualification, whithin one year? Many 0 Few 1. 5 None Η 2 1 0 E Total score for the impact concerning equal opportunities = 5. 5 An additional step can be taken when the scoring scale has already been applied at least two or three times. At that stage, it is possible to validate it, to stabilise the definitions and to use it in a questionnaire survey for indicators. It will also allow for comparisons quantifying harmonised between different measures or programmes, as shown in Box 23. Constructing and using scoring scales Box 23 - From scoring scales to indicators Evaluated measure ! Average of ! comparable measures Proportion of women participants 72% 45% Proportion of women participants who obtained a qualification for a job traditionally occupied by men 5% 23% Proportion of women who enhanced their employability 52% 55% Proportion of women who found a secure job corresponding to their qualification, whithin one year 25% 29% Using ratings for synthesising the evaluation of measures Overview of programme impacts Box 2 4- Example of an impact scoring matrix in the case of an urban development programme Contribution of the measures to the production of expected impacts (score between 0 and 10) Social integration Employ­ ment Urban development (average score for measures 1 and 2) Infrastructure (average score for measures 3 to 8) Collective social actions (average score for measures 9 to 14) Personalised social actions (average score for measures 15 to 19) 1 5 6 6 1 4 5 6 Balance between towns in a region Balance between neighbour­ hoods within a town 1 4 3 2 2 3 3 2 The scoring scale technique has the advantage of giving an overview of the programme in a summarised form which provides a sound basis for writing synthetic conclusions. For example, the mid-term evaluation of an urban development programme, carried out by means of case studies, produced large numbers of qualitative conclusions concerning impacts. To synthesise these conclusions, the evaluation team constructed and used scoring scales. Box 24 shows how all the qualitative conclusions were synthesised in the form of an impact scoring matrix. In such a matrix, the measures (or, here, the groups of measures) can be compared column by column. It could be said, for example, that a particular measure is more successful than another one in terms of its employment impact, but that it is less successful for another criterion. On the other hand, this matrix does Technical solutions for evaluation within a partnership framework not allow for additions or comparisons between columns. A score of 5 in the first column cannot be compared and certainly cannot be added to the score of 5 in the last column. For reasoning or judgements which combine several columns, it is necessary to use the multicriteria analysis technique (see Chapter IV in this Volume). Judging the coherence of p r o g r a m m es in view of the impacts obtained In the example of the urban development programme presented in Box 24, the impact scoring matrix was also presented in the form of a diagram, to visually show the impacts and their complementarity. This diagram, shown in Box 25, is an important step forward compared to the one that could be drawn up at the start of the programme (see Box 7) in so far as it shows the importance of impacts actually obtained, as opposed to the logic of expected impacts only. Box 25 - Contribution of measures to the production of impacts Contribution thickness of the arrows proportional to the scores LJlJ'. Hii^i-; ^ — ts p ^ ^ ^* ^^^^^\s^" *s^ ^^^-~~^~^ Balance between towns in t he region ^ ^ ^ ~" ^^ Balance between ^ V ^ ^ ^ ^ ^ ^ "^ ^ -* in the town Measures Measures 1 and 2 Urban renewal Measures 3 to 8 Infrastructure Measures 9 to 14 Collective social actions Measures 15 to 19 Personalised social actions -^•^^^ ^ * * * * * * S^ Social integration This diagram has the advantage of visually revealing the coherence of the programme. It can be seen, for example, that certain measures contributed very little towards the overall success of the programme. Similarly, certain impacts are more prominent and attained in a complementary manner by Constructing and using scoring scales several measures. On the other hand, the weakness of certain impacts suggests that several objectives are not achieved satisfactorily, although they are considered important by several partners of the programme. Analysing the programme using a matrix of cross impacts Apart from the success of each measure, evaluation must also analyse the interaction between measures. It is to be expected that the impact of a coherent set of measures will be greater than that of the same measures implemented separately. The third tool presented in this Volume, the matrix of cross impacts, can be used to help evaluate the effects of synergy. This tool, based on a synthesis between approaches proposed by several European teams, was formalised and tested within the framework of the MEANS programme. A solution for analysing synergy within a programme Relationship b e t w e en programming and synergy A programme is composed of a set of projects which all have the same objective. Theoretically, a programme might consist of the juxtaposition of independent projects, of which the total effect is simply the sum of the individual effects. In such a situation, programming would be a purely formal exercise, for the objective could be achieved just as easily by managing the projects independently. Programming adds value because the projects reinforce one another, creating an effect of synergy. Attempting to achieve such synergy is one of the raisons d'être of programming. With the reform to the Structural Funds in 1988, the notions of programming and synergy appeared to be inseparable. Emphasis was placed on the need to achieve the highest level of effectiveness by means of an integrated programme. "An integrated programme is aimed at obtaining an effect of synergy between different measures, and at guaranteeing convergence between the effects of the different partners with different projects and responsibilities" (Extract from the Structural Fund regulations, 1988). Box 26 - QUBIST, a tool for the prior assessment of synergy between projects In 1986 the Commission requested Queen's University in Belfast to develop a software package for assisting in the selection of projects. This tool was constructed in the context of the preparation of an integrated development programme in Northern Ireland. The tool applies to all projects which could qualify for inclusion in a programme. For each project examined, the first step is to clarify its probable contribution to the programme objectives (a weight is attributed to the objectives, according to their importance). The probable relationship between the project examined and the other projects (synergy) is then shown. The results of these analyses are grouped together in a synergy matrix which serves to calculate the synergetic value of each project (1 point per relationship identified). The projects are then classified in terms of the best relationship between their cost and their potential synergy. The final selection of the projects takes into account this ranking established on the basis of criteria of synergy, as well as the potential contribution of the projects to the programme objectives. Technical solutions for evaluation within a partnership framework The characteristics of an integrated programme are twofold. The idea is, first, to achieve greater complementarity between projects and thus to optimise the allocation of resources. It is also to mutually reinforce interventions co-funded by the different institutions aiming for the same goal. From the time the principle of programming first appeared, the Commission tried to develop tools for evaluating synergy. The first efforts resulted in the development of a technique called QUBIST in 1986 (see Box 26). This technique was designed for evaluating the potential synergy of projects at the time of their selection. It has rarely been used, in its original form. To accompany the development of programming and evaluation, the Commission wanted to continue the development of tools for analysing synergy, as part of the MEANS programme. A seminar of experts was held in Brussels in 1993, for the purpose of comparing the best cases in this domain. The seminar produced a methodological report and technical proposals. With the ex post evaluation of Objective 2 programmes for the 1989-93 period, the Commission decided to introduce into its terms of reference a question relating to synergy, and to submit the technical proposals of the MEANS programme to evaluation teams. Several evaluation teams were thus able to test the feasibility of these technical proposals. These tests resulted in the improvements and modifications discussed in the following pages. Definition of synergy and related notions The term "synergy" is often used and is part of the standard vocabulary in many domains. As a result, its meaning is sometimes vague. For example, a firm will enter into partnership with another firm with a view to developing synergy. In pharmacology, synergy is said to exist if two substances administered simultaneously produce an effect that differs from that which would have resulted had they been used separately. It is the latter definition that applies in this book, i. e. there is synergy when several public interventions (or several components of an intervention) together produce an impact which is greater than the sum of the impacts they would produce alone (e. g. an intervention which finances the extension of an airport which, in turn, helps to fill tourist facilities, also financed by the intervention). Analysing the programme using a matrix of cross impacts Synergy generally refers to positive impacts. However, phenomena which reinforce negative effects, negative synergy or anti-synergy may also be referred to (e. g. an intervention subsidises the diversification of enterprises while a regional policy helps to strengthen the dominant activity). The fact that synergy can appear within a programme can be explained above all by the notion of a critical mass: a sufficiently strong impulse is necessary to cross a certain threshold in programme impacts. Below this threshold nothing happens and above it there is a sequence of effects, not unlike a snowball effect. Box 27 shows that synergy may result from many other mechanisms. In all cases, synergy is a dynamic notion: it is the product of a process in which different kinds of investments interact within a given territory. Box 27 - The dynamic effects of programming A Cambridge Policy Consultants study, entitled Program Induced Dynamic Effects, suggests a list of mechanisms likely to produce synergy effects: • critical mass effect: an isolated project would have no impact without a critical mass; • scale effect: reduced costs due to the presence of several projects in the same place; • demonstration effect: duplication of the action through example; • adjustment effect: better co-ordination between the different public actions; • capacity effect: an isolated public action would have no impact without an adequate critical mass; • productivity effect: economies of scale of the public action itself improve its productivity; • freeing up effect: the implementation of a project removes a constraint weighing on another project; • impulsion effect: the implementation of one project stimulates that of another project. In the case of socio-economic programmes, synergy can be obtained by different means: co-funding; partnership; articulation of complementary projects within the same territory; convergence of several forms of support towards the same addressees; circulation of information between operators; etc. In all cases, synergy is the product of complex mechanisms which are formed within a system and which materialise over the medium term. Synergy may, moreover, be internal or external, depending on whether it Technical solutions for evaluation within a partnership framework occurs within the system constituted by the programme or spreads to the effects of other public interventions. The diagram in Box 28 represents the mechanisms which produce effects of synergy. Internal synergy is represented in the centre of the diagram and external synergy on the right. The left-hand column represents an imaginary situation (a counterfactual situation), in which the programme measures are independent from one another and from other public interventions. Box 28 - Logical diagram of effects of synergy Funding authorities Joint decision­ making Joint decision- making Programme objectives Measures Sum of isolated effects Integration (In terdepen dence between programme objectives) Complementarity (befween actions and objectives) Integration (Interdependence with the other policy priorities) Internal coordination External coordination Effects of internal synergy Effects of external synergy Socio-economic development Internal synergy is the product of internal coherence composed of three elements: • interdependence of objectives (objectives are linked to one another); • complementarity between measures and objectives (measures have the same objectives); • co-ordination between measures (measures are chosen and implemented so that their effects can meet in time and space). Analysing the programme using a matrix of cross impacts These are not the only elements needed to obtain effects of synergy, but they are essential for an analysis. Internal synergy has many different forms, as the following examples show: • The development of a tourist resort is favoured by the construction of a road to the coast, and both are subsidised by the programme (mutual reinforcement). • Support for university laboratories and incentives for technological development in enterprise allowing the growth of knowledge networks industrial innovation and the promotion of technologies (combination of critical mass and snowball effects). in new results that • The combination of assistance to different levels of an agricultural branch helps to minimise costs (effect of size). The opposite of this notion of synergy is what can be qualified as negative synergy, for example: • The construction of an iron and steel plant and a pleasure resort next to each other on the coast. • The granting of assistance to different competing levels of the same agricultural branch (dispersion effect). The second category of synergy, called external synergy, applies to the mutual strengthening or weakening of the effects of the programme and those of other interventions, e. g. when the aim is to achieve synergy between aid for SMEs granted by Structural Fund programmes and loans granted by the European Investment Bank. Another, less positive example is that of a programme for the integration of jobless people, financed by the European Union, which recruits a proportion of its participants at the expense of training programmes financed by national authorities. This type of negative synergy has been called a "cannibalism effect" by certain evaluation teams. The evaluation budget does not always allow for a study of external synergy. However, in overall evaluation it is advisable to make a list of the main European, national and regional policies related to the domain of the evaluation in order to obtain an idea of potential external synergy. This approach may prove highly relevant, without requiring a time-consuming analysis. Evaluating synergy in three phases Synergy is rarely evaluated because it is not an explicit programme objective, despite it being a key mechanism for achieving objectives. The evaluation of Technical solutions for evaluation within a partnership framework synergy is often limited, in the framework of an a contrario approach, to the observation of a lack of co-ordination or of wastage related to the duplication of interventions. It is also limited by available resources. The time and budget available for evaluation are often mobilised by other areas of intervention, considered priorities, such as employment or the environment. Certain very real technical difficulties exist. First, there are so many interactions which favour synergy that they cannot all be examined. It is also necessary to take into account the absence of reliable data which would make it possible to distinguish the impact of all the measures of a programme from the impact that would have been produced by the same measures had they been implemented separately. The solution proposed in this Volume is based on the use of a tool called the matrix of cross impacts. This solution was designed to be applicable without substantially increasing the volume of work devoted to the overall evaluation of a programme. It should be seen essentially as an additional analysis of information which has inevitably to be collected for other purposes. Box 29 - A procedure for evaluating internal synergy Identification of potential synergies Analysis of synergy effects 2 -4 2 Empirical verification of main effects i 3 Empirical testing of this tool showed that its application had to remain realistic and flexible. It is for this reason that its use has to be broken down into three phases to be adjusted according to the time and budget available for the evaluation: • a phase of construction of the matrix for identifying potential synergy; • a phase of global analysis of effects of synergy by means of the matrix; Analysing the programme using a matrix of cross impacts • a phase of empirical verification of the main effects, selected on the basis of the matrix. Only the first two phases are absolutely indispensable when the evaluator has a very tight schedule. They offer a sound basis for the analysis of synergy within the programme under consideration. The third phase provides information from the field, making it possible to confirm or invalidate the assumptions made by managers in the second phase. It is strongly recommended in cases where the question of synergy must be considered more in-depth, and/or when the first two phases have produced no credible conclusions. Instructions for using the matrix of cross impacts The construction and validation of the matrix of cross impacts consists of several successive stages: formulation of the structure of the matrix; scoring the effects of synergy; validation with the managers; calculation of coefficients of potential synergy; and, lastly, overall presentation. Choice of the level of analysis of the synergy Depending on the structure of the programme concerned, it will be more relevant to analyse synergy between the axes, the measures, the actions or the projects. The level of analysis chosen obviously depends on the number of programme components at each level. Some programmes consist of only a few projects, which makes it possible to rapidly analyse synergy at their level. If the number of projects is very high, it may be preferable to analyse synergy at the measures level. Box 30 -Choice of a level for analysing synergy in an objectives tree Programme Axes Measures Actions Projects 11 Training of staffin SMEs L 12 Support for investments by SMEs hj \/ 1 Support for productive activities Growth of employ­ ment _^ Training courses X. Y. Z Creation -► Enterprises Χ, Υ, Ζ 1 Modernisation -*■ Enterprises Χ, Υ, Ζ ' Industrial - Centres Χ, Υ, Ζ Urban ■ Centres Χ, Υ, Ζ -►Centres Χ, Υ, Ζ The lines in bold type represent the most obvious potential synergy. Technical solutions for evaluation within a partnership framework The choice of a level of analysis can be made by referring back to the objectives tree (see Box 6). Those relations of potential synergy which appear most obvious can be marked on this diagram of the programme, as shown in Box 30. In the example of the Catalonian Objective 2 programme, the most appropriate level of analysis seems to be the measures level. It is at this level that the largest amount of potential synergy was identified. ■ Interaction between measures Once a level of analysis has been chosen, the matrix of cross impacts is constructed with as many lines and columns as there are programme components at that level. The cells in the matrix allow all identified synergy to be marked, as shown in following box. Because the matrix is considered symmetrical at this stage, only half of it will be filled in. Box 31 ­ Construction of a matrix of cross impacts Axes I. II. III. IV. Creation and Protection and Invest­ Improvement development improvement ments in of the of productive of the R&D and telecommuni­ activities environment training cations network measures 11 12 13 21 22 23 31 32 41 42 43 I. Creation and develop­ 1. 1 ment of productive 1. 2 activities 1. 3 II. Protection and 2. 1 improvement of the 2. 2 environment 2. 3 III. Investments 3. 1 in R&D and training 3. 2 IV. Improvement of the 4. 1 telecommunications 4. 2 network 4. 3 X X X X X X X This approach nevertheless lacks precision and remains purely theoretical. Consequently it is necessary to go further, by qualifying in more detail the importance of expected synergy and by validating this analysis in interviews with the programme managers. Analysing the programme using a matrix of cross impacts The following developments constitute the core of the method: they are the minimum required for a reasonable analysis of synergy. Because this phase requires interviews, they should be planned at an early stage in the evaluation agenda. Rating effects of synergy The evaluation team compares pairs of measures to identify any synergy which may exist. When some kind of synergy seems possible, a value on the following scale is chosen corresponding to the size of the effect: •2 • 1 • -1 • -2 for a particularly strong effect of synergy for a weaker effect of synergy the same scale applied to negative synergy the same scale applied to negative synergy Although the evaluation team expects varying degrees of synergy, it can also estimate that certain effects are very likely and others very unlikely. Unlikely effects can be put in brackets. Box 32 - Scoring potential synergy Axes I. Creation and II. Protection and III. Invest­ IV. Improvement development improvement ment in of the telecom­ of productive of the R&D& munications activities environment training network measures 11 12 13 21 22 23 31 32. 41 42 43 I. Creation and development 1. 1 of productive activities II. Protection and improvement of the environment III. Investment in R&D & training IV. Improvement of the telecom- munications network 1. 2 1. 3 2. 1 2. 2 2. 3 3. 1 3. 2 4. 1 4. 2 4. 3 (2) (2) (2) 2 (1) (D 1 (2) Π) (D (2) (2) (2) Π) (2) (1) (2) (1) (2) (1) -1 (1) (2) (2) (1) (1) (1) (1) At this stage, the matrix of cross impacts represents a quantified synthesis of assumptions of synergy, identified by the evaluation team. Although the Technical solutions for evaluation within a partnership framework matrix of cross impacts appears essentially quantitative, it contains only subjective assessments. The comments and remarks of the evaluation team are essential elements which are worth saving. In some cases the synergy within pairs of measures is unidirectional only. For example, a road to the coast is essential for the success of a tourist resort, but the expected development of tourism owing to the road can be partly achieved without building the tourist resort. This is referred to as asymmetrical synergy. This means that a significant positive effect exists if Measure A is added to Measure B, but that the effect is less if Measure Β is added to Measure A. This characteristic is noted in the qualitative comments. Once the matrix has been filled in, the values on either side of the diagonal dividing the diagram are identical in cases of symmetrical synergy. Only half of the matrix is filled in since the other (shaded) part is identical. If cases of asymmetrical synergy (relationship of non-reciprocal interdependence) are noted, they are indicated on the shaded side of the diagonal. Example: Case of asymmetrical synergy in an economic redeployment programme in Catalonia Among other examples of asymmetrical synergy, the evaluation team noted that "training of unemployed people and workers from the environment and R&D sectors has a highly positive effect on efforts to develop and modernise SMEs. On the other hand, support for the development and modernisation of SMEs has a comparatively weaker positive influence on the training of the unemployed and workers from the environment and R&D sectors". Validation with the managers Once the matrix has been filled in, the evaluation team organises interviews or a meeting with the main programme managers. The aim is to discuss and validate assumptions of synergy presented in the matrix. For each question on synergy, the explanations and arguments chosen by the managers are recorded in the form of qualitative comments attached to the matrix, with an indication of the probability of that synergy appearing (see Box 33). The detailed commentary is very important for, in cases where time is limited, it serves to make a qualitative analysis which produces results and can be applied to the implementation of the programme concerned. Analysing the programme using a matrix of cross impacts Box 33 - Comments on potential synergy Relations between measures 11 &12 21 &23 Effects of interactions Managers' opinions Probability yes strong no weak Better qualified staff facilitate the decision to modernise a SME Training favours the choice of non- polluting production methods Coefficients of potential synergy Once the assumptions of synergy have been validated with the managers, the matrix of cross impacts can be finalised by calculating the coefficients of synergy (Cs+). They are calculated for each measure and for the programme as a whole. The formula is as follows: Cs+ = Sum of positive scores (Number of positive scores) χ 2 If all potential synergies between measures were to receive the maximum score, the coefficient would be equal to 1. 00. The coefficient would be equal to 0. 00 for a programme with no positive synergy. If negative synergy exists, the synthetic coefficient of negative synergy is also calculated (Cs_): C = Sum of negative scores (Number of negative scores) χ 2 To facilitate the calculation of coefficients, six columns must be added to the matrix of cross impacts as indicated in Box 34. The columns "s+" et "s-" sum values of synergy and negative synergy respectively, while the columns "n+" and "n-" count the number of positive and negative scores presented in the matrix. Columns Cs+ and Cs- calculate the synthetic coefficients of positive and negative synergy. The calculation of synthetic coefficients can be extended to the entire programme by calculating the average of all the non-zero coefficients Technical solutions for evaluation within a partnership framework (synthetic coefficient of positive synergy = 0. 7 and synthetic coefficient of negative synergy = -0. 5). It is important to note, however, that the advantage of this calculation is limited to those cases where the same work has been done for other programmes by evaluation teams who used the same scoring methods. Since that is highly unlikely to be the case, the coefficients of synergy will be far more useful as criteria for a comparative judgement of measures within the overall evaluation of a programme. Box 34 - Calculation of synthetic coefficients of synergy I. 1. 1 2 2 2 1 2 1 2 1 2 1 1. 1. 1 1. 2 1. 3 II. 2. 1 2. 2 2. 3 III. 3. 1 3. 2 IV. 4. 1 4. 2 4. 3 T. II. III. IV. Tol N° Coef 1. 2 1. 3 2. 1 2. 2 2. 3 3. 1 3. 2 4. 1 4. 2 4. 3 s+ s- n+ n- Cs+ Cs- 2 2 1 1 11 2 1 1 11 1 -1 2 -1 1 2 1 12 1 1 2 11 11 1 2 2 1 1 12 2 2 2 1 -1 -1 10 8 7 5 3 8 6 8 3 5 2 6 7 4 3 3 6 5 5 2 3 2 0. 83 0. 57 0. 87 1 1 0. 83 -0. 5 0. 5 -0. 5 0. 66 0. 60 0. 80 0. 75 0. 83 0. 50 65 -2 46 2 0. 7 0. 5 Empirically verifying the main synergies identified Presenting the global evaluation of synergy On the basis of coefficient calculations complemented by comments, the evaluation team proposes recommendations for improving or triggering effects of synergy within the evaluated programme. In the example above, inspired by the evaluation of the Objective 2 programme in Catalonia, it appears that Measures 1. 1, 1. 3, 3. 2 and 4. 2 are essential for the success of the programme, given their synergetic effects. By contrast, the disappearance of Measure 4. 3 should not affect the effectiveness of the other interventions. The question will be raised for 1. 2 which has a fairly weak coefficient of synergy (0. 57), but which enters into synergy with many other measures (7). It has to be concluded in this case that 1. 2 has a weak potential for synergy although having numerous interactions, since these are individually weak. Measure 2. 1 which combines positive and negative effects of synergy must also be examined in a qualified way. In fact, it is always necessary to qualify the interpretation of coefficients in relation to the calculated synergies (columns "s" and "n" of the table). For a global approach to a programme, positive and negative coefficients of synergy can thus be used as two evaluation criteria among others. For the purpose of multicriteria analysis, as described in the following chapter, these two columns of results can be included in the multicriteria matrix. The scores concerning synergy are found alongside other criteria relating to impacts on the environment, employment, etc. Identification of synergy to be evaluated This phase needs only to be undertaken should the synergy previously identified be of major interest for the continuation or success of the programme. It has the advantage of transforming the assumptions formulated during the preceding phases into a real evaluation of the effects of synergy, as observed in reality. It requires the extension of interviews to include the addressees of the programme (the target public), which means that the evaluators must be able to identify and meet the addressees in order to obtain relevant information. If this phase is to run smoothly, adequate time and resources must be available for the interviews, before the final evaluation report is written. Technical solutions for evaluation within a partnership framework The matrix of cross impacts can also serve to identify the most critical interactions between the measures taken two at a time. An interaction is considered critical if it has been assumed that it produces a high level of synergy (whether positive or negative) or if very different assumptions have in the qualitative been made by the people questioned (mentioned comments). Similarly, interaction between measures can also be considered critical if the managers of those measures coordinate their work particularly well or badly. An example is when rivalry between the partners of a programme impedes the success of measures programmed for the very purpose of creating synergy. When an interaction is considered critical, a "window" is opened in the matrix of cross impacts. The assumptions of synergy are then developed and verified in detail within a limited number of windows. The commissioner of the evaluation is called on to express his or her opinion on the number and position of the windows to open because they influence the duration and cost of the operation. In Box 35, it was considered that the synergy within the (shaded) axes was not a priority. The areas with the highest values (a score of 2) and/or the most disputed values (scores in brackets) were given priority. This resulted in three windows being opened for the empirical verification of synergy. Box 35 - Opening of "windows' of synergy for the empirical verification I. 1. 1 (2) (2) 2 1 (2) (D (2) (1) (2) (D I. II. III. IV 1. 1 1. 2 1. 3 2. 1 2. 2 2. 3 3. 1 3. 2 4. 1 4. 2 4. 3 1. 2 1. 3 II. 2. 1 2. 2 2. 3 III. 3. 1 IV 4. 1 3. 2 4. 2 4. 3 -1 (2) (1) (1) (1) (2) (2) (2) (1) (1) (D (2) (2) (2) (D (1) (1) (D (1) 0 0 0 Analysing the programme using a matrix of cross impacts Empirical verification of the effects of synergy The empirical verification of synergy necessitates interviews with the addressees of the measures concerned. An addressee is an individual, a firm or an institution which has been affected in some way by the programme, either directly or indirectly. The "windows" opened during the preceding stage may concern: • individuals, enterprises or institutions within the same target group; • a set of individuals / firms / institutions within the same territory; • or a set of firms which belong to the same network (branch of production or sector of activity). In all instances, those addressees who have been affected or who might have been affected by the measures studied must be identified. The evaluation team tries, in this way, to compare the assumptions of synergy, made in co-operation with the managers, with that which can be observed empirically. Depending on the number of addressees, the evaluation team holds individual interviews, a focus group or even a series of focus groups, when possible. Further information on focus groups can be found in Volume 3 of this Collection, devoted to evaluation tools. The interviews review assumptions made during the construction of the matrix of cross impacts. Their contribution is qualitative; it is therefore essential to note the beneficiaries' arguments. Exceptionally, an in-depth analysis of the synergy between the measures concerned by a "window" can be made. This implies the multiplication of interviews, to compare information or the carrying out of a series of case studies. This operation makes it possible to understand the nature and mechanisms of a particular effect of synergy in depth. It is expensive and must be planned very early in the course of the evaluation. It can be particularly useful if the conclusions obtained can be generalised to other programmes (external validity). Consolidation of the analysis and recommendations The evaluation the managers' team must systematically compare estimations and its own observations in the field. This comparison will be given to the managers of the measures and of the programme, in a qualitative form. Technical solutions for evaluation within a partnership framework The idea is to explain differences of interpretation, but also to validate the different types of synergy which appear in the programme and to know their nature. In this respect, it is important for the managers to obtain this information as early as possible, so that they can make adjustments to the implementation of the intervention. The matrix of cross impacts is adjusted, taking into account the empirical verifications if this is considered necessary. The calculation of synthetic coefficients of synergy can then be reiterated, measure by measure. Box 36 presents, for each domain, the effects of synergy that were most frequently recorded by the teams evaluating programmes financed by the Structural Funds in areas of industrial redeployment. Box 36 - Main effects of synergy observed in evaluations Domain Aid to enterprise Aid to enterprise Tourism Infrastructure Training Technology 1 » • • Tourism Infrastructure Training Technology - ) > ) - O » - • Key : Effects of synergy • Strong ► Average O Weak in Evaluations which addressed the question of synergy, especially Objective 2 programmes, revealed interesting cases of decentralised co­ ordination. For example, in the Twente and Groningen-Drenthe regions (NL), work groups were used to promote integration between the measures of different axes. In the Bremen region (FRG), a "Coherence Fund" was adopted as an instrument for encouraging operators to co-ordinate. In the Nord-Pas-de-Calais region (F), a "central observation office" has the same mission. IV Using multicriteria analysis within a partnership framework Multicriteria analysis serves to ground synthetic conclusions, taking into account all the criteria considered relevant by the evaluation team. This capacity is particularly useful in partnership evaluation because it allows the expectations of each decision-maker to be taken into account. Multicriteria analysis is traditionally used as a decision-making tool. There are multiple technical variants of this tool, the principles of which are described in this chapter. By selecting some of these variants and using the analysis as an aid for making judgements, this Volume proposes a new use for the technique. It consists of grounding conclusions which classify measures of a programme in terms of their success, according to a procedure acceptable to all the partners. To apply this technique, the evaluation team must have complete conclusions on all the measures and impacts, summarised, if possible, in scoring scales. their main A solution for judging the overall success of a programme Taking into account all relevant criteria for grounding a j u d g e m e nt When it concludes its work, the evaluation team must state its opinion on the respective merits of the different measures and, in particular, on their contribution to the overall success of the programme. If the evaluation does not reach this stage, it cannot lead to recommendations of a sufficiently strategic level in terms of improvement or reorganisation of the programme. However, experience has shown that evaluation teams are reluctant to formulate such conclusions, especially because they imply judgements on the relative value of the results and impacts obtained. If the evaluation team or steering group are reluctant to compare the merits of the different measures, this is often through fear of formulating such a value judgement and especially through fear that this judgement will be considered as too subjective. It is, however, not unusual for an evaluation to formulate value judgements. Is the word "evaluation" itself not related to the word value? In this perspective, the evaluation team must strive to ground its judgement on explicit and accepted criteria. As the following example shows, progress still needs to be made in improving the justification of conclusions. Example: Conclusions containing an ill-justified judgement In the conclusions of an evaluation report, the following two judgements might be presented: "Measure A is greatly appreciated by professionals" "Measure Β has a very high cost per job created" In this example, the evaluation team ends its report conclusively, thus demonstrating a quality which is not always present. Moreover, its judgements are supported by factual elements, which is also a positive point. It is regrettable, however, that these conclusions are not justified more fully. For example, as far as Measure A is concerned, the report does not explain why the professionals' view is enough. As for Measure B, it would be interesting to know why, and in relation to what norm, it can be concluded that the cost per job created is very high. It would also be useful to know whether the measure has other positive effects likely to offset the high cost per job created. Technical solutions for evaluation within a partnership framework If the evaluation team does not adequately explain its judgements, the report as a whole is fragile. The entire credibility of the evaluation is then based on the trust that is placed in the evaluation team's impartiality - something which can always be doubted (if necessary with a bit of bad faith). To avoid this, but without abandoning the idea of producing real conclusions, the evaluation team can help itself by means of several techniques enabling it to explain its judgements and protecting it from the strongest criticism. Box 37 presents four of these techniques and their main characteristics. Box 37 - Aids for making a judgement Number of effectiveness criteria Who decides on the respective importance of the criteria Cost- effectiveness analysis Satisfaction survey Cost- benefit analysis Multicriteria analysis One only Many Many Many (not necessary) The addressees The evaluation steering group The evaluation team, with the opinions of the winners and losers Are the criteria explicit? Yes No Not entirely Yes The simplest technique consists of deciding whether the judgement will refer to a priority impact, generally related to the global objective of the programme. Two criteria are then applied: the achievement of the priority impact, and the inputs mobilised. In such cases the evaluation takes the form of a cost-effectiveness or value-for-money evaluation. For example, it compares the different programme measures in relation to the cost per job created. framework of European Union socio-economic programmes, it is difficult to be satisfied with such a simplified technique. In reality, the contribution of all the measures to the achievement of a global objective (e. g. socio-economic development) is not easy to estimate because such an objective is too vague and distant (see page 21). A situation in which the merits of a measure are judged in relation to several criteria (e. g. competitiveness, diversification, employment, environment, etc. ) is more likely. The evaluation team must be careful not to forget any the In Using multicriteria analysis within a partnership framework relevant criteria, or it could make the conclusions unacceptable for one of the readers of the report. It is therefore necessary to use techniques which allow for judgements to be made by taking several criteria into account. identified all One type of multicriteria judgement, often chosen implicitly, consists of having the addressees judge the measure. When an evaluation team concludes, after a survey on the assisted firms, that "the measure is highly appreciated by the professionals", that amounts to saying that the professionals questioned had the advantages and disadvantages of the measure and synthesised them when they answered the interviewer's questions. After extrapolation from the survey sample, a multicriteria judgement is obtained, but its main drawback is that it is totally opaque. It is impossible to find any trace of what counted in individual it is based only on direct judgements. Its other drawback addressees' opinions, considering that their judgement may be biased by an accommodating attitude towards the administration which granted them assistance. is that For structuring judgements well and taking all impacts into account, the evaluation manuals also suggest cost-benefit analysis. With this technique, each impact is identified and estimated quantitatively by the evaluation team. It is then weighted with a monetary value (in some cases called a fictive price). This value represents what society or, rather, the people concerned are prepared to pay to obtain the impact. Negative impacts are also given a value in terms of the willingness to pay to avoid them. Cost-benefit analysis is a method which theoretically results in a clear judgement in the form of a "yes or no", without requiring any form of comparison. The judgement is positive if the estimated advantages for the groups who win are greater than the estimated disadvantages for the groups who lose. In practice, this simple equation is rarely applicable because so many projects or interventions pass the cost-benefit test that they cannot all receive public funding. A positive cost-benefit balance sheet is therefore not enough for making a final judgement on an isolated intervention. The use of cost-benefit analysis is often tricky because many impacts are difficult to translate into incontestable monetary terms (particularly environmental impacts). Moreover, the technique is extremely demanding in terms of impact quantification, which makes it difficult to apply to a very large socio-economic development programme. In practice, its use is recommended mainly for certain ex ante evaluations of large projects. Technical solutions for evaluation within a partnership framework Unlike the preceding three approaches (cost-effectiveness analysis, surveys and cost-benefit analysis), multicriteria analysis is an approach which requires the steering group of the evaluation to clarify the judgement criteria. This is often an advantage in a partnership context. The technique is not particularly demanding in terms of the quantification of impacts, and qualitative conclusions can easily be taken into account if they have been structured by means of scoring scales. Box 38 -A European workshop on evaluation techniques adapted to a partnership context Evaluations of European Union socio-economic programmes are organised and performed within a context of partnership between the Commission and the national and/or regional authorities. As part of the MEANS programme, three workshops attended by about ten European experts and practitioners were held for the purpose of identifying difficulties of partnership evaluation and proposing technical solutions. The participants recognised that differences of interests frequently appeared when objectives were clarified jointly, when ad hoc indicators were constructed and when benchmarks were established. Many techniques were mentioned during the three workshops: the logical diagram of impacts; scoring scales; multicriteria analysis; SWOT; METAPLAN®; and the colour vote. These techniques are generally so simple that some have been accused of being childish. Their advantage is, however, that they structure the partners' reflections and help to deal with disagreements without threatening working relations. Partnership evaluation is not free from power play and the fact that participants tend to talk the official language of their institutions. If the steering group is broadened to include the main partners of the programme, it is often the victim of a sort of "ritual of courtesy" which papers over differences in points of view and censures certain sensitive questions. Techniques such as multicriteria analysis favour the expression of disagreement within the steering group, which avoids phenomena of self-censorship and enhances the relevance of conclusions. On the other hand, the technique channels discussions within the steering group by avoiding risks of conflict. Evaluation criteria: definition a nd e x a m p l es In order to justify its conclusions, the evaluation team clarifies its judgement references. That is to say, it describes its main criteria, each of which may Using multicriteria analysis within a partnership framework be associated with a measurement unit and a norm to meet, as symbolised in the example of the thermometer in Box 39. Box 39 - Reference judgement, norm and criteria degrees Celsius or % of trainees employed after 12 months measurement unit Judgement reference criterion O temperature or placement rate The main evaluation criteria are relevance, effectiveness, efficiency and utility. Relevance judges whether the objectives correspond to real needs of the economy and society. Efficiency judges whether the results and impacts have been obtained at a reasonable cost or whether they could have been obtained at a lower cost. Effectiveness judges whether the results and impacts obtained will make it possible to achieve the explicit objectives. In the following pages, the notion of effectiveness will be extended to implicit objectives, which gives the term a meaning close to that of the fourth criterion, utility. Utility judges all the results and impacts obtained, whether expected or unexpected, in relation to socio-economic needs. Multicriteria analysis is used below with a strong focus on effectiveness in the broad sense of the term, i. e. the achievement of expected impacts. The analysis is known as "multicriteria" because the partners expect several impacts, each of which is considered a success criterion for the analysis. Instructions for using multicriteria analysis Multicriteria analysis appeared in the 1960s as a decision-making tool. Initially it was developed by mathematicians specialised in operational research. This development was guided for a long time by two separate "schools", one based in France and the other in the United States. These two currents are now tending to merge. Multicriteria analysis serves to illuminate a complex decision, by helping to compare all the solutions envisaged and by taking into account all the points of view considered relevant. As a decision-making tool, multicriteria analysis is currently used in the private sector, for example when a firm chooses a site for location, or in the public sector, for example when a community chooses a way of stocking its waste. In the domain of evaluation in the strict sense of the term, multicriteria analysis is seldom used for purposes other than those closely resembling decision-making aid and, in particular, the ex ante evaluation of transport infrastructure projects. A way of using the technique is suggested below which departs from the context of decision-making, instead taking the form of an aid for making a judgement. Within the framework of socio-economic development programmes, it concerns a judgement on the success of the different measures, for the purpose of drawing synthetic conclusions. This judgement takes into account the main relevant criteria for the steering group. The use of multicriteria analysis in such situations is an innovation. The following instructions for use can, however, be considered operational in so far as they take into account the results of several full-scale tests. The procedure is described in five steps, some of which can be varied. Structuring the analysis Choice of objects to be compared Theoretically, the procedure described below serves to compare the measures of a programme. It is worth investigating, however, whether it could be applied at another level, by breaking down certain measures (i. e. the level of what is often called an "action") or by grouping together similar measures. It is not advisable to compare too few objects (e. g. two or three Technical solutions for evaluation within a partnership framework groups of measures), for multicriteria formalisation cannot be justified in such cases. By contrast, a large number of measures can be compared since there is no limit in this respect (e. g. 50 measures or more). The number of measures analysed could nevertheless be limited by constraints related not to the technique but to data collection. As described above, the analysis is based on an impact scoring matrix which can be simplified if only the most sensitive measures are to be analysed. The decision that must be taken is thus related to cost / utility. If a decision concerning the reallocation of budgets or a new programme is to be clarified, it is logical to focus primarily on the most important measures in budgetary terms, the most innovative measures or the measures whose effectiveness is doubted. Choosing criteria In the case of European Union socio-economic programmes, the success of a measure is normally judged in terms of its contribution to the achievement of the intermediate objectives stated in the programming documents. The main European Union policy priorities (e. g. environment, equal opportunities) are also judgement criteria. A variant consists of relying instead on the implicit objectives of the programme, reconstructed by the steering group or extended work groups, e. g. with the aid of the concept mapping of impacts (Chapter 1). The latter option is presented below. By means of a logical diagram of impacts (see Box 7), and with the steering group's agreement, the evaluation team defines the list of criteria. Each criterion is derived from an intermediate impact. At this stage, the evaluation team must check that the criteria chosen are logically independent from one another. Thus, in Box 40, the impact entitled "improved viability of enterprises" is logically related to the preceding impact entitled "attractive environment". The multicriteria analysis must therefore opt for one of these criteria for judging the measures; it cannot use both. On the other hand, the diagram shows that "attractive environment" and "employability" are considered to be independent impacts. Both can therefore be chosen as judgement criteria. To facilitate the formulation and comprehension of conclusions, it is also important to avoid mixing criteria that are too different. If the evaluation primarily concerns the effectiveness of the programme, multicriteria analysis must focus on the results and impacts obtained by the different Using multicriteria analysis within a partnership framework measures (that is the option chosen in this chapter). If the evaluation focuses primarily on the relevance of the programme, a multicriteria analysis can also be performed but it muet concentrate on the main strengths and weaknesses of the regional economy and the way in which the different measures build on strengths or offset weaknesses. Box 40 ­ Independence and interdependence of evaluation criteria Measures Intermediate impacts R&D \ jr Tourism Industrial wasteland Entreprises Training Opening up to the outside Attractive­ ness of the environment Sectoral diversification Proliferation of SMEs Improved r viability of entreprises Improved employability —*■ ! 1 Interdependent impacts Independent 'mpacts The synergy between the impacts of the different measures is a particular case. It is logical for this synergy to be a judgement criterion in its own right, considering that certain measures are essential for the success of others. If the analysis of the effects of synergy has been performed in the spirit of Chapter III of this volume, it is possible to use a matrix of cross impacts and, in particular, coefficients of synergy for taking this criterion into account in the formulation of a synthesised judgement on the measures. Unlike the number of measures to be compared, which can be very large, the number of criteria must not exceed a reasonable limit. Experience has shown that the participants of a steering group or a work group cannot Technical solutions for evaluation within a partnership framework express valid opinions on more than eight criteria. This is the number chosen in the following example. Example: Criteria chosen for the evaluation of a socio-economic programme in Belgium As part of a mid-term evaluation, a multicriteria analysis was performed, involving an extended steering group of about ten people. Eight criteria were chosen and were given the following names and meanings by the steering group: • Sectoral diversification (the measure orients the region's economic activity towards growth markets). • Strengthening SMEs (the measure improves the share of small and medium-sized firms in the economic activity of the region). • Opening up to the outside (the measure increases exports out of the region). • Improving services to enterprises (the measure enables more businesses to use services in the region). • Better environmental integration (the measure reduces the number of regional firms whose development or survival is hindered by environmental problems). • Better territorial distribution (the measure orients the economic activity of the region towards the most disadvantaged areas). • Better employability (the measure improves the employability of people in the region who have work or are looking for jobs). • Synergy (the measure contributes to the success of other measures in the programme). Multicriteria evaluation matrix Once the judgement criteria and measures (or groups or parts of measures) to be evaluated have been chosen, the evaluation team constructs a multicriteria evaluation matrix. This matrix is a table with as many columns as there are criteria and as many lines as there are measures to be compared. Each cell represents the evaluation of one measure for one criterion. Multicriteria analysis requires an evaluation of all the measures for all the criteria (no cell must remain empty), but does not require that all the evaluations take the same form. As Box 41 shows, the technique can support a mix of quantitative criteria expressed by indicators, qualitative criteria expressed by descriptors, and intermediate criteria expressed by scores. The most pragmatic way of filling in the multicriteria evaluation matrix is to structure all the evaluation conclusions, whether quantitative or qualitative, by means of scoring scales created for that purpose by the evaluation team Using multicriteria analysis within a partnership framework (see page 55). The multicriteria evaluation matrix is then equivalent to the impact scoring matrix described in Box 19. Box 41 - Combination of qualitative and quantitative criteria Criterion Modalities Diversification Employability Environment (% of assisted (impact descriptors) (impact rating for evaluating businesses active the criterion in growth sectors) Measures Investment aid In-house training 7 2% 2 1% between 0 and 10) 3 1 Neutral impact Significant increase in the employability of trainees already qualified; neutral impact for the others Industrial wasteland 5 2% Neutral impact 8 B ox 42 - C a se of e v a l u a t i on c r i t e r ia w h i ch a re t oo s p e c i f ic Criterion Modalities for evaluating the criteria Measures Investment aid In-house training Industrial wasteland Diversification Employability Environment (impact rating (impact rating (impact rating between Oand 10) between Oand 10) between Oand 10) 7 0 0 0 5 0 0 0 8 Before continuing with the multicriteria analysis, the evaluation team must check whether its initial conclusions will allow it to compare measures satisfactorily. In choosing the criteria, the team should already have ensured that they apply to as many measures as possible. The majority of these measures must have produced impacts related to the majority of criteria. In other words, the impact scoring matrix must not have too many neutral, absent or insignificant impacts. If the only scores which are not equal to zero are situated in the diagonal (as in Box 42), it means that the measures Technical solutions for evaluation within a partnership framework evaluated have nothing in common and that their evaluation criteria are specific. In that case, multicriteria analysis cannot be performed. Choice of a procedure for analysis Two main possibilities exist for the evaluation team, for comparing the merits of the different measures: multicriteria analysis by compensation or multicriteria analysis based on outranking. The most well-known variant is the first. It consists of attributing a weight to each criterion and then of calculating a global score for each measure, in the form of a weighted arithmetic average of the scores attributed to that measure for the different criteria. This variant is called "compensatory" because the calculation of the weighted average makes it possible to compensate between criteria. For example, a measure which had a very bad impact on the environment could still obtain a good global weighted score if its impact on employability were considered excellent. No global score is calculated in the outranking variant because the criteria are not all considered commensurable. The analysis is based on multiple comparisons of the type: "does Measure A outrank Measure Β from the point of view of the environment criterion?", "does Measure A outrank Measure Β from the point of view of the employability criterion?", etc. These questions can be answered yes or no (franc criterion) or be qualified, in which case the notions of a weak preference and a threshold criterion are introduced. The analysis makes all possible comparisons and presents a synthesis of the type: "Measure A is at least as good as Measure B, in relation to a majority of criteria (case of agreement), without being altogether too bad in relation to the other criteria (case of disagreement)". This type of analysis often includes a form of protection to avoid a favourable judgement for a measure which would be disastrous from the point of view of the given criterion. For this purpose, a veto threshold is determined for each criterion. Just as qualified preferences can be established between criteria, so too it is possible to qualify the veto threshold by introducing a progressive veto. The introduction of a veto threshold strongly differentiates the logic of outranking from the logic of compensation. If there were a veto threshold, a very bad impact on the environment would make it impossible to consider the measure good, even if its impact on employability were considered excellent. Using multicriteria analysis within a partnership framework Outranking has the advantage of reflecting the nature of relations between public institutions better. When it is said that "Measure A outranks Measure Β for the majority of criteria, and that it is not excessively outranked for any of the criteria", a mode of reasoning is adopted that is very familiar to policy-makers. Everyone knows that a public decision is easier to make if the majority of stakeholders agree and if there is not a radically opposed pressure group. Moreover, there is often a correspondence between evaluation criteria and evaluation stakeholders. In cases where the steering group is extended to about ten partners, it is not unusual for participants to identify themselves strongly with the "environment" or "employment" criteria. In this situation the outranking variant will probably better reflect the collective process of formulating a judgement within the steering group. From a technical point of view, the compensation variant is very easy to implement with any spreadsheet. The outranking variant, on the other hand, requires the use of specialised software which differ according to the type of analysis, of which the most well-known are presented in Box 43. Box 43 - Variants of multicriteria analysis using outranking The main variants of multicriteria analysis which use outranking are: ELECTRE I - This variant functions with an agreement index and a disagreement index, presented in the form of scores. A disagreement threshold (a veto) is introduced for all the criteria. The outranking and veto thresholds are of the franc type. The software processes a situation in which the best measure(s) must be chosen, for example a situation in which the aim is to identify best practice. ELECTRE TRI - This variant serves to sort measures into different categories, for example, the most successful measures, measures which have no significant impact and intermediate measures. ELECTRE II produces a ranking of measures, from the most successful to the least successful. Outranking and veto thresholds are of the franc type. ELECTRE III also performs a classification, but introduces vague outranking relationships. PROMETHEE uses only an index of agreement and introduces progressive outranking. For more information, see the annexed bibliography: Vincke 1989. Outranking does not always produce clear conclusions. The analysis may result, for example, in a classification which considers that some good measures clearly outrank the others, that other measures pose the same Technical solutions for evaluation within a partnership framework problem because they are frequently outranked, and that a third group of measures cannot be ranked. By contrast, if the analysis is based on compensation it is always conclusive. Throughout the rest of this chapter, the compensation variant has been opted for because of its technical simplicity and its capacity to produce clear conclusions. In a logic of overall evaluation of a programme and in a perspective of budgetary reallocation, it is considered that the readers of the evaluation report require conclusions on all the measures evaluated. The formula of the weighted global score is appropriate, provided that each member of the steering group has confirmed that there is no major problem which might justify the use of the veto system. ■ Choice of assessors Multicriteria analysis is an aid for helping the evaluation team to make judgements. It enables the team to explain its judgements and to write final conclusions without being accused of subjectivity. In reality, the technique produces a judgement based on factual and objective elements concerning impacts, but also on the impartial integration of the points of view and preferences of the main partners. It is a technique which cannot be implemented without the effective participation of the programme partners. When using multicriteria analysis, the evaluation team has to hold individual interviews or focus group interviews with a number of people whose points of view are considered most relevant for judging the programme measures. These people are referred to below as assessors. The first option consists of considering the members of the evaluation steering group as assessors. In order to do this, the steering group must be large enough to reflect the main points of view. Even if the evaluation has almost always been commissioned by one of the partners, it would be dangerous to consider evaluation as a unilateral exercise controlled by a single decision-maker for his or her exclusive use. For this reason a steering group which allows for the expression of several points of view is often instituted in practice. The extension of the steering group to about ten stakeholders has many advantages: the people invited into the group voice the expectations and fears expressed at their level and improve the relevance of the questions asked. Their presence also makes the evaluation more credible. Greater diversity within the steering group brings together more Using multicriteria analysis within a partnership framework points of view, facilitates their confrontation, and helps to raise the level of consensus. Each participant contributes his or her own questions, value system, judgement criteria and norms. Provided that it can take advantage of this plurality, the evaluation reaches richer, more credible and useful conclusions. The taking into account of several partners' expectations allows for better acceptance of results and increases the interest that will be shown in them. Volume 1 of the Collection features a series of recommendations for constituting and managing an extended steering group. If the steering group has been extended to include more than ten persons, not all of them will be able to participate effectively in the multicriteria analysis. A group of six to ten assessors will then be chosen within the steering group. As shown below, the evaluation team will have to work with the assessors, either collectively or through individual interviews. a Expression of preferences between criteria The expression of preferences is a step intended to establish the weighting of the different criteria. The assessors' preferences are taken into account according to one of the following variants. o Direct expression of the assessors' preferences Example: Creation of a rating-weighting system in England In the framework of a socio-development programme in the North-East region of England, for the period 1991-93, a small group of ten persons developed a project selection system. The work group met several times during a three-month period. Its members were representatives of the main authorities responsible for the programme and its implementation. The group established a list of criteria and the weight of each criterion, as well as the scoring scales for the different criteria. Some meetings were conflictual and the work went through a crisis period. The members of the group nevertheless managed to reach an agreement, given the excellent spirit of partnership in the region and unanimous support for the regional development policy. After the work group had reached agreement, all the agencies involved in the programme were consulted and the authorities responsible for the programme validated the system. The system was then applied to the selection of all the projects and served as a model for most other British programmes. A first variant consists of bringing together the assessors and asking them to express their preferences in the form of a weighting attributed to each Technical solutions for evaluation within a partnership framevi/ork criterion. This can be done by means of a vote through the distribution of points. The discussion can also be conducted by means of several successive meetings, as in the example above. Revealing preferences by classification of profiles Another variant consists of presenting the assessors with "profiles" of measures or projects described in such a way that they reveal preferences between criteria. For example, the assessors may be asked to compare two "theoretical" measures A and Β with the same impacts represented schematically, as in Box 44. Box 44 ­ Profiles of two measures which favour two criteria ** / f t Α*·° J? ·& ¿Ρ & jr j? ¿f cf ¿r </ r~\_r^--- G o od i m p a ct / \ <6 é~<x f V Neutral i m p a ct Profile of Measure A ­ Profile of Measure B (KJ~ o The theoretical measure A will be described as a measure of which the impact is neutral, that is, nil, for all the criteria except the "environmental integration" criterion, for which its impact is good. In other words, the measure systematically helped to improve the environmental viability of the addressees' activities. The second theoretical measure Β will be described as a measure of which the impact is neutral, that is, nil, for all the criteria except the "strengthening of SMEs" criterion for which its impact is good. Using multicriteria analysis within a partnership framework In other words, the measure affected a large proportion of SMEs and contributed decisively to the improvement of their competitiveness. The assessors have to choose one of these t wo profiles and, if possible, must state whether their preference is weak, average, strong or very strong. The exercise is repeated for all the pairs of profiles. All the answers are processed by means of a software package which attributes a weight to each impact, expressed as a percentage so that the weightings add up to 100%. The software used is presented in the chapter on scoring scales (see page 57). Revealing preferences through the ranking of real projects Example: Classification of real projects by assessors A French urban development programme, co-financed at the regional and national levels, was evaluated mid-term. Eight evaluation criteria were selected. The evaluation team conducted a case study survey of about twenty completed projects. Each project was the subject of a monograph describing its impacts in terms of the different criteria. The evaluation team then had individual and confidential interviews lasting about two hours, with six assessors appointed by the steering group. These interviews took place as follows: • The evaluation team showed the assessor several monographs of projects with which he was thoroughly familiar. It checked whether the monographs closely reflected reality in the assessors' view. • The assessor was then Informed about three or four other monographs of projects unknown to him. The projects were chosen so that some were more successful from the point of view of certain criteria, while others were more successful from the point of view of other criteria. The monographs were generally considered sufficiently precise to make it possible to judge the projects. • The assessor was then invited to classify these three or four projects intuitively, sorting them from the best, i. e. the most worthy of programme funding, to the "worst", i. e. the least worthy. • The evaluation team and the assessor discussed the reasons for his classification. They deduced a weighting of criteria which was discussed and validated by the assessor. It was confirmed that the assessors had very different points of view on the importance to be given to the different criteria. The choice offered to the assessors in the preceding variant could have the drawback of seeming artificial. To avoid this problem, it is preferable to ask Technical solutions for evaluation within a partnership framework the assessors to state their preferences between real projects. This variant was applied in the evaluation of a French urban development programme (see Box 24). It is the one which should be chosen by the evaluation team whenever possible. The procedure is presented in the example above. • Different points of view of assessors Assessors' preferences can be revealed in meetings or in individual interviews. In the first case, a single weighting system for criteria can be deduced. In the second, the evaluation team and steering group can decide to establish average weightings, which has the effect of effacing different points of view among the assessors. By contrast, a variant consists of maintaining the individual weightings of each assessor. In this case, the anonymity of the assessors must be respected when the weightings carried out individually by them are processed. It may then be interesting to present the weightings separately, In the example of the urban for a particular category of assessor. development programme cited above, the evaluation team presented weightings, as indicated in Box 45, which depended on whether the assessors claimed to identify more with national or regional concerns. Box 45 - Differences of weighting between assessors weak weighting - strong Criterion 1 Criterion 2 Criterion 3 Criterion 4 3 Assessors close to national concerns ã] assessors close to regional concerns Using multicriteria analysis within a partnership framework Multicriteria classification of measures Global score and classification of measures At this stage of its work, the evaluation team has all the elements it needs to calculate global weighted scores for the different measures. It is considered that the results and impacts of each measure have been evaluated in relation to the same criteria, that all these evaluations have been presented in the form of scores in an impact scoring matrix, and that there is a weighting system which expresses the average preferences of assessors for a particular criterion. The global score is calculated by multiplying each elementary score by its weighting and by adding the elementary weighted scores, as in the example in Box 46 which uses the impact scoring matrix in Box 24. Box 46 -Calculation of the global scores of measures Contribution of measures to the production of expected impacts (score between 0 and 10) Urban development (average scores for measures 1 and 2) Infrastructure (average score for measures 3 to 8) Collective social actions (average score for measures 9 to 14) Personalised social actions (average score for measures 15 to 19) Weighting coefficient for criteria Social integration Employ­ ment Balance between neighbour­ hoods in a town Balance between towns in the region Average weighted score 1 5 6 6 1 4 5 6 1 4 3 2 1. 2 3 3 4. 1 4. 4 2 2 4. 2 35% 20% 20% 25% 100% Technical solutions for evaluation within a partnership framework Based on weighted average scores, the evaluation team can classify measures by order of contribution to the overall success of the programme. This synthesised judgement on the effectiveness of measures can be considered sound and impartial provided that the evaluation team has taken all the necessary precautions presented below. • Have the evaluation criteria validated by the steering group. • Present the conclusions on the impacts of each measure, as well as the impact scoring matrix summarising them, and have them validated. • Establish the weighting coefficients for criteria, with the help of the assessors chosen with the agreement of the steering group. • When veto thresholds have been applied to certain criteria, check with the steering committee that this is preferable to using an outranking approach. If, on the contrary, the evaluation team chooses and weights the criteria itself, without any interaction with its partners, the impartiality of the results will suffer and the multicriteria analysis will be less useful. Multicriteria-multijudge analysis If preferences among criteria are revealed through individual interviews with the assessors, and if these interviews show strongly divergent points of view between them, it is possible to establish several classifications of the measures. On the same impact scoring matrix, the evaluation team can apply different systems of weighting (a set of different weightings for each assessor). Differences between the weighted global scores and hence differences in ranking will result, since each measure can be considered a success from the point of view of one assessor and a failure from the point of view of another. Box 47 shows how twenty measures were rated by six different assessors, based on the "multicriteria-multijudge analysis model". In this example, the graph shows areas of consensus and dissension. For certain measures, there is consensus on the fact that they were successful (e. g. 2 and 12) or that they were not (e. g. 19 and 20). On the other hand, certain measures are the subject of heterogeneous judgements in the group of assessors (e. g. 1, 4 and 11). Multicriteria evaluation can therefore be organised with a view to producing a single synthetic conclusion at the end of the evaluation or, on the contrary, with a view to producing conclusions adapted to the preferences and Using multicriteria analysis within a partnership framework priorities of several different partners. In the case of European Union socio­ economic programmes, the different levels of partnership (European, national and regional) may be concerned. Each of these levels is legitimate in establishing its own priorities and expressing its own preferences between criteria. Box 47 -Multicriteria-multijudge ranking of measures according to their contribution to the success of the programme Assessors O CD 12 3 4 5 6 9 10 11 12 13 14 15 16 17 18 19 20 Measures The objective of multijudge analysis is therefore to take into account these different priorities in the production of evaluation results. The technique is well suited to the way in which partnerships function in so far as it outlines areas of consensus in which the partners agree on the ranking of measures, and areas of dissension which reveal the measures considered successful for some and unsuccessful for others. Experience has shown that consensual conclusions are generally in the majority. This can be explained by the fact that the different weightings apply to the same impact scoring matrix. Thus, a measure which has a low score for all the criteria will never have a high weighted global score, irrespective of the differences of priorities between partners. The different points of view of the partners the conclusions resulting from empirical cannot strongly contradict Technical solutions for evaluation within a partnership framework observation if these conclusions show that certain measures are really part of best practice and that others pose real problems of effectiveness. Experience also shows that the partners are far more willing to accept the conclusions of the report if the evaluation team has recorded their opinions carefully and taken the trouble to take their preferences into account in presenting its conclusions. Using the analysis to produce useful conclusions Recommendations on the reallocation of budgets In the context of socio-economic programmes, evaluation can influence decisions on the reallocation of budgets, either while the programme is underway or during the preparation of the following programme. The main decisions in this respect are taken at the measures level. Measures judged to be the least successful must be re-examined with a view to either reducing their effectiveness. Where relevant, recommendations can also be made to increase the budgets of those measures ranked as being the best. their budgets or re-organising to enhance them Diffusion of best practice Multicriteria analysis serves to identify the areas of success and the most the programme. The communication and effective measures of dissemination of these conclusions among the programme actors is very useful. Those measures judged as being the most successful (best practice) can be presented during seminars, and brochures or monographs describing them can be disseminated. This diffusion may also include the media, if the authorities running the programme wish to show the public how the funds of the programme were spent. It is also possible to publish concrete examples of successful measures with a view to informing the managers of similar measures financed elsewhere. The exchange of information on practices identified as being the best is an excellent way of enhancing the performance of European cohesion policy in the long run. Feedback on project selection methods The choice of evaluation criteria, their precise definition and their weighting constitute a useful contribution to multicriteria analysis. This work makes it possible to formulate a clear, complete and coherent description of the intentions and priorities of the programme partners. It is then possible to use these results to spread clear messages to the managers of the measures and the operators. Technical solutions for evaluation within a partnership framework ^ The results of multicriteria analysis can also serve to enhance the project selection process. It is relatively easy to transfer criteria, scoring scales and weightings to the project selection system if this system is also organised on the basis of scoring-weighting (see the example described on page 60). By basing the selection of projects on the same logic as the evaluation of measures, the chances of stimulating and funding projects which contribute effectively to the programme priorities are increased. To be globally successful a programme must select its projects on criteria which resemble its own effectiveness criteria as closely as possible. In-depth analysis If the "multijudge" variant of multicriteria analysis has been performed, it is also important to know how to use its non-conclusions, i. e. those measures judged favourable from certain points of view and unfavourable from others. In this case, the steering group can recommend that the authorities responsible for the programme launch a more in-depth evaluation to ascertain why there was disagreement on these measures. Was the disagreement due to inadequate definition of the criteria? Did the criteria take the policy objectives into account adequately? Did it take the points of view of all the partners into account in a fair and balanced way? Conclusion four alternative techniques for the overall This book has presented evaluation of a socio-economic development programme. All four techniques are constructed in the same spirit. They all stress the importance of interaction between the evaluation team and the partners represented in the extended steering group. They are structured and formalised without being too demanding in terms of quantitative data, and can easily be combined, as shown in the conclusion. Combination of the four alternative techniques To sum up the four techniques, the evaluation procedure can be broken down into four standard steps as shown in Box 48. The evaluation team starts by examining the programming documents to identify expected impacts. If necessary, it clarifies these impacts with the help of the steering group (step 1). It then has to choose quantitative indicators or qualitative descriptors to study the impacts (step 2). The team uses these instruments to observe results and impacts in the field (step 3). The evaluation is concluded with a synthetic judgement about the programme. This judgement takes into account the different impacts obtained and the relative importance of these impacts for the different partners (step 4). In the case of an ex ante evaluation, step 3 is replaced by a forecast of impacts, based on available knowledge. Box 48 - Sequence of the four technical solutions 1 2 3 4 Clarify the main Choose indicators Gather information Formulate a expected impacts and descriptors I and analyse it to synthetic for each impact estimate impacts judgement on the different impacts obtained Draw up the concept map of impacts Construct scoring Evaluate Apply multicriteria analysis scales measures by empirical observation of results and impacts in the field Analyse the Conclude the synergy between evaluation and make recommendations measures by means of a matrix of cross impacts Technical solutions for evaluation within a partnership framework Box 49 - Empirical study of impacts by means of parallel case studies A case study examines an individual project (the case) and concludes with a monograph describing it in a few pages. The researcher starts by gathering all available documents on the project and its management. He or she then visits the site and questions the manager of the measure to which the project belongs, as well as the operator responsible for the project. He or she also holds individual interviews with several direct and indirect addressees. Each case study involves high costs (between 2,000 and 5,000 euro) but the number of case studies is sufficiently low to contain the total cost of the evaluation within a reasonable limit. A typical survey of parallel case studies concerns between ten and thirty projects, observed and analysed with the same analytical grid. This type of survey enables information to be obtained that can be generalised, on the addressees' motives, their real needs, the reasons for their participation in the programme, their way of judging the quality of the programme, the mechanisms of the appearance of impacts, the existence of unexpected effects, etc. Because the case study survey is essentially qualitative, its main function is to better understand the functioning of the programme. Case studies can also produce quantitative or qualitative estimations of impacts, or estimations structured by means of a scoring scale. In this case, each monograph produces conclusions on the impacts of the project studied and that project only. The limited number of case studies does not allow for extrapolations of a statistical nature. It is nevertheless possible to use a case study survey to produce systematic conclusions about the results and impacts of the measures of a socio-economic programme. For example, one or more cases considered "typical" of each measure could be selected. The conclusions obtained would be used to estimate the size of the impacts, measure by measure. A more useful approach consists of studying two projects per measure, one project considered highly successful by the managers of the measure, and a project considered a relative failure. In this way a range of effects can be estimated, measure by measure. The conclusions formulated by the evaluation team can then usefully be based on a survey of parallel case studies, on the understanding that other sources of information cannot be neglected (monitoring data on outputs, opinions of the managers of the measures, analysis of the impacts of comparable measures, etc. ). For a more complete presentation of the case study technique, the reader is referred to Volume 3 of the MEANS Collection. For undertaking an overall evaluation of a programme in a context of partnership, it is possible to articulate the different techniques presented in this Volume, as indicated in Box 48. This diagram shows, however, that the Conclusion application of the four techniques does not constitute a complete evaluation. In particular, they do not concern the core of the evaluation, that is, the empirical observation of the results and impacts produced by the measures in the field (step framed with a bold line in the diagram). No evaluation has meaning if this function has not been fulfilled. Different strategies can be adopted for the empirical observation of results and impacts, given that the main difficulty here consists of observing several impact clusters for many measures which are not intended for the same publics. Each evaluation team must choose the best approach possible, in relation to the characteristics of the programme and its publics. Box 49 shows a possible solution, based on the principle of parallel case studies. A global approach which is above all quick There are several ways of combining the alternative techniques presented in this Volume. The first approach presented here facilitates the implementation of an evaluation. This approach may proceed as follows: • Create an extended steering group (see pages 29 and 106). • Identify expected impacts in the programming documents (see page 30). • Define between five and seven impact clusters, using the METAPLAN® technique, according to the procedure presented on page 23. This step requires an initial meeting with the steering group, possibly combined with the validation of the evaluation plan. • Evaluate the impacts of the measures (this step is important and difficult, but is common to all types of evaluation). • Combine all the qualitative and quantitative conclusions concerning the impacts of the measures, in the form of a synoptic table (see Box 16). • Transform the synoptic table of conclusions into an impact scoring matrix, according to the simplified procedure presented on page 55. This step necessitates a second steering group meeting, possibly combined with the presentation of the surveys carried out by the evaluation team. • Construct the matrix of cross impacts, analyse all the potential effects of synergy and produce coefficients of synergy (see page 81 onwards). • Ask the steering group to weight the criteria, according to the simplified procedure presented on page 107. This step necessitates a steering group meeting, possibly combined with the presentation, by the evaluation team, of an initial series of conclusions. Technical solutions for evaluation within a partnership framework • Calculate weighted global scores for all the measures (see page 111) and use them as a basis for global conclusions and recommendations for the re-allocation of inputs. As Box 50 shows, this entire procedure hardly requires more work than any good quality evaluation. Box 50 -Time required for implementing a simplified approach Required time (depending on the complexity of the programme) Constitution of extended steering group Identification of expected impacts and constitution of clusters Evaluation of measures Synoptic table of conclusions and impact scoring matrix Identification of potential synergy and global analysis of effects of synergy Weighting of criteria and calculation of weighted scores Production of evaluation report, synthesised conclusions and recommendations Additional time compared to another evaluation Min. no. of days Max. no. of days (not included in the evaluation) (not included in the evaluation) 4 15 (as in any other evaluation) (as in any other evaluation) 3 4 3 6 15 6 (as in any other evaluation) (as in any other evaluation) 14 42 • A global approach which is above all technically rigorous In the second approach proposed, all four techniques are used in their most rigorous form. The approach, which may be applied to a mid-term or ex post evaluation, consists of the following steps: • Constitute several sectoral work groups, as well as an extended steering group which will function as a transversal group (see pages 29, 38 and 106). • Draw the concept map of impacts (see pages 30 to 39). Conclusion • For each impact cluster, define a structured scoring scale (see Boxes 20 to 22) based on the impact statements comprising that cluster (see Box 12). • Construct the matrix of cross impacts and study the synergy between the measures (see pages 81 to 85) and identify synergy to studied more in- depth. • Select one or more typical projects for each measure, as well as the projects which produce the most potential synergy. • Conduct a survey of parallel case studies of all the selected projects. Present the conclusions of each case study in the form of a series of scores, using scoring scales constructed for the evaluation (see Box 20) • Write up qualitative conclusions, measure by measure, based on the case studies and on all the other available and relevant information. Summarise all the conclusions in the form of an impact scoring matrix (see Box 19). • Perform the multijudge variant of multicriteria analysis (see pages 99 to 114). • Conclude the evaluation and proceed to recommendations on the reallocation of budgets or changes in strategy (see page 115). It is not realistic to give indications of costs for the implementation of this more technical approach because it has to be finely adjusted to the context and nature of the evaluated programme. It should be noted, however, that it involves a series of meetings with the programme partners in the form of group work, steering group meetings or individual interviews with the assessors chosen for the multicriteria analysis. The approach also includes empirical observation of the effects of the measures. It therefore involves visits in the field and interviews with the addressees. In short, because of the concern for technical quality, the evaluation is scheduled to last at least eight months. It is therefore clear that this approach is more cumbersome than those adopted until now by many evaluations. This additional investment in evaluation can be justified if the return on investments is sufficient. It should be remembered that the results which can be expected from evaluations, and have generally not been achieved satisfactorily to date, are: • Conclusions and recommendations which propose and justify budgetary reallocations and changes of priorities between the measures of a programme. Technical solutions for evaluation within a partnership framework Conclusions and recommendations based not on absorption rates and completion rates, but on a real examination of impacts and on the contribution of the measures to the overall success of the programme. Conclusions and recommendations which impartially take into account all the opposing points of view, in the framework of partnership. Appendix Bibliographie references Bana e Costa, CA. and Vansnick, J. C. (1994) 'MACBETH ­ An interactive path towards the construction of the cardinal value functions', in International Transactions in Operations Research, 1(4), 489­500. Guba, E. G. , Lincoln, Y. S. (1989) Fourth Generation Evaluation, Newbury Park: Sage Publications. Hemerson, M. E. , Morris, L. L. , Fitz­Gibbon, G. T. (1987) How to Measure attitudes?, London: Sage publications. Knox, C. (1995) 'Concept mapping in policy evaluation: a research review of Community relations in Northern Ireland', in Evaluation, the International Journal of Theory, Research and Practice 1(1), 65­79. Miller, D. C. (1991) 'Guide for the selection and construction of social scales and indices', in Handbook of Research Design and Social Measurement, London: Sage publications, 174­188. Monnier, E, (1992) Evaluations de l'action des pouvoirs publics, Paris: Economica. OECD ­ DAC Expert Group on Aid evaluation (1996) The Role of Rating Systems in Aid Management Experiences of DAC Member Organisations, Paris: OECD (unpublished). Schärlig, Α. , (1985) Décider sur plusieurs critères, Lausanne: Presses polytechniques et universitaires romande. Roy, B. (1997) 'Un chaînon manquant en Recherche Opérationnelle ­ Aide à la Décision : les conclusions robustes', Cahiers du LAMSADE, 144, Paris: Université Dauphine. Toulemonde, J. , Fontaine, O, Laudren, E. , Vincke, Ph. (1998) 'Evaluation ¡n partnership: Practical suggestions for improving their quality', in Evaluation, the International Journal of Theory, Research and Practice, 4(2). Trochim, W. M. K. (1989), 'An introduction to concept mapping for planning and evaluation', in Evaluation and Program Planning, 12, 1­16. Vincke, Ph. , (1989) L'aide multicritère à la décision, Bruxelles: Editions de l'Université de Bruxelles,. English translation (1992), Multicriteria Decision­aid, Chichester: Wiley. Technical solutions for evaluation within a partnership framework Yin, R. K. , Kaftarian, S. J. , Jacobs, N. F. (1996) 'Empowerment Evaluation at Federal and Local Levels: Dealing with quality', Fettermann & al. Empowerment evaluation, London: Sage, 188-207. Yoon, K. P. (1995) Multiple Attribute Decision Making, London: Sage publications. Young, F. W. W. , Harris, D. G. (1994) 'Multidimensional Scaling', Norusis M. J. SPSS Professional Statistics: SPSS Inc. Index Index action 17, 8 1, 99 addressee 18, 23, 50, 87, 89 axis 17, 22, 8 1, 88 Β best practice 105, 114, 115 cannibalism effect 77 casestudy 43, 67, 120, 123 colour vote 96 commissioner 88 comparability 64 comparison group 49 compensation 104 complementarity 68, 74, 76 concept mapping of impacts 11, 29, 100, 122 concerned group 50 co-ordination 76 cost-benefit analysis 95 cost-effectiveness analysis 94 counterfactual situation 76 credibility 56, 94 criteria 20, 96 critical mass 75 dead-weight 50 descriptor 6 1, 62, 64, 102, 119 direct addressee 95 economic and social cohesion 17, 21 effectiveness 97 efficiency 97 evaluation committee 56 evaluation team 23, 3 1, 49, 58, 103, 110 ex ante evaluation 12, 21,23, 99 ex post evaluation 49 expert panel 56, 59, 61 extrapolation coefficient 49 focus group 31 funding authority 12 global impact 23, 27 global objective 17, 2 1, 24, 94 Η harmonisation 64 I impact 18 impact scoring matrix 56, 58, 63, 67, 100, 111 impact statement 32, 61 impartiality 94, 112 implicit objective 27, 3 1, 97,. 100 in-depth evaluation 49 indicator 19, 20, 2 1, 24, 44, 50, 64, 96, 102, 119 individual interview 3 1, 89, 106, 107, 112 integrated programme 73 interdependence 76 intermediate impact 26, 100 internal coherence 76 internal synergy 75 Technical solutions for evaluation within a partnership framework logical diagram of impacts 68 logical framework 23, 24 M manager 26, 27, 3 1, 39, 79, 84, 89 matrix of cross impacts 12, 52, 8 1, 82, 123 measure 17, 43 METAPLAN® 37, 96, 121 mid-term evaluation 2 1, 27, 49 monitoring 2 1, 49 multicriteria analysis 12, 52, 96, 99, 123 multicriteria-multijudge analysis 112 N needs 97 negative synergy norm 97 77 objectives tree 24, 26, 27, 82 operator 26, 27, 3 1, 39, 50 output 18 outranking 104 overall evaluation 12, 13, 19, 20, 2 1, 4 9 , 7 8, 117 partnership evaluation 96 performance 50 policy priority 100 programme 17 project 18 project selection 59, 64 qualitative data 29 quantification 95, 96 questionnaire survey 43 ranking 73, 105, 112, 113 rating 59, 107 regression analysis 49 relevance 97 result 18 robustness 22 sample 95 scoring 1 2 , 3 4 , 5 1 , 5 5 , 5 8 , 8 1 , 8 6 , 9 6, 107, 109, 123 specific impact 19 specific objective 21 stakeholder 11 steering group 29, 56, 6 1, 96, 99, 100, 105, 106, 116, 121,122 success 24, 34, 112, 115 SWOT 30, 96 synergy 5 1, 73, 74, 79, 83, 87, 88 synoptic table of conclusions 5 1, 55, 58,63 w weighted global score 59 weighting 37, 60, 73, 104, 107, 109, 110, 111, 112, 115 Acknowledgements The preparation of this Volume took place over several years from 1993 to 1998. The preparatory work included the following steps: • Several workshops and meetings allowed the points of view of experts in different themes and the transversal (or multi-thematic) perspective of those responsible for the Structural Funds to come together. • A series of pilot evaluations allowed the technical solutions proposed in this book to be tested and stabilised. The production of this Volume was coordinated by Miguel Benito Alonso and Andrea Mairate (D. G. XVI. G. 2 : Coordination of evaluation) and involved several Directorate- Generals of the European Commission, notably D. G. II, V, VI, XIV and XVI and in particular Philippe Goybet and Anastassios Bougas. This Volume has been written by the Centre for European Evaluation Expertise, notably with the participation of C. Diaz-Gomez and Céline Fontaine. It was translated into English by Liz Libbrecht. The authors wish to thank all those who contributed, either directly or indirectly, to the production of this Volume, and in particular the following experts: J. Aldas (IVIE - Valencia), J. Anselmo (Université Bocconi - Milan), C. Bana e Costa (CESUR, Lisboa), J. Bignè (IVIE - Valencia), J. Cuadrado (QUASAR - Madrid), C. Gillo (Ministère de l'Equipement, Paris), S. Green (Department of Environment, London), C. Knox (University of Ulster), J. Malone (Ernst & Young, London), S Martin (Warwick Business School, United Kingdom), C. Mouilleron (RIDER - University of Louvain, Belgium), M. Rey (CEAT, Lausanne, Switzerland), P. Tyler (Cambridge Policy Consultants, UK), Ph. Vincke (Université Libre de Bruxelles), RViveret (Cour des Comptes, Paris). The authors wish also to thank everyone from regional institutions who co-operated very openly to the pilot evaluations, and more particularly: Javier Quesada-lbanez (Communauté autonome de Valencia, Spain), Luc Vandendorpe (Région VVallonne, Belgium),Manuel Vaquero et Delphine Vincent (Région Poitou-Charentes et Préfecture de Région, France). European Commission Evaluating socio-economic programmes Technical solutions for evaluation within a partnership framework Luxembourg: Office for Official Publications of the European Communities, 1999 1999— 133 pp. — 14. 8x21 cm ISBN 92-828-6626-2 Venta · Salg · Verkauf · Πωλήσεις · Sales · Vente · Vendita · Verkoop · Venda * Myynti · Försäljning BELG1QUE7BELG1É ÖSTERREICH J un De Lannoy Avenue du Rol 202/Kcningsiaan 202 B­1190 Bruxelles/B russel T«. (32-2)538 43 06 Fax (32­2) 538 08 41 E­mail fean de lannoy Omfoboard be URL: http "www ¡ean­do­lannoy bo La librairie européenne/ De Europese Boekhandel Rue de la Loi 244/Wetslraat 244 B­I040 Bruxelles/Brussel Tél. (32­2) 295 26 39 Fox (32­2) 735 08 60 E­mail: mailOhbeurop be URL: http "www. iibeurop be Moniteur belge/Belgisch Staatsblad Rue de Louvain 40­4 2/Le uve rise weg 40­42 B­1000 Bnjxelles/Brusser Tel. (32­2) 552 22 11 Fax (32­2) 511 0184 DANMARK J. H. Schultz Information A/S Hersledvang 10­12 DK­2620 Albertslund TU. (45)43 63 23 00 Fax (45) 43 63 19 69 E­mail: scbultzO schnitz dk U RL http //www schutti. dk DEUTSCHLAND B undosa n ie I ger Verlag GmbH V e π r ιβΟ sa b ι e 11 ung Amsterdamer Straße 192 D­50735 Köln Tel (49­221) 97 66 BO Fax (49­221 ) 97 66 B2 78 E­Mail: vertrieb O bundesanzeigef de U R I: httpy/www­bundesanzetger de ΕΛΛΑΔΑ/GREECE G. C. EleflheroudakJs SA Inlemat onal Bookstore Panepisümiou 17 GR­10564 Athina Tet(30­1)331 41 8071/2/3/4/5 Fax (30­1) 323 98 21 E­mail: elebooksOnetor. gr ESPAÑA Boletín Oficial del Estado Trafalgar. 27 E­29071 Madrid Tel (34)915 38 21 11 (Libros). 913 84 17 i 5 l S u s c n p) Fax (34) 915 38 21 2t (Lx&ros). 913 84 17 14(Suscnp) E­mail: Chentes O com boe es URL: http­Jnffww boe es Mundi Prensa U b r o s, SA Castello. 37 E­28001 Madrid Tel (34)914 36 37 00 Fax (34)91575 39 98 E­mail: lxxeriaOmundiprensa. es URL: http y/www. mundxxensa com FRANCE Journal otticici Service des pubhcabons des CE 26. rue Desa« F­75727 Paris Cedex 15 Tel (33)140 58 77 31 Fax (33) 140 58 77 00 URL: httpJ/www journal­officiel gouv Ir IRELAND Government Supplies Agency Pubbcalions Section 4­5 Harcourt Road Dublin 2 Trt (353­1)661 31 11 Fax (353­1) 475 27 60 ITALIA Licosa SpA Via Duca di Calabria. 1/1 Casella postale 552 1­50125 Firenze Tel (39)055 64 83 1 Fax (39)055 64 12 57 E­mail licosaOftbccit URL: hitp www ttbcc rl1>cosa LUXEMBOURG Messageries du livre SAHL 5, rue Rarrleisen L­2411 Luxembourg Tel. (352)40 10 20 Fax (352) 49 06 61 E­mail: maHOmdllu URL http//www moi lu NEDERLAND SDU Servicecentrum Uitgevers Chnstoflel Plantx/islraat 2 Postbus 20014 2500 EA Den Haag Tel. (31­70) 378 98 80 Fax (31­70) 378 97 83 E­rnaü" sduOsdu nl URL: http /«ww sdu nl Universi ta t sbuchhendlung GmbH Kohlmarkt 16 A­1014 Wien Tel. ( 4 3 ­ 1 ) 5 3 1 6 1 1 00 Fax(43­1)53 16 1l 67 E ­ M ail : beste ι len Ο ma n ζ co. a I URL: http J/www. manz. al/index htm PORTUGAL Distribuidora de Livros Bertrand Ld. " Grupo Bertrand. SA Rua das Terras dos Vales. 4­A Apartado 60037 P­2700 Amadora Tel. (351­1)495 90 50 Fax(35l­1,496 02 55 Imprenta Nacional­Casa da Moeda, EP Rua Marques Sa da Bandeira, 16­A P­1050 Lisboa Codex Tel (351­1)353 03 99 Fax (351­1) 353 02 94 E­mail: oal. incmOrruil. telepac. pt URL: hrtp^/www incm. pt SUOMI/FINLAND Akateeminen Kirjakauppa/ Akademiska Bokhandeln Keskuskatu 1/Centralgatan 1 PL/PB 128 FIN­00101 Helsinki/Helsingfors Ρ. /tfn (358­9) 121 44 18 F/lax (358­9) 121 44 35 Sähköposti: akaiitausOakaleemnencom URL: httpJ/www akateeminen com SVERIGE B T J AB Traktorvägen 11 S 221 82 Lund TIn (46­46) 18 00 00 Fax (46-46) 30 79 47 E­post: bt)eu­pubObt| se URL: httpJ/www bt| se UNITED KINGDOM The Stationery Office Ltd International Sales Agency 51 Nine Elms Lane London SW8 5DR Tel (44­171)873 90 90 Fax (44­171)873 84 63 E­mail: ipa enquiries O theso. couk URL: http^/www Ihe­stabonery­otfice co uk ISLAND Boka bud Larusar Blondel SkOlavôrdustig. 2 IS­101 Reykjavik Tel. (354)551 56 50 Fax(354)552 55 60 NOflGE S wet β Norge AS Ostenjoveien 18 Boks 6512 Etterstad N­0606 Oslo Tel. (47­22) 97 45 00 Fax (47­22) 97 45 45 SCHWEIZ/SU ISSE/S VIZZE ΠΑ Euro Info Center Schweb: c/o OSEC Stamplenbachsiraße 85 PF 492 CH­8035 Zurich Tet (41­11365 53 15 Fax ¡41­1) 365 54 11 Ε­maa: eicsOosec eh URL: tinp ,'. www osoc ctveica BALGARUA Europres« Euromedia Ltd 59. btvd Vitosha BG­1000 Sol­a Tel (359­2)980 37 66 Fai (359­2) 980 42 30 E­mail: MilenaOmbox. at. bg CESKAREPUBLIKA ÚSIS NIS­piodejna Havelkova 22 CM 30 00 Praha 3 Tel (420­2) 24 23 14 86 Fax (420­2) 24 23 11 14 E­mail: nkpospOdec nis cz URL http /. usiscr cz CYPRUS Cyprus Chamber of Commerce and Industry PO Sox 1455 CY­1509 Nicosia Tel (357-2)66 95 00 Fax (357­2)661044 E­mail oametiapOccci org cy HRVATSKA Medlalrada Lid Pavia Hatza 1 HR­10000 Zagreb Tel. (385­1)481 94 1 Fax (385­1)481 94 1 MAGYARORSZÂG Euro Info Servie· Europa Haz Margitsziget PO Box 475 H­1396 Budapest 62 Tel (36­1)350 60 25 Fax (36­1)350 90 32 E­mail: euroinloOmail. matav. hu URL: hup. 'Vwww euiomlo hu'mdox him MALTA Miller Distributor» Ltd Matta International Airport PO Box 25 Luqa LOA 05 Tel. (356)66 44 88 Fax (356) 67 67 99 E­mail: gwirthOusa nel POLSKA Ars Polona K ra k ow s k ie Priedmiescie 7 Skr. pocztowa 1001 PL­OO­950 Warszawa Tel (48­22)826 12 01 Fax (48­22) 826 62 40 E­mail ars_potObevy hsn com pi ROMANIA Euromedia Str. G­ral Benhelot Nr 41 RO­70749 Bucuresti Tel (40­1)315 44 03 Fax (40­1 )314 22 86 ROSSIYA CCEC 60­letrya Oktyabrya Av. 9 117312 Moscow Tel. (7­095) 135 52 27 Fax (7­095) 135 52 27 SLOVAKIA Centrum VTl SR Nâm. Slobody. 19 SK­81223 Bratislava Tel. (421­7) 54 41 83 64 Fax (421­7) 54 41 83 64 E­mail: europOtbbl sltkstuba. sk URL· http^www. sttk sluba sk SLOVENUÀ Gosp­xiarskl Vestnlk Dunajska cesta 5 SLO­l00OL|ubl|ana Tel ( 3 8 6 ) 6 1 3 0 9 1 6 40 Fax (386) 6 1 3 0 9 1 6 45 E­mail europOgvestnik si URL: httpJnvww rjveslrMk si TURKIYE Oünya Intotel AS 100. Yd Mahallessi 34440 TR­80050 BaocUar­lslanbul Tel. (90­212)629 46 89 Fax (90­212) 629 46 27 E­mail infotelOdunya­gazate. com tr AUSTRALIA Hunter Publications PO Box 404 3067 Abbotsford. Victoria Tel. (61­3)94 1 7 5 3 61 Fax (61­3) 94 19 71 54 E­mail : jpda vies O oz ema il com. au CANADA Les editions La Liberté Inc. 3020. chemin Samte­ Foy Gl X 3V Sainte­Foy. Québec Tel, (1­418)658 37 63 Fax (1­800) 567 54 49 E­mail libetleOmediomqc ca Renouf Publishing Co. Ltd 5369 Chemin Canotek Road Unit 1 KU 9J3 Ottawa. Onlario Tel. (1­613) 745 26 65 Fax (1­613) 745 76 60 E­mail: order. deptOrenoufbookscom URL: http7/www renoufbooks. com EGYPT The Middle East Observer 41 Sher. i Streel Cairo Tel. (20­2)392 69 19 Fax (20­2) 393 97 32 E­mal mafouoaOmeobservef com eg URL: h tip. //www meob server com eg EESTI INDIA Eesti Kaubandus­Tooatuekoda {Estonian Chamber ol Commerce and Industry) Toom­Koc* 17 EE­0001 Tan­nn Tel (372) 646 02 44 Fax(372)646 02 45 E­mail: einfo O koda ee URL: htlp : www koda ee EBIC India 3rd Floor. Υ. Β Chavan Centre Gen J, Bhosale Marg 400 021 Mumbai Tel (91-22)282 60 64 Fax (91 ­22) 285 45 84 E­mail: ebicOçaasOmOl. vsnl. net. in URL· httpJ/www ebKindia com ROY International 41. Mishmar Hayarden Street PO Box 13056 61130 Tel Aviv Tel. (972­3)649 94 69 Fax (972­3) 848 60 39 E­mail: royilOnetvision. net. il URL· httpJ/www. royint. co it Sub­agent lor the Palestinian Authority: Index Information Services PO Box 19502 Jerusalem Tel. (972­2) 6 2 7 1 6 34 Fax (9722)627 12 19 JAPAN PSI­Japan Asani Sanbancho Plaza 1206 7­1 Sanbancho. Chiyoda­ku Tokyo 102 Tel. (81­3)32 34 69 21 Fax (81­3)32 34 69 15 E­mail: booksOpsi­)apan. co jp URL: http. '/www psi­|apan. com EBIC Malaysia Level 7, Wisma Hong Leong 18 Jalan Perak 50450 Kuala Lumpur Tel. (60­3) 262 62 9B Fax (60­3) 262 61 98 E­mail: ebic­klOmol. nelrny Mundi Prensa Mexico, SA de CV Río Panuco No 141 Colonia Cuauhtemoc MX­06500 Mexico. DF Tel (52-5) 533 56 58 Fax (52­5) 514 67 99 E­mail 101545 2361 Ocompuserve. com EBIC Philippines I DIN Floor, PS Bank Tower Sen. Gil J. Puyal Ave cor Tindalo St. Maketi City Metro Manilla Tel. (63-2) 759 66 80 Fax (63-21 759 66 90 E­mail: eccpcomQglooe. com ph URL: httpjAvww. eccp. com EBIC Sri Lanka Trans Asia Hotel 115 Sir chittampalam A Gardiner Mawatha Colombo2 Tel. (94­1)074 71 50 78 Fax (94­1)44 87 79 E ­ mail : ebicsl O Umin com EBIC Thalland 29 Vanisse Building, Sth Floor SoiCh. dlom Ptoenchil 10330 Bangkok Tel (66­2)655 06 27 Fax (66­2) 655 06 28 • E­mail: eblcbkkOksclS. th. com URL: http /www. ebÆbkk org UNITED STATES OF AMERICA Bornen Associates 4611­F Assembly Dnve U n h am MD207Ô6 Tel (1­800) 274 44 47 (lollfree telephone) Fax (1­800) 865 34 50 (loll free lax) E­mail: query O beman com URL: hftpj/vnvw bernan. com Bitte wenden Sie sich an «In Büro Ihrer Wahl/ Please coniaci Ih« salts office of your choice/ Veuille* vous adresser su bureau de vente de votre choix Office for Officiel Publications of the European Communities 2. rue Mercier L­2985 Luxembourg Tel (352) 29 29­42455 Fax (352) 29 29­42758 E­mail: info infoOopoce cec. be URL· http://eur­opeu. lnt The MEANS collection: "Evaluating socio-economic programmes" This six-volume set gives wider public access to the lessons of the MEANS programme which the European Commission introduced with a view to Impro­ ving and promoting evaluation methods. Its publication meets a growing need for reliable benchmarks to deal with the problems of evaluation encoun­ tered when implementing public measures in general and Structural Fund ope­ rations in particular. In this context, the set constitutes a unique methodological guide that can solve the technical and organisational problems arising in the course of evaluation work and for which the existing literature provides little help. The collection is aimed not only at professional evaluators, for whom it will provide a valuable handbook, but also at non-specialists, in particular manag­ ing authorities and structural programme Monitoring Committees. Evaluation design and management Each volume is designed to be independent of the others, yet they are also complementary. Together, they deal with all aspects of the evaluation of socio­ economic programmes: I. II. Selection and use of indicators for monitoring and evaluation III. Principal evaluation techniques and tools IV. Technical solutions for evaluation within a partnership framework V. Transversal evaluation of impacts on the environment, employment and other intervention priorities VI. Glossary of 300 concepts and technical terms OFFICE FOR OFFICIAL PUBLICATIONS OF THE EUROPEAN COMMUNITIES L-2985 Luxembourg